Server Host Setup

1. Introduction

This page explains what machine is required to run Sharemind HI, and how to set it up.

2. Processor Requirements

The machine requires a CPU with the Intel® SGX instruction set. Consult the documentation of your CPU to verify:

the availability of the Intel® SGX instruction set,
the availability of EPID or ECDSA attestation (whichever of the two you need),
the availability of large enclaves if needed.

3. Operating System Requirements

When using a bare metal server: Ubuntu 22.04
When using a VM: Ubuntu 22.04
When using Docker: Guest Ubuntu 20.04 or Ubuntu 22.04, Host any Linux with Kernel >= 5.11

Let us know if you have different requirements.

4. BIOS / UEFI Firmware Update

The machine’s BIOS / UEFI firmware needs to be updated to the latest version which fixes known security vulnerabilities. Note that vendors and cloud service providers might need additional time to provide BIOS updates after Intel® released necessary updates.

5. BIOS / UEFI Configuration

SGX needs to be enabled: If the option Enabled is available, this option shall be used. Otherwise, if only Software Controlled is available, SGX must be enabled separately. Please refer to the Software Enabling Application for Linux.
Owner EPOCH: The Owner EPOCH value is used during key derivation by Intel® SGX. You can add additional entropy to this key derivation process by overwriting the preset value, e.g. setting a custom value upon acquiring the hardware and once before disposing of it. However, once done you should not change it anymore as long as you need the Sharemind HI Server to restore its state from a previous boot.
Hyper Threading needs to be disabled: It is insufficient to disable logical cores using kernel parameters or at runtime. This option may only be available after updating the BIOS / UEFI firmware.
The iGPU needs to be disabled: Only if your processor is affected by Intel Security Advisory 00219.

Whenever the BIOS / UEFI firmware is updated, the configuration needs to be reviewed for any unwanted changes to the required options.

6. ECDSA attestation

The setup consists of the following major steps:

Subscription to the Intel® Provisioning Certification Service for ECDSA Attestation (PCS).
Setup a service which downloads and stores the certificates and additional data (attestation collateral) required for quote verification. We use Intel®'s reference implementation, the Intel® SGX DCAP Provisioning Certification Caching Service (PCCS).
Fill the PCCS with all necessary data: CPU information from the SGX platform and the respective quoting collateral from the Intel® Provisioning Certification Service to support remote attestation.

If you use a cloud server: Your cloud provider might already provide an internal Provisioning Certification Caching Service, in which case you should follow the documentation of your cloud provider and skip this section.

The setup shown here is a condensed setup guide. You might also refer to the official documentation with more options from Intel®: A Quick Install Guide, or a detailed installation reference.

Figure 1. The Caching Service and additional helper libraries need to be installed. The Intel® SGX DCAP Orientation Guide contains a more complete drawing. Note: The DCAP Quote Verification Library used in Sharemind HI contains custom modifications to make it more portable.

6.1. PCS Subscription

Go to https://api.portal.trustedservices.intel.com/provisioning-certification
Go to "Get PCK Certificate/s" API section and click "Subscribe" button
Review API summary page and confirm subscribtion request.
On the subscribtion details page you’ll find active subscriptions. Find your Intel SGX PCS subscription and API keys. There are two keys. Both are valid. Pick one to use during the platform setup.

6.2. Platform Setup

6.2.1. Setup the PCCS

This installs the PCCS, a caching service between the Sharemind HI Server and the Intel® servers. This could be installed on a non-SGX server if need be.

# Install node.js
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt install nodejs

# Install SGX components
wget -O - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key \
    | sudo apt-key add -
echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu jammy main' \
    | sudo tee /etc/apt/sources.list.d/intel-sgx.list > /dev/null
sudo apt update
sudo apt install sqlite3 python3 build-essential

# NB This is interactive and requires your input:
sudo apt install sgx-dcap-pccs
# default listening port
# local connections only: N
# PCS API key: enter primary or secondary key
# caching fill method: REQ
# admin password used for administering the PCCS service
# user password used for provisioning clients and
#      by provision systems that are running
#      runtime workloads (this is required below!)
# generate HTTPS key/cert for PCCS: Y

If need be, the PCCS can be configured (CachingFillMode) to not connect to the Intel® Provisioning Certification Service directly to support situations where the PCCS does not have internet access. The above documentation configured the REQ mode which makes use of PCKIDRetrievalTool, as described in the next section. By default (RefreshSchedule) the PCCS refreshes the information from the Intel® Provisioning Certification Service once per day.

6.2.2. Provision the Intel® SGX Platform

# Install SGX components, if not done already by the previous step on the same machine.
wget -O - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key \
    | sudo apt-key add -
echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu jammy main' \
    | sudo tee /etc/apt/sources.list.d/intel-sgx.list > /dev/null
sudo apt update
sudo apt install sgx-pck-id-retrieval-tool libsgx-dcap-ql

# Tell `PCKIDRetrievalTool` how to connect to our PCCS. Maybe change the URL if
# the PCCS runs on a separate machine.
echo "proxy_type = direct" \
    | sudo tee -a /opt/intel/sgx-pck-id-retrieval-tool/network_setting.conf \
    > /dev/null
echo "PCCS_URL=https://localhost:8081/sgx/certification/v4/platforms" \
    | sudo tee -a /opt/intel/sgx-pck-id-retrieval-tool/network_setting.conf \
    > /dev/null
# You can overwrite the user_token, too. You set this while installing `sgx-dcap-pccs` above.
# sudo nano /opt/intel/sgx-pck-id-retrieval-tool/network_setting.conf

# Send platform info to the PCCS
PCKIDRetrievalTool

Check on the server which runs the PCCS whether new data is present:

# Check that data got into the PCCS.
sqlite3 -box /opt/intel/sgx-dcap-pccs/pckcache.db \
    'select fmspc, type, version, created_time, updated_time from fmspc_tcbs;'

Details on the data stored by the PCCS

If you are interested in what information is stored by the PCCS, here is a quick overview:

Each SGX-capable platform family is represented by an fmspc value:
- CPU family, model, stepping
- System board (they are paired with)
Platforms have Trusted Compute Base (TCB) information:
- Multiple TCB levels: Hardware patches (BIOS/microcode), Software patches, BIOS configuration
- TCB levels have a status: Security patching status (current, behind), or current patches but insecure BIOS configuration.
SGX-capable server systems are part of a platform family
In the Caching Service’s database an SGX-capable server system is uniquely identified by combination of:
- quid (pseudo platform hardware identifier)
- pceid (platform certification enclave id)
- eppid (encrypted platform provisioning id)
- platform manifest (multi-socket platforms only)
SGX-capable server systems are issued Provisioning Certification Key (PCK) certificates
- per server
- per TCB level (a system can have multiple PCK certificates, each at a different TCB level)
The Caching Service must store
- fmspc and tcbinfo for each platform
- qeid, pceid, eppid for each server; and platform manifest for multipackage servers
- PCK certificates for each server at each TCB level: M systems × N TCBlevels = M*N certificates
- Identities of the Quoting Enclave and the Quote Verification Enclave (not used by Sharemind HI)
- Processor and Platform CRLs
- All certificate chains

The PCKIDRetrievalTool is no longer needed at that point.

6.2.3. Prepare the SGX Environment for the Sharemind HI Server

# Install SGX components
wget -O - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key \
    | sudo apt-key add -
echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu jammy main' \
    | sudo tee /etc/apt/sources.list.d/intel-sgx.list > /dev/null
sudo apt update
sudo apt install libsgx-quote-ex libsgx-dcap-ql libsgx-dcap-default-qpl \
    libsgx-aesm-quote-ex-plugin libsgx-dcap-quote-verify

# Uncomment ECDSA quoting type, restart
sudo sed -E -i 's/#(default quoting type = ecdsa_256)/\1/' /etc/aesmd.conf
systemctl restart aesmd
systemctl status aesmd

# Configure the quote provider library
# For production, set USE_SECURE_CERT to “TRUE”.
sed -i -E 's/"use_secure_cert": true/"use_secure_cert": false/g' \
    /etc/sgx_default_qcnl.conf
# If the PCCS runs on a separate machine, configure the correct URL.
#sed -i -E 's|"pccs_url": ".*"|"pccs_url: "https://example.com/path"|g' \
#    /etc/sgx_default_qcnl.conf

6.3. Validation of SGX Installation

Check if the SGX kernel module is set up properly:

$ sudo dmesg | grep sgx
[    0.432025] sgx: EPC section 0x70200000-0x75ffffff
...

Check if the Architectural Enclave Service Manager is set up properly:

$ systemctl status aesmd.service
● aesmd.service - Intel(R) Architectural Enclave Service Manager
     Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-08-25 13:30:43 EEST; 51s ago
...

If the output is similar to the above, then the server should be ready to run SGX applications.

7. EPID attestation

7.1. SGX Platform Software

The SGX platform software package contains services and Intel enclaves which are required to perform operations such as enclave provisioning and enclave remote attestation.

. /etc/os-release # for $UBUNTU_CODENAME.

# Install HTTPS repository support for APT
sudo apt install apt-transport-https

url="https://download.01.org/intel-sgx/sgx_repo/ubuntu"

# Add SGX repository key
wget -qO - "$url/intel-sgx-deb.key" | sudo apt-key add -

# Add the package repository
echo "deb [arch=amd64] $url $UBUNTU_CODENAME main" \
    | sudo tee /etc/apt/sources.list.d/intel-sgx.list

# Update package information
sudo apt update

# Install the required packages
sudo apt install \
    libsgx-urts \
    libsgx-uae-service \
    sgx-aesm-service \
    libsgx-aesm-launch-plugin \
    libsgx-aesm-epid-plugin

7.2. Validation of SGX Installation

Check if the SGX kernel module is set up properly:

$ sudo dmesg | grep sgx
[    0.303850] sgx: EPC section 0x90200000-0x95f7ffff

Check if the Architectural Enclave Service Manager is set up properly:

# Check if the Architectural Enclave Service Manager started correctly
$ systemctl status aesmd.service
● aesmd.service - Intel(R) Architectural Enclave Service Manager
   Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-05-08 11:47:56 EEST; 1 day 6h ago
...

If the SGX kernel module output and the AESMD service output is similar to the above, then the server should be ready to run SGX applications.

8. Sharemind HI Server Installation

First you need an APT repository URL provided by Cybernetica, which also contains the APT repository access credentials. The URL is usually in the form https://<username>:<password>@repo.cyber.ee/sharemind/apt/<path> where <username>:<password> and <path> are replaced with the access credentials and a path respectively.

Next, the following commands must be run to install the Sharemind HI server:

# Add the Sharemind HI repository key:
wget -qO - https://repo.cyber.ee/sharemind/apt/pubkey.gpg | sudo apt-key add -

# Install HTTPS repository support for APT, if not already done in previous steps:
sudo apt install apt-transport-https

# Add the Sharemind HI repository:
user="..." # provided in a separate document
pass="..." # provided in a separate document
path="HI-${user}"
repo_url="https://${user}:${pass}@repo.cyber.ee/sharemind/apt/${path}"

. /etc/os-release # for $UBUNTU_CODENAME.

echo "deb [arch=amd64] $repo_url $UBUNTU_CODENAME non-free" \
  | sudo tee /etc/apt/sources.list.d/sharemind-hi.list

# Resynchronize the APT sources, including the Sharemind HI repository added:
sudo apt update

# Install the Sharemind HI server:
sudo apt install sharemind-hi-server
sudo apt-mark hold sharemind-hi-server

The package contains, among other things, the sharemind-hi-server executable which runs the Sharemind HI server, a default configuration for the server at /etc/sharemind-hi/server.yaml and the sharemind-hi.service systemd service description to start the server.

Optionally, if you want to run the sample_task in the default server configuration you also need to install the sharemind-hi-sample-task-enclave package:

sudo apt install sharemind-hi-sample-task-enclave

9. Group Configuration

The user which will run the sharemind-hi-server binary needs to be inthe sgx_prv group:

sudo usermod -a -G sgx_prv <username>

10. Testing the Sharemind HI Server with the default configuration

The default configuration can be used to test the Intel® SGX and server setup. This only works if the Sharemind HI Server IP address is in the allowlist at Cybernetica to access the remote attestation proxy service (see Intel® Attestation Service). The sharemind-hi-sample-task-enclave and sharemind-hi-client packages are required. Start sharemind-hi.service through systemd (see Running the server). Create the sharemind-hi-client default configuration and run:

# Point ServerTrustRootCertificatePath to the example self-signed certificate
# shipped with sharemind-hi-server: /etc/sharemind-hi/certs/localhost.crt
sed -i'' 's_/etc/ssl/certs_/etc/sharemind-hi/certs/localhost.crt_' \
    "$HOME/.config/sharemind-hi/client.yaml"

# Explicitly perform remote attestation. During regular use, this is performed
# automatically when needed.
sharemind-hi-client \
    -c "$HOME/.config/sharemind-hi/client.yaml" \
    -a attestation

If this command succeeds, the overall setup is correct and the server state should be reset and the sharemind-hi-sample-task-enclave and sharemind-hi-client packages can be removed.

If this command fails, it may be due to one of the following reasons:

The server machine’s IP address is not in the allowlist at Cybernetica to access the remote attestation proxy service (see Intel® Attestation Service).
The machine’s BIOS is not updated ("Error: Trusted computing base out of date.").
Hyper Threading is not disabled in the BIOS ("Error: Additional configuration needed.").

The BIOS related issues can be bypassed for a non-production setup by adding / modifying the following configuration options in the "Attestation:" section in the client configuration:

# $HOME/.config/sharemind-hi/client.yaml

# ...

Attestation:
  # ...
  # DO NOT USE THESE OPTIONS IN A PRODUCTION SETUP
  AllowTCBOutOfDate: true
  AllowUnresolvedSecurityAdvisories: true

For more support please contact Cybernetica.