Sharemind HI As Data Analysis Platform

The tutorial will create an application that performs privacy preserving set intersection using a sample scenario. The sample scenario models a situation where a number of input parties each have a set of sensitive elements they do not wish to disclose in full to other parties, but wish to find elements that are present in each set, i.e. to find the set intersection of the inputs. As an example this can abstract a conglomerate of companies trying to identify joint clients without disclosing their full client list to the other companies, or financial institutions trying to find common potentially fraudulent accounts without disclosing all of their high-risk accounts.

To keep the focus on operations with Sharemind HI, the tutorial will further constrain the scenario to only two parties providing inputs and an additional party receiving an output. A single task enclave will be used to implement the set intersection, and the inputs and output are assumed to be proper sorted sets (with no duplicate elements) formatted as 32-bit signed integers.

The provided example DFC already defines three stakeholders, sh1, sh2, and sh3, with test certificates.

These test certificates are allowed all possible roles encoded in them and will initially work for getting the rest of project ready. The actual certificates should be generated and distributed to the stakeholders in the deployment step.

As the sample scenario includes exactly three input parties, the existing stakeholders can be renamed to better fit the scenario, for example to input_provider1, input_provider2, and output_consumer.

For other use-cases a number of pre-made test certificates and their corresponding recovery key pairs are located in the Sharemind HI Development bundle: /path/to/HI-bundle/lib/cmake/sharemind-hi/task-enclave-project-default-files/ca_stakeholders/. Additional certificates can be generated by following the Certificate Setup page.

Now the Stakeholders section of the DFC should look as follows:

The defined stakeholders also need to be assigned roles. The example DFC has listed the three input parties as auditors and only the first stakeholder as an enforcer.

The exact role assignments are highly dependent on the specific use-case and the capabilities of the involved parties. As such the assignments need to be analyzed thoroughly before a real deployment. For this tutorial we can keep the default assignments, however, as the stakeholders were renamed the change should also be reflected in the role assignments. The role assignment section of the DFC should look as follows:

The next section in the DFC defines all task enclaves included in the service. Defining a new task enclave requires specifying an unique name, the fingerprints of the built and signed enclave, a list of stakeholders that can run the enclave, and, optionally, how long the information about the task can be retained.

The sample scenario only requires a single task enclave to run the set intersection algorithm, so we will edit the existing definition. The name and fingerprint entries of the enclave can be kept as is. During development of the enclave the fingerprints are filled automatically by the test scripts. The fingerprints should be fixed in place during the deployment step, once the enclave has been fully tested and audited.

As for the Auditor and Enforcer roles, assigning the Runner role requires analyzing the project needs and specific use case. Any of the involved stakeholders, or a combination of them, could be assigned as runners, or an entirely new party could be added to run the task. For the sample scenario we will assign the output_consumer stakeholder as the sole Runner.

Lastly, the data retention time for the enclave information can be set. The data retention time is counted from the moment of starting the enclave, and all data related to the the enclave (who and when ran the enclave) will be deleted. It is important to consider local regulations related to data processing and data retention when choosing an appropriate data retention time. If data retention is not required and data can be stored indefinitely, then the DataRetentionTime entry can be omitted entirely. As the sample scenario does not require any data retention, the entry can be removed.

The Tasks section of the DFC should now look as follows:

Lastly the DFC defines all data topics used in the service, along with access rights for each topic. Defining a topic requires specifying a unique name for the topic, a list of stakeholders and tasks that are allowed to upload data to the topic, a list of stakeholders and task that are allowed to download data from the topic, and, optionally, the data retention time similar to the task enclaves.

The test DFC uses three input and three output topics, however, the sample scenario only requires a single input topic and a single output topic. Hence we can delete the definitions for topics input2, input3, output2, and output3, and keep the definitions for topics input1 and output1. The remaining topics can also be renamed if needed, however the names input and output fit the sample scenario nicely.

To allow the input parties to upload data into the service, they need to be listed as Producers for a topic. Similarly, to allow the receiving party to download the results, they need to be listed as a Consumer for a topic. As the output_consumer should not be able to access the inputs, it is important that they are not listed as a consumer in the input topic, but only in the output topic. In order for the task enclave to process the inputs and generate an output, it has to be allowed to download data in the input topic, and to upload data to the output topic.

Similarly to the task enclave definition, the sample scenario does not require any data retention, so any DataRetentionTime entries can be removed.

The final DFC should look as follows:

Modifying the test/client.sh script to match the sample scenario requires

For simplicity, we will assume that the inputs and the output are proper sorted sets formatted as 32-bit integers. Perl’s pack function can be used to write both the inputs and the expected output into temporary files looks as follows:

We can also update the correctness testing at the end of the file:

Performing the attestation and the enforcer approving the DFC are important steps in ensuring the security of the final solution. As the sample scenario uses the same stakeholders and roles as the template, no changes are required in these steps to fit the new use case.

The next change has to be done in the data upload step, where the last stakeholder is no longer an input party. The data upload operations for the first two stakeholders can remain as is, and the upload operation for the third party should be removed leaving the sections as follows:

The taskRun action can only be successfully called by the output_consumer stakeholder, as they are the only stakeholder listed as the Runner of the task in the DFC. Similarly, with the dataDownload action, only the output_consumer stakeholder is allowed to download data from the output. Updating the task run and download steps are the last changes needed and leaves the whole test as follows:

At this point the test will still fail, however it now fails at the final step when checking the correctness of the output. As the test enclave simply copies any data in the input topic to the output topic, the output will be equal to the first data item uploaded. To make the test succeed, the set intersection algorithm has to be implemented in the task enclave.

While the actual process of setting up, starting, and tearing down the secure enclave is long and complex, Sharemind HI hides and abstracts most of it from the task enclave developer. The only part of the task enclave lifecycle that the developer needs to participate in, is implementing the data processing performed inside the enclave. The data processing flow inside the enclave only has a single entrypoint, the run(TaskInputs const & inputs, TaskOutputs & outputs) method in src\enclave\Enclave.cpp. From the perspective of the task enclave developer, this is the equivalent to the "main" function of the enclave.

The run method is invoked asynchronously by the taskRun action from the client libraries and provides the developer access to all of the input and output topics available to the enclave. Everything related to how the data is parsed, transformed, or manipulated in between the input and output is left for the developer.

The test task enclave provides a number of example operations that can be performed in the enclave, including file operations, working with the inputs and outputs, accessing the DFC, and using the streams header. It is recommended to work through the examples before writing your own code as it can help introduce the tools provided by the Sharemind HI SDK.

Access to the input and output topics are provided through the TaskInputs and TaskOutputs classes defined in the sharemind-hi/enclave/task/TaskIO.h header (all available headers are located in HI-bundle/include/sharemind-hi/sharemind-hi/). The encrypted inputs in a topic can be accessed with inputs.topic(topicName).data or alternatively it is possible to access a single data item using it’s data ID with inputs.get(topicName, DataId). The encrypted inputs are then decrypted using encrypted_input.decrypt(inputPlaintext).

Writing data to a topic is done with outputs.put(topicName, data, dataSize). Note that it is only possible to write to the end of an output topic, and the data is encrypted automatically before writing.

Once familiar with the provided test code, the content of run method can be cleared, and replaced with the implementation of the set intersection algorithm. Using the constructs and examples in the test code, the initial attempt may look something like this:

This implementation fits the sample scenario and passes the test written in the previous section. However, juggling the data items and decryption distracts from the logic of the algorithm. Sharemind HI offers a number of useful headers and pre-made functions to ease the development process. One of the more useful headers is the Streams.h library.

Using large amounts of memory inside an enclave can occur significant performance penalties and as such it is important to keep the memory footprint in the enclave as low as possible. Utilizing streams is a natural way to process elements as they come in and not store entire inputs in the enclave memory. The streams library provides a convenient way to read the data coming from the input topics and to write data to the output topics. Additionally there are implementations for a number of operations common in data processing workflows, allowing the developer to use similar constructs as in other languages and environments.

Using the tools provided in the streams header allows to replace the decryption and formatting of each input with a single line: stream::into<int32_t>(inputs, "input"). The into function takes the last data item in the specified topic, and generates elements matching the template type (in this case int32_t). The library also provides a number of other functions to create stream sources, like vec to create a source from a std::vector, and range to create a source of sequential integers.

Elements generated by the source can be forwarded, using the pipe operator >>=, to intermediate "pipes" or stream terminating "sinks". Pipes process the incoming elements to generate new ones that can be again forwarded to new stream operations. Examples of pipes are filter, which can filter out elements based on a custom predicate, smap, which can apply a custom function to each element to transform them, and join, which joins two sorted streams into a single sorted stream of the common elements.

Sinks consume any elements and terminate the stream to produce a single output. Examples of sinks are foreach, which similarly to smap applies a custom function to each input, collect, which gathers all incoming elements into a std::vector, and encryptedOutput, which stores all incoming elements directly in an output topic.

The set intersection problem is perfectly solved by the join method, as it can reduce two input streams into a stream of only the common elements. Turning inputs into stream sources is best done using the into method. However, the into method only retrieves the last item in a topic. While a more complex approach can be taken to create a source from each item in a simple topic, a simpler method of adding a second input topic suits our scenario. Adding a new topic requires updating the DFC:

The test also needs to be updated to reflect the new topics:

Now all that is left is to join the two inputs and write the result to the output:

The result is a significantly more compact and readable code, that is more efficient and better suited to the enclave environment.