An Experimental Evaluation of TEE technology Evolution: Benchmarking Transparent Approaches based on SGX, SEV, and TDX (2024)

2.1 Process-based TEE

In Process-based TEEs, a process is split into two parts: one considered secure (trusted) and the other considered not secure (untrusted). The secure part is located in encrypted memory, managing sensitive computations, while the non-secure part communicates with the operating system and moves I/O from the encrypted memory to other parts of the system. Data transfer into and out of this secure memory zone is tightly regulated, with stringent controls over the data size and type that is allowed to cross the boundaries. Ideally, data transferred to or from the encrypted memory should be encrypted during transit and only decrypted within the TEE, ensuring that it is only accessible to the software operating within the TEE. The widely adopted Process-based TEE for server-side security is Intel Software Guard eXtension (SGX) [2].
SGX enables the creation of secure enclaves within the processor, isolating sensitive code and data from the rest of the system thanks to the extension of Intel’s Instruction Set Architecture (ISA) by 18 new instructions. Sensitive code and data are stored in the Enclave Page Cache (EPC), a specific 128 MiB / 256 MiB (for SGX v1 or v2) section of memory set aside during the system startup for storing the code and data of enclaves. Any attempt to access an enclave’s page outside of the EPC results in a page fault. The SGX driver collaborates with the CPU to determine which pages to remove from the cache. The memory encryption engine (MEE) ensures that communication between the CPU and system memory remains secure, and it is also responsible for preventing tampering and providing replay protection. An enclave can only run in user mode (ring3) since the host OS is considered untrusted. This means that system calls cannot be invoked from inside the TEE. The only way to execute them is through well-defined interface calls outside the enclave. An important SGX feature for verifying the integrity and security of an enclave is the attestation mechanism. SGX supports a local attestation, used for communication between enclaves on the same platform, and a remote attestation used for demonstrating trustworthiness to external entities.

Developing applications for SGX can be challenging because the application must be refactored into trusted (within the enclave) and untrusted components. This porting requires careful design to ensure security and can be a complex and time-consuming process.This is where SGX runtimes came to the rescue. They act as intermediate layers that abstract away the complexity of SGX and allow developers to run applications within SGX enclaves with much fewer changes.By using these runtime environments, developers can more easily take advantage of the security benefits of SGX, allowing sensitive or critical applications to be deployed in potentially untrusted environments.
The common approach adopted by these runtimes to enable quasi-transparent porting is to execute system calls directly inside the enclave via a Library OS (LibOS), i.e., a new paradigm trend where kernel functions are available to user space (ring3) programs in a form of a library.There are several runtime environments for Intel SGX (e.g. SCONE, Gramine, Occlum, SGX-LKL).In this work, we focus on Gramine and Occlum because they stand out among the open-source solutions as the widely adopted ones [14].

2.1.2 Occlum-SGX

Occlum [4], while sharing similarities with Gramine as a runtime environment, owes its spread and adoption to its strong community support. Being an open-source project, Occlum benefits from contributions from a range of developers, which helps in its development and maintenance. This community involvement can be important for ensuring the tool stays updated and relevant.This system introduces Software Fault Isolation-Isolated Processes (SIPs) within a LibOS in an enclave’s single address space.Software Fault Isolation (SFI) is a technique for sandboxing untrusted modules in different domains.The novel aspect of this proposal is the Memory Protection Extensions-based (MPX), Multi-Domain SFI (MMDSFI), which supports an unlimited number of domains without restrictions on their addresses and sizes.This allows for enhanced intra-enclave isolation, including isolation between processes and between a process and the LibOS. To ensure the security and compliance of these isolation mechanisms, an independent binary verifier called the Occlum verifier is introduced. This verifier statically checks ELF binaries to ensure they adhere to MMDSFI’s security policies.

2.2 VM-based TEE

VM-based TEEs foresee that an entire VM memory is encrypted using keys sealed in the hardware, which prevent interference by a malicious VMM. Current technologies such as Intel TDX and AMD SEV provide dedicated ephemeral encryption keys for each VM, thus also protecting the VMs from each other.

2.3 Motivation

4.1 Environment

Figure 3 shows our experimental setup. In order to get results as comparable as possible, we configured our machines hosting the workloads with the same amount of virtual CPUs (vCPUs) cores (4 cores), the same amount of virtual RAM (vRAM) (16GB), and similar NIC characteristics.
If the workload requires to be stimulated in a client/server topology, there is the need for an external benchmark client to do requests to the server. In this case, in order to prevent interference, a separate VM was deployed. Low-latency channels between the client and server are important to get fair results. Hence, the benchmark VM was deployed in the same Availability Zone as the workload VM.
At the time of writing this paper, there is no public availability of an Intel TDX machine. Experiments on this technology were conducted using a server offered by Intel for research purposes, which mounts an Intel Xeon Platinum 8480CTDX ($2.0GHz$, Turbo at $3.8GHz$). Well-known virtualization tools like QEMU and libvirt are needed for Intel TDX to enhance the confidentiality of active workloads. For the effective operation of a confidential VM, various elements within the virtualization stack must be compatible with TDX hardware. Intel is actively engaged in integrating comprehensive TDX software support into the upstream versions of the Linux kernel, QEMU, and libvirt. We leveraged the patched versions of Linux kernel, QEMU, and libvirt to deploy a guest TDX VM. Even in this case, the same amount of cores and memory are configured on the mounted QEMU Confidential VM.
The TDX machine embeds the SGX extension as well. So, we decided to run SGX (i.e., Gramine and Occlum) and Native runs – useful to set the baseline for the comparison – in the TDX server. In this way, we can provide a fair comparison of Native, TDX, and SGX.At the same time, we also want to provide information on how SGX performance compares between the old and new generations of CPUs. Hence, we instantiated a Standard DCsv3-series VM, which uses the 3rd Generation Intel Xeon Scalable (Icelake) 8370C ($2.9GHz$, Turbo at $3.5GHz$) with the SGXv2 capabilities (i.e., larger EPC and support for dynamic memory allocation (EDMM)).
Experiments on AMD SEV leveraged a Standard DCasv5-series VM, which uses AMD’s third-Generation EPYCTM 7763v processor ($2.5GHz$, Turbo at $3.5GHz$). We selected this server because it is one of the best-performing SEV-enabled AMD machine, which is comparable with the Intel machine.
In terms of software, all VMs were configured with Ubuntu 22.04.3 LTS. We use the latest versions of Gramine-SGX (v1.6), and Occlum (v0.30.0). The SGX SDK software stack relies on version 2.22.

4.2 Managing CPUs with different Clock Frequencies

In our experimental setup, we make an identical configuration of vCPUs and vRAM across the different nodes. However, there are still some factors that could potentially skew the fairness of our results such as the range of CPU clock frequencies. The clock frequency varies slightly among the nodes we selected for our experiments. This variation is significant enough that it must be considered when analyzing our experimental results to ensure accuracy and fairness.
To address this potential discrepancy, we adopt a method of normalization for our calculated overheads. This normalization process involves using microbenchmark results that are publicly available for the specific CPUs used in our servers [15][16][17]. These benchmarks provide a detailed analysis of the performance capabilities of these CPUs under various conditions. By incorporating these benchmark results into our analysis, we can adjust our data to account for the differences in clock frequency. This normalization allows us to compare performance metrics more accurately across different hardware setups. It ensures that any observed differences in the experimental results are due to the factors we are testing, rather than inherent differences in the hardware’s basic processing speed. In essence, this approach helps us isolate the variables we are interested in studying, by mitigating the impact of an external variable – the clock frequency range – that could otherwise introduce an element of unfairness into our results. This careful consideration and adjustment for hardware differences underscore the rigor and precision we are applying in our experimental analysis.

4.3 Workloads

To ensure that our evaluation of Trusted Execution Environment (TEE) approaches is thorough and reflective of real-world scenarios, it is crucial to test them across a diverse range of workloads. These workloads should mirror the variety of applications commonly used in practice, each with its unique resource utilization characteristics. We have carefully selected several prevalent workloads, categorized based on their primary resource demands: CPU-intensive, memory-intensive, and I/O-intensive.

CPU-Intensive Workloads.For tasks that demand substantial CPU resources, particularly for computation-intensive processes, we selected the following workloads:

Memory-intensive Workloads. For workloads that predominantly consume memory resources, we included:

I/O-intensive Workloads.To evaluate the performance under I/O stress, we selected:

For each of these workloads, we conducted multiple runs to ensure reliability and accuracy in our results. Specifically, we set a confidence interval of $95\%$ and empirically verified that 10 repetitions of our experiments were enough to achieve the aforementioned target. The outcomes were averaged to account for any anomalies and to provide a more accurate representation of the performance. This rigorous testing methodology allows us to comprehensively assess the effectiveness of TEE approaches across a spectrum of real-world applications, ensuring that our findings are both valuable and applicable to a wide range of scenarios.

4.4 Benchmarks

We employed a variety of benchmarking tools to stimulate the different workloads. It is important to notice that their final configuration — e.g., the number of connections, and the ranges of parallel clients — was obtained empirically after several preliminary experiments aimed at reaching the workload saturation point.

Redis. We used redis-benchmark, a tool specifically designed for the REDIS key-value store. It is used to measure the performance of a Redis server by running a series of predefined tests. In our study, we used redis-benchmark to make typical operations of SET and GET, thus evaluating the throughput and latency of the Redis server during writing and reading from memory. In terms of configuration, we kept the total number of connections to a fixed value of $100k$ and varied the number of parallel clients from $10$ to $1000$.

NGINX & NodeJS. The wrk2 benchmark was adopted to generate a significant load against NGINX and NodeJS. It provides a flexible scripting interface that allows us to simulate different types of HTTP requests, which is crucial for testing the performance of NGINX as a web server and NodeJS in server-side scripting scenarios. By adjusting the number of connections, threads, and the duration of the test, we were able to assess how these servers handle high traffic and concurrent connections. The benchmark was configured with a duration of $30s$, a fixed number of connections to $100$, and a varying number of clients ranging from $10$ to $16000$.

Vault. The vault-benchmark tool — specifically designed for Hashicorp Vault — helps in evaluating the performance of Vault in various scenarios, including reading and writing secrets, authentication requests, and other secret management operations. We used the vault-benchmark to determine the throughput and response times of Vault during static_secret_writes operations, which is critical for understanding its scalability and reliability in a production environment. The benchmark was tuned with $numkvs=100$, and a $kvsize$ varying in the range $[10,800]$.

TensorFlow. Even in this case, we used built-in models to evaluate performance:

5.1 Memory-Intensive Workloads

Figure 4 shows the graph on Redis performance. The $x$-axis refers to the throughput and the $y$-axis to the latency. As expected, Gramine (yellow line with ’X’ marker) and Occlum (blue line with star marker) have the worst performance. It can be observed they reported the highest latency and low throughput, although Gramine performs better than Occlum, which experiences a sharp peak, and then a rapid decrease, suggesting inefficiency, especially at lower throughputs. The TDX solution (red line with square marker) has surprisingly the highest throughput and a latency that is comparable with the one observed on the Native (black line with circle marker). The Native solution has a steady increase in latency with high throughput. Furthermore, we can notice that Native and TDX have a similar trend.SEV (green line with triangle marker) has higher latency at the beginning, which sharply increases with a small increase in throughput, indicating a potential bottleneck in handling higher loads. However, it is important to notice that the difference in performance between TDX and SEV depends on two factors: the CPU typology and the security technology itself. The impact of the CPU typology can be obtained using publicly available benchmarks [15][17], which tell us that the AMD CPU is on average $40.7\%$ slower than the Intel CPU, which hosted the execution of all the other versions of the workloads (Native, TDX, Gramine, Occlum). If we subtract the $40.7\%$ we can argue that the actual overhead of SEV with respect to TDX can be considered of $\approx 22\%$.
In Figure 5, we report the Vault performance. The rightest points correspond to the lowest kvsize. The increase of kvsize causes the decrease of the throughput and a rise in the latency. Even in this case, we observed that TDX provides the highest throughput. It is interesting to notice that SEV has a throughput similar to the Gramine and Native solutions but at the same time, it has the lowest latency. A different story is Occlum, which experienced very bad performance as can be noticed by the graph highlighting the high latency and the low throughput.

5.2 CPU-Intensive Workloads

Figures 6 and 7 show bar graphs depicting the inference time of PyTorch and TensorFlow workloads, respectively. On the $x$-axis, we report the different benchmarks used for the evaluation.
For what concerns PyTorch, as expected the Native environment consistently exhibits the shortest inference times across all benchmarks. Gramine-SGX performs exceptionally well, often approaching the performance of the Native environment. Unlike the other evaluations, Gramine-SGX behaves better than TDX across all benchmarks. Occlum also exhibits better performance than usual, which is on the same level as TDX. SEV demonstrates significantly longer execution times, with approximately $40\%$ of this increase attributed to inherent CPU characteristic differences.
In the case of TensorFlow, the situation is different. TDX experienced a low inference time, which sometimes is even better than the Native one. While Gramine-SGX and SEV reported a slightly higher execution time. Occlum was the worst one with a higher inference time, up to $6\times$.
What becomes evident from these findings is that Process-based TEEs tend to experience only minor performance degradation when subjected to CPU-intensive workloads. In certain instances, their performance closely matches that of VM-based TEEs, while in others, such as with Gramine-SGX, they even surpass VM-based TEEs in terms of performance.

5.3 I/O-Intensive Workloads

Figure 8 shows the performance of NGINX. The Native has the highest throughput. The latency remains the lowest across all throughputs. TDX shows a significant drop in throughput compared to Native, stabilizing just above 0.4. The latency is low when the throughput is below this point but rises sharply as the throughput increases. The average overhead of TDX with respect to the Native is $28.6\%$. SEV has a lower maximum throughput than TDX. The latency is in the same range of TDX at low throughputs but does not increase as sharply. Again, if we remove the impact of the AMD CPU, we can consider SEV $8.7\%$ slower than TDX. Gramine-SGX exhibits a similar pattern to SEV, but with slightly higher latency across all throughput levels. Even in this case, Gramine-SGX behaved better than Occlum-SGX but still slower than the VM-based TEEs, reporting an overhead of $67.6\%$.
Figure 9 compares the performance of the NodeJS workload. The Native has the best performance as before, showing high throughput and low latency. While TDX has a slightly higher latency than SEV.It is important to highlight that, for this particular workload, both Gramine-SGX and Occlum-SGX were not able to sustain the same amount of concurrent benchmark requests used for TDX, SEV, and Native (in the range of $[10,8000]$). The webserver stuck after $500$ requests. Hence, a smaller range of requests has been used (i.e., $[10,500]$). This explains why the latency of Gramine-SGX and Occlum-SGX is extremely low compared to the other approaches.

5.4 Resource Usage

Besides evaluating the speed of our workloads, we are also interested in examining the CPU utilization across the various TEE approaches to better understand the trade-offs between security and computational efficiency.
In Figure 10 we show the average CPU usage for the different workloads. As expected, the Native runs exhibit the lowest CPU usage for all applications. TDX shows competitive CPU usage, close to Native levels in most applications. AMD’s SEV consistently shows higher CPU usage compared to TDX and sometimes even higher than Gramine and Occlum. Both process-based TEEs show variable CPU usage across different applications. Gramine generally maintains moderate CPU usage, indicating a balanced performance. Occlum, however, has notably high CPU usage with PyTorch, which could be indicative of less efficient handling of CPU-intensive workloads or a lack of optimization for this particular application. For Redis and NodeJS, all TEEs have relatively similar CPU usage, with TDX marginally outperforming the others. This suggests that for certain types of workloads, the choice of TEE might not significantly impact CPU efficiency. The PyTorch workload stands out with its high CPU usage in the Native environment, while TDX optimizes this usage considerably. Occlum, however, seems to struggle with this workload. TDX appears to be the most efficient in terms of CPU usage across a range of applications, closely followed by Gramine. SEV tends to have higher CPU overhead, while Occlum’s performance is highly variable, performing well in some cases but not in others.