Benchmark PSA API and TLS throughput

[mschulz-at-hilscher]

Suggested enhancement

Currently, benchmark.c provides benchmarks for the legacy mbedtls APIs. As the default crypto API is becoming PSA based, also the PSA APIs shall be benchmarked and the results compared to the legacy mbedtls API's performance.

Additionally, there is currently no benchmark for actual end-to-end TLS throughput. It shall be benchmarked as well, not over the wire but simply using buffers to write to and read from.

In my benchmark experiments, I came to the following conclusions:

1. AES CBC ciphersuites without ETM are damn slow due to the constant time HMAC used to counter Lucky13. However, they are so slow that especially embedded systems might be stuck in doing almost nothing else than calculating HMACs. Hence, it would be good to enforce ETM on AES CBC ciphersuites as only countermeasure for Lucky13 and not support non-ETM CBC ciphersuites anymore.
2. NULL ciphersuites also use constant time HMACs, which makes them almost as slow as the AES CBC ciphersuites without ETM. However, I do not see why NULL ciphersuites would need constant time HMACs. Which attack is supposed to be countered by this countermeasure? If there is no valid reason to use constant time HMACs, use regular HMACs instead.
3. Compared to the legacy mbedtls APIs, PSA APIs are highly inefficient. On every crypto operation, key management operations are performed, for example, to lock the key slot and get the key, verify that a certain operation is allowed to be done with a key, ... Especially for short TLS records, the overhead is significant and reduces the TLS throughput. If a key in a certain keyslot is chosen to be used as TLS session key, it should be sufficient to perform all those key management checks once, when the session is setup, but during frame processing, the implementation shall stupidly just perform the crypto operations with no further overhead.
4. Using PSA APIs to perform AES GCM operations during SSL encryption and decryption, the psa_aead_encrypt function is called in mbedtls_ssl_encrypt_buf, for example. Compared to the legacy API, psa_aead_encrypt internally calls psa_aead_setup on every single frame. Which calls mbedtls_gcm_setkey that calculates H for GCM operations from the AES key. With the legacy mbedtls API, mbedtls_gcm_setkey was only called once per session and not once per frame. Hence, the use of PSA APIs significantly reduces the TLS throughput of AES GCM. Here, it would be better to cache H with the AES keys that are used for GCM operations, so that it does not need to be recalculated everytime.

Justification

Mbed TLS needs this because otherwise we cannot measure the significant performance decrease that some architectural design decisions introduce. Next step would be to create additional tickets to solve the inefficiencies.

[ronald-cron-arm]

Hi,
Thanks for your interest in Mbed TLS. We are aware of the need to benchmark the PSA APIs, although I could not find an issue tracking this work.
You mentioned having performed some benchmarking experiments on your side. Would you be willing to contribute them upstream?
Thanks.

[mschulz-at-hilscher]

In my project, I like to perform the benchmark as part of the unit tests. Once, I have a baseline, I also want to check that updates of the TLS library do not introduce performance regressions. The test suite also checks for stack and heap usage. Unfortunately, the heap operations for RSA and ECC are unnecessarily high. Especially the "resizing" of bignum allocations where calloc is used instead of realloc that first zeros the target buffer before copying data over. I already reduced the ECC heap allocations to a minimum by preallocating the bignum structures on the heap. If there is some interest, I could also create a PR for that.

Regarding the benchmark test suites, they are not finished yet, but if you want, take a look at the attached files. In the end, the same code could be added to the benchmark.c as well, which was my first approach. Let me know, if you would like to see something like this as a PR.

suites.zip

[gilles-peskine-arm]

Hence, it would be good to enforce ETM on AES CBC ciphersuites as only countermeasure for Lucky13 and not support non-ETM CBC ciphersuites anymore.

We considered that, and even removing CBC altogether. Unfortunately, a client cannot require EtM: if it advertises CBC cipher suites and EtM support, the server may pick CBC without EtM.

NULL ciphersuites also use constant time HMACs, which makes them almost as slow as the AES CBC ciphersuites without ETM. However, I do not see why NULL ciphersuites would need constant time HMACs.

There's no reason for null cipher suites to use constant-time HMAC, they just got caught in the set of cipher suites using HMAC. We tried fixing that, but it got stalled due to a lack of review bandwidth.

(Incidentally, for the past 10+ years, Mbed TLS and TF-PSA-Crypto have been maintained by Arm, plus at most one non-Arm engineer — currently one engineer from Nordic. Applications for joining the maintenance team are open. The main requirement for the job is to contribute significant and steady review bandwidth.)

On every crypto operation, key management operations are performed, for example, to lock the key slot and get the key, verify that a certain operation is allowed to be done with a key, ... Especially for short TLS records, the overhead is significant and reduces the TLS throughput.

I wouldn't have thought that the key lookup would add significant overhead unless there's contention on the lock. Do you have profiling data that shows otherwise?

Compared to the legacy API, psa_aead_encrypt internally calls psa_aead_setup on every single frame. Which calls mbedtls_gcm_setkey that calculates H for GCM operations from the AES key. With the legacy mbedtls API, mbedtls_gcm_setkey was only called once per session and not once per frame. Hence, the use of PSA APIs significantly reduces the TLS throughput of AES GCM. Here, it would be better to cache H with the AES keys that are used for GCM operations, so that it does not need to be recalculated everytime.

Having to re-prepare the key (e.g. the AES key schedule) is a significant overhead, indeed. The reason it happens is that when we started to add driver support, we decided to use the external key representation as the key representation passed to drivers. This decision was driven by architectural simplicity, and a desire to get it working first and get it fast later.

Now, we do have a drive to make it faster, by supporting a custom key representation which could mean, for example, that the AES key schedule is cached between uses. The PSA crypto driver working group is working on a specification, and it's in the plans for TF-PSA-Crypto. But so far we don't have the bandwidth to work on it.

[mschulz-at-hilscher]

If you like, I could review the PR to avoid constant-time HMACs with null cipher suites.

I wouldn't have thought that the key lookup would add significant overhead unless there's contention on the lock. Do you have profiling data that shows otherwise?

This is the throughput result for our embedded system on chip with hardware acceleration for AES GCM and ChaChaPoly. It is not fully optimized yet, but you can clearly see that the impact of using the PSA API is quite high, especially for small chunks getting processed. For 1000 byte frames for example, the throughput loss is 50 percent. I implemented the hardware drivers in the psa_crypto_driver_wrappers.h.

We have a system with crypto accelerators for symmetric ciphers directly attached to the main processor cores. So the drivers are implemented as transparent drivers. Additionally, we have a Secure Enclave with PSA Crypto API and forward asymmetric key operations there using on opaque driver. In that case, the keys are managed twice, once in the main processor and another time in the secure enclave so key slots with all the overhead are allocated twice and keys are represented by two key ids. Here, it would be better to be able to directly route requests for keys store in the secure enclave to the latter and not manage additional key slots in the main processor. To this end, one could use the key it itself to indicate the location of the key instead of placing this in the life time lifetime handled by a key slot. This would also save resources when persisting keys as only the secure enclave key slot would need to be persisted and not the additional one in the main processor as well.

Now, we do have a drive to make it faster, by supporting a custom key representation which could mean, for example, that the AES key schedule is cached between uses. The PSA crypto driver working group is working on a specification, and it's in the plans for TF-PSA-Crypto. But so far we don't have the bandwidth to work on it.

It is possible to join the working group and help with the specification of an optimized API?

[gilles-peskine-arm]

If you like, I could review the PR to avoid constant-time HMACs with null cipher suites.

At this point, the best way to move forward is if you could resubmit it for the current development branch, adapting the code and tests if necessary, and of course fixing anything that you find wrong. Then I'll be able to review it quickly and finding just one more reviewer from the team should be doable.

It is possible to join the working group and help with the specification of an optimized API?

Absolutely, yes! Your concerns are similar to others who make similar kinds of hardware, but the more vendors are involved in the discussion, the more chances we have that the result will be good for everyone. Our primary communication channel is on the TrustedFirmware Discord server and we have a regular call on alternate Mondays (I can send you the link in email or a Discord PM).