Introduction

We are devoted to keeping the mbed TLS code base secure and put a lot of effort in our Quality Assurance and testing of the code. This is an ever increasing process, so this article will be adapted over time to reflect this!

Types of testing

So, what types of testing does mbed TLS provide?

We automatically check:

  • Compilation on different systems
  • Code coverage
  • Regressions, test vectors, corner cases
  • Interoperability with other SSL libraries
  • Different build configurations
  • Memory leaks
  • Memory integrity (bounds, initialization)

mbed TLS uses a Buildbot environment to run all these tests on a number of different environments.

Test details

Test Framework

There are currently over 6000 automated test vectors that are run in the tests/ directory when make check is run. With CMake, make memcheck runs these tests under Valgrind.

For running these test we have our own test framework that combines 'generic' test functions with specific test values. The build system generates the parsing test applications, such as test_suite_aes.ecb and then runs them with the different test case input available. The system keeps in mind the current configuration from config.h and makes sure only relevant tests are run.

A sample output for some tests looks like this:

$ ./test_suite_ecb
AES-128-ECB Encrypt NIST KAT #1 ................................... PASS
AES-128-ECB Encrypt NIST KAT #2 ................................... PASS
AES-128-ECB Encrypt NIST KAT #3 ................................... PASS
AES-128-ECB Encrypt NIST KAT #4 ................................... PASS
AES-128-ECB Encrypt NIST KAT #5 ................................... PASS
<snip>
AES-256-ECB Decrypt NIST KAT #10 .................................. PASS
AES-256-ECB Decrypt NIST KAT #11 .................................. PASS
AES-256-ECB Decrypt NIST KAT #12 .................................. PASS

----------------------------------------------------------------------------

PASSED (77 / 77 tests (0 skipped))

Interoperability testing

Because our test vectors can only test individual functions, we also run interoperability tests to check live SSL connections against OpenSSL and GnuTLS.

The test script tests/compat.sh checks each available ciphersuite in the current build as a server and a client against mbed TLS, OpenSSL and GnuTLS (when available). These tests are run for each protocol version, with and without client authentication. In total over 1600 combinations are tested.

And depending on the settings, memory leaks are checked for automatically as well.

A sample output for some tests looks like this:

$ ./compat.sh
<snip>
P->P ssl3,no TLS-PSK-WITH-RC4-128-SHA .................................. PASS
P->P ssl3,no TLS-PSK-WITH-3DES-EDE-CBC-SHA ............................. PASS
P->P ssl3,no TLS-PSK-WITH-AES-128-CBC-SHA .............................. PASS
P->P ssl3,no TLS-PSK-WITH-AES-256-CBC-SHA .............................. PASS
P->P ssl3,no TLS-DHE-PSK-WITH-RC4-128-SHA .............................. PASS
P->P ssl3,no TLS-RSA-PSK-WITH-RC4-128-SHA .............................. PASS
P->O tls1,no TLS-ECDHE-ECDSA-WITH-RC4-128-SHA .......................... PASS
P->O tls1,no TLS-ECDHE-ECDSA-WITH-3DES-EDE-CBC-SHA ..................... PASS
P->O tls1,no TLS-ECDHE-ECDSA-WITH-AES-128-CBC-SHA ...................... PASS
P->O tls1,no TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA ...................... PASS
P->O tls1,no TLS-ECDH-ECDSA-WITH-RC4-128-SHA ........................... PASS
P->O tls1,no TLS-ECDH-ECDSA-WITH-3DES-EDE-CBC-SHA ...................... PASS
<snip>
------------------------------------------------------------------------
PASSED (2143 / 2143 tests (0 skipped))

Option testing

In addition to standard interoperability with other libraries, different behaviors that should occur during a handshake or afterwards are tested for. For this we use the script tests/ssl-opt.sh.

A sample output for some tests looks like this:

$ ./ssl-opt.sh
<snip>
Session resume using cache #8 (openssl client) ......................... PASS
Session resume using cache #9 (openssl server) ......................... PASS
Max fragment length #1 ................................................. PASS
<snip>
Renegotiation #0 (none) ................................................ PASS
Renegotiation #1 (enabled, client-initiated) ........................... PASS
<snip>
Authentication #1 (server badcert, client required) .................... PASS
Authentication #2 (server badcert, client optional) .................... PASS
<snip>
Authentication #9 (client no cert, openssl server optional) ............ PASS
Authentication #10 (client no cert, ssl3) .............................. PASS
SNI #0 (no SNI callback) ............................................... PASS
SNI #1 (matching cert 1) ............................................... PASS
<snip>
DTLS reassembly: fragmentation, nbio (openssl server) .................. PASS
DTLS proxy: reference .................................................. PASS
DTLS proxy: duplicate every packet ..................................... PASS
DTLS proxy: duplicate every packet, server anti-replay off ............. PASS
DTLS proxy: inject invalid AD record, default badmac_limit ............. PASS
DTLS proxy: inject invalid AD record, badmac_limit 1 ................... PASS
DTLS proxy: inject invalid AD record, badmac_limit 2 ................... PASS
DTLS proxy: inject invalid AD record, badmac_limit 2, exchanges 2 ...... PASS
DTLS proxy: delay ChangeCipherSpec ..................................... PASS
DTLS proxy: 3d (drop, delay, duplicate), "short" PSK handshake ......... PASS
DTLS proxy: 3d, "short" RSA handshake .................................. PASS
<snip>
------------------------------------------------------------------------
PASSED (269 / 269 tests (0 skipped))

Different build configurations

The script tests/scripts/test-ref-configs.pl builds and checks different build configurations.

The current configurations tested are:

  • A minimal TLS 1.1 configuration (configs/config-mini-tls1_1.h)
  • A Suite B compatible configuration (configs/config-suite-b.h)
  • A minimal "modern" PSK configuration using TLS 1.2 and AES-CCM-8 (configs/config-ccm-psk-tls1_2.h)
  • A configuration without the SSL modules, as used by Picocoin (configs/config-picocoin.h)

Memory checks

Both automatic and manual tests are run with the Valgrind memory checker and with various sanitizers from GCC and Clang: ASan for memory bounds checking, MemSan to detect uninitialised memory issues, and UBSan to detect undefined behaviors according to the C standard.

Test systems

At the moment we test on the following Operating Systems:

  • Debian with Intel 32-bit, Intel 64-bit, and ARM 32-bit
  • FreeBSD i386
  • Win32 32-bit and 64-bit

With a mix of the following compilers / IDE environments:

  • GCC (with CMake and make)
  • Clang
  • MinGW32
  • MS Visual Studio 12
  • armcc

Static analysis

The mbed TLS source code is automatically checked with the Coverity static analysis scanner for security issues before every new release. You can find more details on the mbed TLS page on Coverity.

In addition, we manually use Clang's static analyzer and more recently Infer on our codebase.

Fuzzing

We use Codenomicon Defensics for fuzzing of our (D)TLS server (all versions) and our X.509 code before every new release.

Did this help?