RFC compliance issues in TLS 1.3 alert handling and post-handshake behavior

## Overview

This issue aggregates eight RFC-compliance / interoperability problems that were revalidated against Mbed TLS `v4.1.0-92-g45ad2fc5ee` (commit `45ad2fc5ee6e5ca2a7236f44e88275038d870464`).

These appear to be protocol-conformance defects rather than security vulnerabilities, so I am reporting them together in one public issue. Each individual problem is summarized below with `Summary`, `Expected behavior`, and `Actual behavior`. Detailed system information, reproduction steps, and additional notes are available in the accompanying PoC bundle's per-issue `report.md` files.

## Included issues

1. Post-handshake `ClientHello` / `CertificateRequest` handling does not emit the required fatal `unexpected_message` alert.
2. An empty server `Certificate` message triggers the wrong alert.
3. Oversized records do not emit the required `record_overflow` alert.
4. Two malformed `TLSInnerPlaintext` cases do not emit the required `unexpected_message` alert.
5. A protected `ChangeCipherSpec` record is ignored instead of rejected.
6. Further I/O remains possible after receiving a fatal alert.
7. An MD5-signed certificate triggers `certificate_unknown` instead of `bad_certificate`.
8. TLS 1.3 `KeyUpdate` is not implemented / handled.

## PoC / reproduction materials

The accompanying PoC bundle can be attached as a single archive, with one subdirectory per issue. The corresponding per-issue `report.md` files in that bundle contain the detailed system information, reproduction steps, and additional notes for each case.

## Issue 1: Post-handshake ClientHello / CertificateRequest does not trigger fatal unexpected_message


### Summary

On an established TLS 1.3 connection, receiving a handshake message out of
context (e.g. a post-handshake `ClientHello` from a client, or a
`CertificateRequest` when the client did not send the `post_handshake_auth`
extension) returns `MBEDTLS_ERR_SSL_UNEXPECTED_MESSAGE` to the application but
never sends the `unexpected_message` fatal alert that RFC 8446 requires.

### Expected behavior

RFC 8446 requires the connection to be terminated with an `unexpected_message`
**fatal alert** in two post-handshake situations:

> "Because TLS 1.3 forbids renegotiation, if a server has negotiated TLS 1.3
> and receives a ClientHello at any other time, it MUST terminate the connection
> with an 'unexpected_message' alert."
> — Section 4.1.2

> "A client that receives a CertificateRequest message without having sent the
> 'post_handshake_auth' extension MUST send an 'unexpected_message' fatal alert."
> — Section 4.6.2

That is, the peer must see a fatal `unexpected_message` alert
(`MBEDTLS_SSL_ALERT_MSG_UNEXPECTED_MESSAGE`, alert code 10) on the wire before
the connection is torn down.

### Actual behavior

No alert is transmitted. When `mbedtls_ssl_read()` pulls a handshake record on an
established connection, the dispatch reaches
`ssl_tls13_handle_hs_message_post_handshake()` in `library/ssl_msg.c`. This
handler only accepts `NewSessionTicket` for clients; every other message falls
through to a catch-all that returns the error code directly — without calling
`MBEDTLS_SSL_PEND_FATAL_ALERT()` or `mbedtls_ssl_send_alert_message()`:

```c
// library/ssl_msg.c:5459-5485
static int ssl_tls13_handle_hs_message_post_handshake(mbedtls_ssl_context *ssl)
{
    MBEDTLS_SSL_DEBUG_MSG(3, ("received post-handshake message"));

#if defined(MBEDTLS_SSL_CLI_C)
    if (ssl->conf->endpoint == MBEDTLS_SSL_IS_CLIENT) {
        if (ssl_tls13_is_new_session_ticket(ssl)) {
            /* ... NewSessionTicket handling ... */
        }
    }
#else
    (void) ssl;
#endif /* MBEDTLS_SSL_CLI_C */

    /* Fail in all other cases. */
    return MBEDTLS_ERR_SSL_UNEXPECTED_MESSAGE;   // no PEND_FATAL_ALERT called
}
```

The caller in `mbedtls_ssl_read()` simply returns the error to the application,
again without sending any alert:

```c
// library/ssl_msg.c:5742-5748
        if (ssl->in_msgtype == MBEDTLS_SSL_MSG_HANDSHAKE) {
            ret = ssl_handle_hs_message_post_handshake(ssl);
            if (ret != 0) {
                MBEDTLS_SSL_DEBUG_RET(1, "ssl_handle_hs_message_post_handshake",
                                      ret);
                return ret;   // error code returned to application, no alert sent
            }
```

For comparison, equivalent unexpected-message conditions encountered *during* the
active handshake do pend a fatal alert — e.g. the generic handshake fetcher in
`ssl_tls13_generic.c` and the duplicate-HelloRetryRequest handler in
`ssl_tls13_client.c`. The post-handshake path is the exception.

## Issue 2: Empty server Certificate triggers the wrong alert


### Summary

When a TLS 1.3 server sends an empty `Certificate` message (empty
`certificate_list`), the client aborts the handshake with the wrong alert: it
sends `certificate_required` / `NO_CERT` (alert 41) instead of the `decode_error`
alert (alert 50) that RFC 8446 §4.4.2.4 requires for an empty *server*
certificate list.

### Expected behavior

RFC 8446 distinguishes an empty `Certificate` from the server (always a message
format error) from an empty `Certificate` from the client (an authentication
choice):

> "If the server supplies an empty Certificate message, the client MUST abort
> the handshake with a 'decode_error' alert."
> — Section 4.4.2.4

> "The server's certificate_list MUST always be non-empty."
> — Section 4.4.2.4

So the alert for an empty server `certificate_list` must be `decode_error`
(alert 50), not `certificate_required`/`NO_CERT` (alert 41).

### Actual behavior

`ssl_tls13_validate_certificate()` in `library/ssl_tls13_generic.c` handles the
`peer_cert == NULL` case that results from an empty server `certificate_list`.
The code comment correctly cites the RFC requirement that the server's
`certificate_list` MUST always be non-empty, but the alert it pends is
`MBEDTLS_SSL_ALERT_MSG_NO_CERT` (alert 41) rather than
`MBEDTLS_SSL_ALERT_MSG_DECODE_ERROR` (alert 50):

```c
// library/ssl_tls13_generic.c:657-671
#if defined(MBEDTLS_SSL_CLI_C)
        /* Regardless of authmode, the server is not allowed to send an empty
         * certificate chain. (Last paragraph before 4.4.2.1 in RFC 8446: "The
         * server's certificate_list MUST always be non-empty.") With authmode
         * optional/none, we continue the handshake if we can't validate the
         * server's cert, but we still break it if no certificate was sent. */
        if (ssl->conf->endpoint == MBEDTLS_SSL_IS_CLIENT) {
            MBEDTLS_SSL_PEND_FATAL_ALERT(MBEDTLS_SSL_ALERT_MSG_NO_CERT,
                                         MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE);
            return MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE;
        }
#endif /* MBEDTLS_SSL_CLI_C */
```

The relevant alert constants in `include/mbedtls/ssl.h`:

```c
#define MBEDTLS_SSL_ALERT_MSG_NO_CERT               41  /* 0x29 */   // sent today
#define MBEDTLS_SSL_ALERT_MSG_DECODE_ERROR          50  /* 0x32 */   // RFC-required
```

This appears to reuse the client-side empty-certificate semantics (where
`certificate_required`/`NO_CERT` is appropriate) for the server-side case (where
it is a message-format error and `decode_error` is required).

## Issue 3: Oversized record does not trigger record_overflow


### Summary

A received TLS record whose length exceeds 2^14 bytes is rejected with
`MBEDTLS_ERR_SSL_INVALID_RECORD`, but the connection is never terminated with a
`record_overflow` alert. The `MBEDTLS_SSL_ALERT_MSG_RECORD_OVERFLOW` constant
(alert 22) is defined but never referenced by any alert-sending code in
`library/`.

### Expected behavior

RFC 8446 §5.1 limits record lengths:

> "The length MUST NOT exceed 2^14 bytes. An endpoint that receives a record that
> exceeds this length MUST terminate the connection with a 'record_overflow'
> alert. Implementations MUST NOT send record fragments that are longer than
> allowed by the maximum fragment length."
> — Section 5.1

So an oversized received record must produce a fatal `record_overflow` alert
(alert 22).

### Actual behavior

The length check returns an error code but sends no alert:

```c
// library/ssl_msg.c:4009-4014
        /* Check record content length */
        if (rec->data_len > MBEDTLS_SSL_IN_CONTENT_LEN) {   // 16384 = 2^14
            MBEDTLS_SSL_DEBUG_MSG(1, ("bad message length"));
            return MBEDTLS_ERR_SSL_INVALID_RECORD;   // no alert sent
        }
```

The downstream error handler only maps `MBEDTLS_ERR_SSL_INVALID_MAC` to a
`bad_record_mac` alert, and leaves `MBEDTLS_ERR_SSL_INVALID_RECORD` unhandled:

```c
// library/ssl_msg.c (record-read error handling)
            /* Error out (and send alert) on invalid records */
#if defined(MBEDTLS_SSL_ALL_ALERT_MESSAGES)
            if (ret == MBEDTLS_ERR_SSL_INVALID_MAC) {
                mbedtls_ssl_send_alert_message(ssl,
                                               MBEDTLS_SSL_ALERT_LEVEL_FATAL,
                                               MBEDTLS_SSL_ALERT_MSG_BAD_RECORD_MAC);
            }
            // MBEDTLS_ERR_SSL_INVALID_RECORD is not handled -> no alert sent
#endif
            return ret;
```

Consistent with this, `MBEDTLS_SSL_ALERT_MSG_RECORD_OVERFLOW` is defined but is
not passed to `mbedtls_ssl_send_alert_message` or `MBEDTLS_SSL_PEND_FATAL_ALERT`
anywhere in `library/`:

```c
// include/mbedtls/ssl.h:537
#define MBEDTLS_SSL_ALERT_MSG_RECORD_OVERFLOW       22  /* 0x16 */
// defined, but no call site references it
```

## Issue 4: Malformed TLSInnerPlaintext cases do not trigger unexpected_message


### Summary

Two malformed-record conditions that RFC 8446 §5.4 requires to be rejected with
an `unexpected_message` fatal alert are instead handled without that alert:

1. A Handshake/Alert record whose `TLSInnerPlaintext.content` is zero-length
   (TLS13_024).
2. A decrypted record whose entire cleartext is all-zero, so no content-type
   marker can be found (TLS13_025).

In the first case the record is silently consumed (only a DoS counter
increments); in the second it produces `MBEDTLS_ERR_SSL_INVALID_RECORD`, which is
not mapped to any alert.

### Expected behavior

RFC 8446 §5.4:

> "Implementations MUST NOT send Handshake and Alert records that have a
> zero-length TLSInnerPlaintext.content; if such a message is received, the
> receiving implementation MUST terminate the connection with an
> 'unexpected_message' alert."

> "If a receiving implementation does not find a non-zero octet in the cleartext,
> it MUST terminate the connection with an 'unexpected_message' alert."
> — Section 5.4

Both conditions must therefore result in a fatal `unexpected_message` alert
(alert 10).

### Actual behavior

**Case 1 — zero-length inner content (TLS13_024):** After decryption,
`ssl_prepare_record_content()` checks `rec->data_len == 0`. The explicit
rejection of zero-length non-app-data records is guarded by
`MBEDTLS_SSL_PROTO_TLS1_2` and only fires for TLS 1.2. For TLS 1.3 the code falls
through to `ssl->nb_zero++` and, after several occurrences, returns
`MBEDTLS_ERR_SSL_INVALID_MAC` (DoS protection) — which maps to `bad_record_mac`,
not `unexpected_message`:

```c
// library/ssl_msg.c:3918-3940 (excerpt)
        if (rec->data_len == 0) {
#if defined(MBEDTLS_SSL_PROTO_TLS1_2)
            if (ssl->tls_version == MBEDTLS_SSL_VERSION_TLS1_2
                && rec->type != MBEDTLS_SSL_MSG_APPLICATION_DATA) {
                /* TLS v1.2 explicitly disallows zero-length messages which are not application data */
                return MBEDTLS_ERR_SSL_INVALID_RECORD;
            }
#endif /* MBEDTLS_SSL_PROTO_TLS1_2 */
            // TLS 1.3: no rejection here — falls through
            ssl->nb_zero++;
            if (ssl->nb_zero > 3) {
                /* ... possible DoS attack ... */
                return MBEDTLS_ERR_SSL_INVALID_MAC;   // -> bad_record_mac, not unexpected_message
            }
        }
```

**Case 2 — all-zero cleartext (TLS13_025):** `ssl_parse_inner_plaintext()` scans
backward for a non-zero content-type octet. When the entire cleartext is zero it
returns -1:

```c
// library/ssl_msg.c:496-504
static int ssl_parse_inner_plaintext(unsigned char const *content,
                                     size_t *content_size,
                                     uint8_t *rec_type)
{
    size_t remaining = *content_size;
    /* Determine length of padding by skipping zeroes from the back. */
    do {
        if (remaining == 0) {
            return -1;   // all bytes are zero — no content type found
        }
        remaining--;
    } while (content[remaining] == 0);
    ...
```

`ssl_decrypt_buf()` maps that to `MBEDTLS_ERR_SSL_INVALID_RECORD`, again with no
alert:

```c
// library/ssl_msg.c:1810-1816
        ret = ssl_parse_inner_plaintext(data, &rec->data_len, &rec->type);
        if (ret != 0) {
            return MBEDTLS_ERR_SSL_INVALID_RECORD;   // not mapped to any alert
        }
```

## Issue 5: Protected ChangeCipherSpec is ignored instead of rejected


### Summary

In TLS 1.3, mbed TLS does not distinguish a plaintext `ChangeCipherSpec` (CCS)
record — tolerated for middlebox compatibility — from a *protected* (encrypted)
CCS record, which RFC 8446 §5 requires to be rejected with an `unexpected_message`
alert. Both are silently ignored via `MBEDTLS_ERR_SSL_CONTINUE_PROCESSING`, so a
protected CCS triggers no alert.

### Expected behavior

RFC 8446 §5 (and Appendix D.4) draw a clear line:

- A *plaintext* CCS record (outer record type = `change_cipher_spec`, 20) is
  tolerated as a compatibility measure and may be ignored.
- A *protected* (encrypted) CCS record (outer record type = `application_data`,
  23, whose inner content type is `change_cipher_spec`) is forbidden:

> "An implementation which receives any other change_cipher_spec value or which
> receives a protected change_cipher_spec record MUST abort the handshake with an
> 'unexpected_message' alert. If an implementation detects a change_cipher_spec
> record received before the first ClientHello message or after the peer's
> Finished message, it MUST be treated as an unexpected record type."
> — Section 5

### Actual behavior

`ssl_prepare_record_content()` only short-circuits decryption for the *outer*
record type `CHANGE_CIPHER_SPEC` (the plaintext case). A protected CCS has outer
type `application_data`, so it is decrypted normally and its inner content type
becomes `change_cipher_spec`:

```c
// library/ssl_msg.c:3818-3836 (excerpt)
    /*
     * In TLS 1.3, always treat ChangeCipherSpec records
     * as unencrypted. ...
     */
#if defined(MBEDTLS_SSL_PROTO_TLS1_3)
    if (ssl->transform_in != NULL &&
        ssl->transform_in->tls_version == MBEDTLS_SSL_VERSION_TLS1_3) {
        if (rec->type == MBEDTLS_SSL_MSG_CHANGE_CIPHER_SPEC) {
            done = 1;   // plaintext CCS: skip decryption
        }
        // protected CCS (outer type != CCS): done stays 0 -> decryption proceeds
    }
#endif /* MBEDTLS_SSL_PROTO_TLS1_3 */
```

After processing, both kinds of CCS reach the same TLS 1.3 dispatch branch, which
unconditionally returns `MBEDTLS_ERR_SSL_CONTINUE_PROCESSING`, causing
`mbedtls_ssl_read_record()` to loop to the next record with no alert:

```c
// library/ssl_msg.c:4959-4965
#if defined(MBEDTLS_SSL_PROTO_TLS1_3)
        if (ssl->tls_version == MBEDTLS_SSL_VERSION_TLS1_3) {
            MBEDTLS_SSL_DEBUG_MSG(2,
                                  ("Ignore ChangeCipherSpec in TLS 1.3 compatibility mode"));
            return MBEDTLS_ERR_SSL_CONTINUE_PROCESSING;   // plaintext AND protected both ignored
        }
#endif /* MBEDTLS_SSL_PROTO_TLS1_3 */
```

## Issue 6: Fatal alert reception does not block further I/O


### Summary

After mbed TLS receives a fatal alert from the peer, it records the event in
`ssl->in_fatal_alert_recv` and returns `MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE`, but
the flag is never used as a guard. Subsequent `mbedtls_ssl_read()` and
`mbedtls_ssl_write()` calls keep working on the connection, contrary to RFC 8446
§6 which requires that no further data be sent or received after an error alert.

### Expected behavior

RFC 8446 §6:

> "Upon receiving an error alert, the TLS implementation SHOULD indicate an error
> to the application and MUST NOT allow any further data to be sent or received
> on the connection. Servers and clients MUST forget the secret values and keys
> established in failed connections..."

> "All the alerts listed in Section 6.2 MUST be sent with AlertLevel=fatal and
> MUST be treated as error alerts when received, regardless of the AlertLevel in
> the message."
> — Section 6

So once a fatal alert is received, the connection must be considered dead: no
further reads or writes should succeed.

### Actual behavior

Receipt of a fatal alert sets the flag but changes no connection state:

```c
// library/ssl_msg.c:4986-4991 (excerpt)
        if (ssl->in_msg[0] == MBEDTLS_SSL_ALERT_LEVEL_FATAL) {
            MBEDTLS_SSL_DEBUG_MSG(1, ("is a fatal alert message (msg %d)",
                                      ssl->in_msg[1]));
            ssl->in_fatal_alert_recv = 1;
            ssl->in_fatal_alert_type = ssl->in_msg[1];
            return MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE;
            // ssl->state is NOT changed; no I/O guard is activated
        }
```

A scan of `library/` shows `in_fatal_alert_recv` has only three references:

| Location | Usage |
|----------|-------|
| `ssl_msg.c:4989` | **Assignment**: set to 1 on fatal-alert receipt |
| `ssl_msg.c:5077` | **Getter**: `mbedtls_ssl_get_fatal_alert()` — passive query, not a guard |
| `ssl_tls.c:1309` | **Initialization**: cleared to 0 on session reset |

There is no guard in `mbedtls_ssl_read()` (`ssl_msg.c:5655`) or
`mbedtls_ssl_write()` (`ssl_msg.c:5921`). In particular, the write path only
checks `ssl->state != MBEDTLS_SSL_HANDSHAKE_OVER` before sending:

```c
// library/ssl_msg.c:5921-5946 (excerpt)
int mbedtls_ssl_write(mbedtls_ssl_context *ssl, const unsigned char *buf, size_t len)
{
    int ret = MBEDTLS_ERR_ERROR_CORRUPTION_DETECTED;
    ...
    if (ssl->state != MBEDTLS_SSL_HANDSHAKE_OVER) {
        if ((ret = mbedtls_ssl_handshake(ssl)) != 0) {
            return ret;
        }
    }
    // in_fatal_alert_recv is never checked before proceeding to send
    ret = ssl_write_real(ssl, buf, len);
    ...
```

Consequences after a fatal alert is received on an otherwise-established
connection:

- **Read path:** a subsequent `mbedtls_ssl_read()` re-enters the read loop and
  can deliver further application data to the application if any arrives.
- **Write path:** a subsequent `mbedtls_ssl_write()` proceeds past the handshake
  guard (state is still `HANDSHAKE_OVER`) and transmits a new TLS record.

## Issue 7: MD5-signed certificate triggers certificate_unknown instead of bad_certificate


### Summary

A certificate signed with an MD5-based signature is correctly rejected, but the
alert sent is `certificate_unknown` (46) instead of the `bad_certificate` alert
(42) that RFC 8446 §4.4.2.4 requires for an MD5-signed certificate. The
`MBEDTLS_X509_BADCERT_BAD_MD` flag set by the X.509 layer is not handled by the
TLS alert-selection chain.

### Expected behavior

RFC 8446 §4.4.2.4:

> "Any endpoint receiving any certificate which it would need to validate using
> any signature algorithm using an MD5 hash MUST abort the handshake with a
> 'bad_certificate' alert. SHA-1 is deprecated, and it is RECOMMENDED that any
> endpoint receiving any certificate which it would need to validate using any
> signature algorithm using a SHA-1 hash abort the handshake with a
> 'bad_certificate' alert."
> — Section 4.4.2.4

So an MD5-signed certificate must produce a fatal `bad_certificate` alert
(alert 42), not `certificate_unknown` (alert 46).

### Actual behavior

The X.509 layer correctly rejects MD5: the default certificate profile allows
only SHA-256/384/512, so `x509_profile_check_md()` sets the
`MBEDTLS_X509_BADCERT_BAD_MD` flag (`0x4000`, `include/mbedtls/x509.h:101`) and
verification returns `MBEDTLS_ERR_X509_CERT_VERIFY_FAILED`.

The TLS alert selection in `mbedtls_ssl_verify_certificate()` then maps
`verify_result` flags to a TLS alert via a priority-ordered `if`/`else if` chain.
That chain enumerates nine flags but **not** `BADCERT_BAD_MD`, so a certificate
flagged *only* with `BADCERT_BAD_MD` falls through to the `else` branch and gets
`MBEDTLS_SSL_ALERT_MSG_CERT_UNKNOWN` (46):

```c
// library/ssl_tls.c:8839-8867 (excerpt)
    if (ret != 0) {
        uint8_t alert;
        if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_OTHER) {
            alert = MBEDTLS_SSL_ALERT_MSG_ACCESS_DENIED;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_CN_MISMATCH) {
            alert = MBEDTLS_SSL_ALERT_MSG_BAD_CERT;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_KEY_USAGE) {
            alert = MBEDTLS_SSL_ALERT_MSG_UNSUPPORTED_CERT;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_EXT_KEY_USAGE) {
            alert = MBEDTLS_SSL_ALERT_MSG_UNSUPPORTED_CERT;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_BAD_PK) {
            alert = MBEDTLS_SSL_ALERT_MSG_UNSUPPORTED_CERT;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_BAD_KEY) {
            alert = MBEDTLS_SSL_ALERT_MSG_UNSUPPORTED_CERT;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_EXPIRED) {
            alert = MBEDTLS_SSL_ALERT_MSG_CERT_EXPIRED;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_REVOKED) {
            alert = MBEDTLS_SSL_ALERT_MSG_CERT_REVOKED;
        } else if (ssl->session_negotiate->verify_result & MBEDTLS_X509_BADCERT_NOT_TRUSTED) {
            alert = MBEDTLS_SSL_ALERT_MSG_UNKNOWN_CA;
        } else {
            alert = MBEDTLS_SSL_ALERT_MSG_CERT_UNKNOWN;   // BADCERT_BAD_MD lands here -> alert 46
        }
        mbedtls_ssl_send_alert_message(ssl, MBEDTLS_SSL_ALERT_LEVEL_FATAL, alert);
    }
```

Alert constants in `include/mbedtls/ssl.h`:

```c
#define MBEDTLS_SSL_ALERT_MSG_BAD_CERT              42  /* 0x2A */   // RFC-required for MD5
#define MBEDTLS_SSL_ALERT_MSG_CERT_UNKNOWN          46  /* 0x2E */   // sent today
```

## Issue 8: TLS 1.3 KeyUpdate is not implemented / handled


### Summary

The TLS 1.3 `KeyUpdate` handshake message (RFC 8446 §4.6.3) appears to be
unimplemented. There is no code to receive or act on an incoming `KeyUpdate`, no
code to send one, and no `MBEDTLS_SSL_HS_KEY_UPDATE` handshake-type constant. An
incoming `KeyUpdate` falls into the post-handshake catch-all and is rejected
without the `unexpected_message` alert.

### Expected behavior

RFC 8446 §4.6.3 defines the `KeyUpdate` handshake message (handshake type 24):

```
struct {
    KeyUpdateRequest request_update;
} KeyUpdate;

enum { update_not_requested(0), update_requested(1), (255) } KeyUpdateRequest;
```

with these requirements:

> "The KeyUpdate handshake message is used to indicate that the sender is updating
> its sending cryptographic keys."

> "Implementations that receive a KeyUpdate message prior to receiving a Finished
> message MUST terminate the connection with an 'unexpected_message' alert."

> "If the request_update field is set to 'update_requested', then the receiver
> MUST send a KeyUpdate of its own with request_update set to
> 'update_not_requested' prior to sending its next Application Data record."

So after the handshake, a receiver of a `KeyUpdate` must update its *read* keys,
and if `update_requested` was set, must send its own `KeyUpdate` before the next
outgoing application-data record.

### Actual behavior

A search of `library/` and `include/` finds no implementation of `KeyUpdate`:

- No references to `key_update`, `KEY_UPDATE`, `KeyUpdate`, or `keyupdate`.
- No references to `request_update`, `update_requested`, or
  `update_not_requested`.
- No state-machine state for receiving or processing a `KeyUpdate`.
- No `MBEDTLS_SSL_HS_KEY_UPDATE` constant. The defined handshake types in
  `include/mbedtls/ssl.h` go up to `MBEDTLS_SSL_HS_FINISHED` (20) plus
  `MBEDTLS_SSL_HS_MESSAGE_HASH` (254); there is no entry for type 24:

```c
// include/mbedtls/ssl.h (handshake-type constants, excerpt)
#define MBEDTLS_SSL_HS_HELLO_VERIFY_REQUEST     3
#define MBEDTLS_SSL_HS_NEW_SESSION_TICKET       4
#define MBEDTLS_SSL_HS_END_OF_EARLY_DATA        5
#define MBEDTLS_SSL_HS_ENCRYPTED_EXTENSIONS     8
#define MBEDTLS_SSL_HS_CERTIFICATE             11
#define MBEDTLS_SSL_HS_SERVER_KEY_EXCHANGE     12
#define MBEDTLS_SSL_HS_CERTIFICATE_REQUEST     13
#define MBEDTLS_SSL_HS_SERVER_HELLO_DONE       14
#define MBEDTLS_SSL_HS_CERTIFICATE_VERIFY      15
#define MBEDTLS_SSL_HS_CLIENT_KEY_EXCHANGE     16
#define MBEDTLS_SSL_HS_FINISHED                20
#define MBEDTLS_SSL_HS_MESSAGE_HASH           254
// no MBEDTLS_SSL_HS_KEY_UPDATE (should be 24)
```

Because `KeyUpdate` is not a recognised handshake type, the post-handshake
handler `ssl_tls13_handle_hs_message_post_handshake()` (in `library/ssl_msg.c`)
cannot dispatch it; it falls through to the catch-all
`return MBEDTLS_ERR_SSL_UNEXPECTED_MESSAGE` — which itself sends no TLS alert.
So today an incoming `KeyUpdate` is not honoured and triggers no
`unexpected_message` alert either.

[mbedtls_rfc_compliance_bundle.zip](https://github.com/user-attachments/files/29154046/mbedtls_rfc_compliance_bundle.zip)

Location	Usage
`ssl_msg.c:4989`	Assignment: set to 1 on fatal-alert receipt
`ssl_msg.c:5077`	Getter: `mbedtls_ssl_get_fatal_alert()` — passive query, not a guard
`ssl_tls.c:1309`	Initialization: cleared to 0 on session reset

Uh oh!

RFC compliance issues in TLS 1.3 alert handling and post-handshake behavior #10781

Description

Overview

Included issues

PoC / reproduction materials

Issue 1: Post-handshake ClientHello / CertificateRequest does not trigger fatal unexpected_message

Summary

Expected behavior

Actual behavior

Issue 2: Empty server Certificate triggers the wrong alert

Summary

Expected behavior

Actual behavior

Issue 3: Oversized record does not trigger record_overflow

Summary

Expected behavior

Actual behavior

Issue 4: Malformed TLSInnerPlaintext cases do not trigger unexpected_message

Summary

Expected behavior

Actual behavior

Issue 5: Protected ChangeCipherSpec is ignored instead of rejected

Summary

Expected behavior

Actual behavior

Issue 6: Fatal alert reception does not block further I/O

Summary

Expected behavior

Actual behavior

Issue 7: MD5-signed certificate triggers certificate_unknown instead of bad_certificate

Summary

Expected behavior

Actual behavior

Issue 8: TLS 1.3 KeyUpdate is not implemented / handled

Summary

Expected behavior

Actual behavior

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions