Summary
When the server initiates termination, it sends a FIN first, and the client responds FINACK. The server should then send the final ACK. But in current code path, the firewall endpoint is removed before ACK is sent out, so the egress firewall drops it. The client never receives the final ACK and ultimately closes through a timeout rather than a graceful termination.
Root cause
The root cause is the ordering in FreeRTOS. After receiving the client’s FINACK, FreeRTOS prepares the final ACK and then calls:
vTCPStateChange(..., eCLOSE_WAIT);
This synchronously invokes on_tcp_connect(..., false), which removes the firewall endpoint. And after vTCPStateChange() returns, FreeRTOS attempts to send the final ACK, but the egress firewall rejects it because the connection tuple in the firewall no longer exists.
Here is the relevant code:https://github.com/FreeRTOS/FreeRTOS-Plus-TCP/blob/27a96812e84413763db60c08ec41311b75897e8c/source/FreeRTOS_TCP_State_Handling.c#L288
Also, I believe the same issue also happens failed handshakes path. If the server receives an invalid packet while in state SYN_RECEIVED, FreeRTOS will abort the handshake by calling vTCPStateChange() first. This invokes on_tcp_connect(..., false) and removes the firewall endpoint. FreeRTOS then constructs and attempts to send an RST msg, but the egress firewall will drop it because the tuple has already been removed.
Relevant code:https://github.com/FreeRTOS/FreeRTOS-Plus-TCP/blob/27a96812e84413763db60c08ec41311b75897e8c/source/FreeRTOS_TCP_State_Handling.c#L382
Affected component
lib/tcpip/network_wrapper.cc::network_socket_close() and lib/tcpip/network_wrapper.cc::on_tcp_connect(), the later one is the FreeRTOS callback on receiving a packet, which will inherently remove the endpoint tuple in the firewall.
Impact
The connection will be forced to terminated after 30 seconds without following the full 4-way handshake termination rule. (Forced by ipconfigTCP_HANG_PROTECTION_TIME in FreeRTOS)
PoC
Here is the output when running example 05.HTTP server with extra firewall instrumentation.
Because the client never received that ACK, it remained in LAST_ACK and repeatedly retransmitted FIN,ACK. Those are the repeated ingress drops:
FW-OBS: DROP ingress proto 6 ... remotePort 0xd304
Suggested fix
When the first egress FIN is observed, set the firewall hole to a special state: only 1 incoming FINACK and 1 egress ACK is allowed. Once the budget is consumed, the hole will be dropped. That will ensure the firewall hole dropping happens after the egress of final ACK.
Summary
When the server initiates termination, it sends a FIN first, and the client responds FINACK. The server should then send the final ACK. But in current code path, the firewall endpoint is removed before ACK is sent out, so the egress firewall drops it. The client never receives the final ACK and ultimately closes through a timeout rather than a graceful termination.
Root cause
The root cause is the ordering in FreeRTOS. After receiving the client’s FINACK, FreeRTOS prepares the final ACK and then calls:
vTCPStateChange(..., eCLOSE_WAIT);
This synchronously invokes on_tcp_connect(..., false), which removes the firewall endpoint. And after vTCPStateChange() returns, FreeRTOS attempts to send the final ACK, but the egress firewall rejects it because the connection tuple in the firewall no longer exists.
Here is the relevant code:https://github.com/FreeRTOS/FreeRTOS-Plus-TCP/blob/27a96812e84413763db60c08ec41311b75897e8c/source/FreeRTOS_TCP_State_Handling.c#L288
Also, I believe the same issue also happens failed handshakes path. If the server receives an invalid packet while in state SYN_RECEIVED, FreeRTOS will abort the handshake by calling vTCPStateChange() first. This invokes on_tcp_connect(..., false) and removes the firewall endpoint. FreeRTOS then constructs and attempts to send an RST msg, but the egress firewall will drop it because the tuple has already been removed.
Relevant code:https://github.com/FreeRTOS/FreeRTOS-Plus-TCP/blob/27a96812e84413763db60c08ec41311b75897e8c/source/FreeRTOS_TCP_State_Handling.c#L382
Affected component
lib/tcpip/network_wrapper.cc::network_socket_close()andlib/tcpip/network_wrapper.cc::on_tcp_connect(), the later one is the FreeRTOS callback on receiving a packet, which will inherently remove the endpoint tuple in the firewall.Impact
The connection will be forced to terminated after 30 seconds without following the full 4-way handshake termination rule. (Forced by
ipconfigTCP_HANG_PROTECTION_TIMEin FreeRTOS)PoC
Here is the output when running
example 05.HTTP serverwith extra firewall instrumentation.Because the client never received that ACK, it remained in LAST_ACK and repeatedly retransmitted FIN,ACK. Those are the repeated ingress drops:
FW-OBS: DROP ingress proto 6 ... remotePort 0xd304Suggested fix
When the first egress FIN is observed, set the firewall hole to a special state: only 1 incoming FINACK and 1 egress ACK is allowed. Once the budget is consumed, the hole will be dropped. That will ensure the firewall hole dropping happens after the egress of final ACK.