Server closes connection after client shuts down the sending side of its socket

[chschu]

Basic Infos

Hardware

Hardware: ESP-12F on NodeMCU clone (Geekcreit Doit)
Core Version: 2.3.0

Description

A WiFiClient instance (or the underlying TCP/IP lib) obtained from WiFiServer::available() seems to close the TCP connection when it receives a FIN packet. According to the TCP spec, it should go to the CLOSE_WAIT state, which would still allow data to be sent in the opposite direction.

This happens if a client connects to a server running on the ESP, sends some data and shuts down the writing side of the socket before receiving a response. The client socket's reading side is still open in this case, and the server should be able to send a response to the client.

This issue also occurs with ESP8266WebServer, because it uses WiFiServer/Client.

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E Module)
Flash Size: 4MB (3M SPIFFS)
CPU Frequency: 80Mhz
Upload Using: SERIAL

Sketch

#include "ESP8266WiFi.h"

#define WIFI_SSID "..."
#define WIFI_PASS "..."

WiFiServer server(1400);

void setup() {
  Serial.begin(115200);
  delay(10);

  WiFi.begin(WIFI_SSID, WIFI_PASS);
  Serial.print("Connecting to WiFi.");
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("OK");

  server.begin();
}

void loop() {
  WiFiClient client = server.available();
  if (client) {
    Serial.printf("status before delay: %d\n", client.status());
    delay(1000);
    client.write("Hello, World!\n");
    Serial.printf("status after delay: %d\n", client.status());
    client.stop();
    Serial.printf("status after stop: %d\n", client.status());
  }
}

Debug Messages

Client that shuts down the sending side immediately

Shell command:

sleep 0 | nc 192.168.178.42 1400

(sending side is shut down immediately, because input pipe of netcat is closed after sleep 0)

Client output:

<nothing>

Serial output:

status before delay: 4
status after delay: 0
status after stop: 0

(4 is ESTABLISHED, 0 is CLOSED)

Wireshark capture:

Client that waits for the server to close the connection (works correctly)

Shell command:

sleep 2 | nc 192.168.178.42 1400

(this gives the server enough time to respond and close the connection)

Client output:

Hello, World!

Serial output:

status before delay: 4
status after delay: 4
status after stop: 0

(4 is ESTABLISHED, 0 is CLOSED)

Wireshark capture:

[chschu]

Here comes a small C client that connects to the server running on the ESP (change the IP address accordingly), shuts down the sending side of its socket (unless SHUTDOWN_WRITE is 0) and then receives the response:

#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <sys/errno.h>
#include <unistd.h>

#define SHUTDOWN_WRITE 1

int main() {

    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd < 0) {
        fprintf(stderr, "Failed to create socket: %s\n", strerror(errno));
        return 1;
    }

    struct sockaddr_in serv_addr;
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = inet_addr("192.168.178.42");
    serv_addr.sin_port = htons(1400);
    if (connect(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0) {
        fprintf(stderr, "Failed to connect to host: %s\n", strerror(errno));
        return 1;
    }

#if SHUTDOWN_WRITE
    if (shutdown(sockfd, SHUT_WR) < 0) {
        fprintf(stderr, "Failed to shut down writing side of socket: %s\n", strerror(errno));
        return 1;
    }
#endif

    char response[256];
    while (1) {
        memset(response, 0, sizeof(response));
        int n = read(sockfd, response, sizeof(response) - 1);
        if (!n) {
            // EOF (server closed connection)
            break;
        }
        if (n < 0) {
            fprintf(stderr, "Failed to read from socket: %s\n", strerror(errno));
            return 1;
        }
        printf("%s", response);
    }

    close(sockfd);

    return 0;
}

If the shutdown is enabled, the call to read fails with the error Connection reset by peer, indicating that the server closed the connection. If the shutdown is disabled, the response is received correctly.

[chschu]

Looks like the issue is caused by the receive callback recv_ret_t ClientContext::_recv(tcp_pcb* pcb, pbuf* pb, err_t err), which is called by lwIP with pb == 0 if the remote side sends a FIN. Because this function returns ERR_ABRT, lwIP will abort the whole connection.

[igrr]

This function returns ERR_ABRT because it first calls tcp_abort for the PCB. I think I did this on purpose as a workaround for some issue, but I don't remember what the issue was.

[me-no-dev]

it's there because that is the only indication we have that the connection is closed from the remote end. Else we will have pcb objects around and fill the maximum of 5 quite easily.
As a matter of fact, I'm currently fighting this same problem on ESP32.
I had the impression that CLOSE_WAIT is to be set on connections when we are the closing side (we send the firs FIN)

[chschu]

In the "active close" (we're the closing side) case, the socket will be go to the FIN_WAIT1 state. CLOSE_WAIT is used only for the "passive close" (they're the closing side) case.

After a "passive full close" (both directions closed), the remote side will respond with RST packets to anything we're sending.

After "passive half close" (them->us closed by them, us->them open), the remote side will respond with the normal ACKs.

I don't think there is a way to distinguish the two without sending something to the remote side and handling the RST in case of a "passive full close".

[me-no-dev]

Thanks for clarifying :)
Now do you have a real use case? Because getting that to work would require changes probably in LwIP, so proper disconnect is correctly handled. If we comment those lines now, every connection that we do not explicitly close, will stay open indefinitely.
And there is no way for us to know when the client actually closed the sockets receiving end.
We can't just send probing data as that would interfere with any protocol.
How does linux handle this?

[chschu]

I'm currently building both client and server, so I can just keep both directions of the socket open until the response has been received. The browsers and tools I tested also seemed to behave that way.

Strictly speaking, the server must close the socket explicitly once it is done with the client. If a client dies (power button, network cable), it might never send a FIN, so the server will probably run out of resources if it doesn't close the socket itself.

Most of the time, sockets are closed automatically in the destructor of some higher-level class. If they're not, I'd rather consider it a bug in that higher-level class (or in the application) than in the socket implementation.

I think linux just keeps the socket half-open indefinitely. Running out of resources is not much of an issue there.

[me-no-dev]

we can have a total of 4 or 5 client connections at any one time on ESP8266. Keepnig unnecessary connections open is really not an option here. In AsyncTCP I have implemented timeouts and so on that can coop with sudden loss of connection, but imagine this: You have 4 connections already open, one drops and then tries to reconnect. It will be stopped in SYN because the ESP has all connections open (I have been there many times)

[devyte]

@chschu @me-no-dev @igrr
is there anything that can be done here, or should I just close the issue?

[me-no-dev]

This is where it happens: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/include/ClientContext.h#L463

A setting could be introduced to not close the connection, but it would be user's responsibility to do so. That is the only solution I can come up with.

[igrr]

Closing the pcb will still be done automatically once WiFiClient goes out of scope, i.e. ~WiFiClient calls ClientContext::unref which calls ClientContext::close which calls tcp_close/tcp_abort. So perhaps application doesn't need to do anything? The only consideration against this change is that which @me-no-dev has given a few comments above: in case the remote end closes connection and immediately tries to re-connect, we will have two live pcbs for some time instead of one, which will affect memory use. Let's keep this open and revisit after the upcoming release.

…

On Thu, Oct 12, 2017 at 1:50 PM, Me No Dev ***@***.***> wrote: This is where it happens: https://github.com/esp8266/ Arduino/blob/master/libraries/ESP8266WiFi/src/include/ClientContext.h#L463 A setting could be introduced to not close the connection, but it would be user's responsibility to do so. That is the only solution I can come up with. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2569 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJceuQxVbgJSCVe_mAbFHJQAW5Y2Qcgks5sraimgaJpZM4KMAjO> .

[devyte]

@igrr @me-no-dev code has advanced, including changes in ClientContext. Is there anything further to look into for this one?

[coddicted]

By not sending response to client, client.status() remains 4 but client data doesn't get printed after that. Is it mandatory to close connection after every query by a client?
In AT command mode, i am able to send data from client continuously until the server goes down after a minute or so on its own.

[devyte]

@d-a-v do you know anything about this?