TCP Details

TCPDUMP Setup and Processing

1. Start with WTSS Access Logs. 

1. Look for 502 (or which ever type you need to look for) errors. I use this awk command.

1. awk '($9 ~ /502/)' access.log | awk '{print $7}' | sort | uniq -c | sort -r | more

1. From the generated list, identify a request that you wish to analyze. 

1. You could start with something you are already targeting (like a specific call/endpoint that you know is breaking)

1. In my specific case, I have a plethora of varied requests that are showing a 502 error. So I start with 2 characteristics that I am looking for

1. It should be UI based, so it ignores obvious non conferment requests e.g. API calls, messages to the queue etc.

1. I look for UI requests that have specific identifying information. An example:

1. Path Entry in NGINX Access Log: /ic/integration/home/faces/global?Adf-Window-Id=n5l17geg8&Adf-Page-Id=0

1. Here n5l17geg8 is a unique identifier that hopefully separates this from other requests to this endpoint.

1. faces is a word pattern that to me tells that this is a UI call. I am being very approximate, depending on your circumstances, you may need to guarantee this. 

1. Now run the following command:

1. awk '($9 ~ /502/)' access.log | grep “n5l17geg8”. This allows us to narrow down to a specific request we can then debug.

1. In our case we will start looking at the following request:

1. "10.0.64.3" "-" "-" [22/Mar/2019:07:25:32 +0000] "POST /ic/integration/home/faces/global?Adf-Window-Id=n5l17geg8&Adf-Page-Id=0HTTP/1.1" "502" "563" "https://xxxxx/ic/integration/home/faces/global" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" "4225" "6551" "-" "wdcdevqsic-xxxxx" "170.251.60.100, 10.0.64.5" "xxxxxxx" "9999" "21.183" "10.0.64.8:443" "0.000" "21.183" "21.183" "502" "-" "-" "-" "-" "-"

1. The important parts of the request are underlined. Please note that sometime the content size can also be seen as a unique identifier for the request. In the worst case, time at which the request came in is your best friend, unless someone is pounding the same URL.

1. Subtracting time taken from time local. 

1. 07:25:32 - 21.183 ~ around 07:25:10/11.

1. We take this time and the path to search it in the wireshark UI. Search in the pane like this:

1. http.request.uri eq "/ic/integration/home/faces/global?Adf-Window-Id=n5l17geg8&Adf-Page-Id=0"

1. From the list of entries look for the same time stamp: 07:25:10/11 (get more specific if needed). In our case it is 10.817

1. Since there is no exact match at the time. (but multiple matches, we look on the other server first as well)

1. There are 2 requests in that approximate time, so we look at 07:25:01 and then 07:25:11.

1. Right click, follow, tcp stream. Here we got lucky as the first request was not the failing one … but the next one was and they are both on the same stream.

1. However here we see that the connection was created at 07:20:55 and terminated at 07:25:32. 

1. Check the stream using tcp.stream eq 25677 (you will have some number some like that in your dump details)

1. An open connection and 

1. Provide item sequencing (Send in order order "1, 2", arrive in the order "1, 2" at the opposite end, error-free). 

1. When Unix programs do any sort of I/O, they do it by reading or writing to a file descriptor. An example of this would be a call to the socket() system routine. 

1. It returns the socket descriptor. This file descriptor is simply an integer associated with an open file (that can be a network connection or socket, among other obvious options).

1. Different structs that track TCP-specific info like Sequence numbers, Current window size etc 

1. A Receive buffer (or “queue”) and 

1. A Write buffer (or “queue”). 

1. A new data packet comes arrives on the network interface (NIC) or vNIC, 

1. By an interrupt from the NIC, or by 

1. By Polling the NIC for data. 

1. Decodes the packet and 

1. Source IP, Port, 

1. Destination IP, and port. 

1. Using aforesaid information look up the "struct Sock" (for the socket) in memory associated with the TCP connection. 

1. If there are NO sequencing issues, data payload from the request is copied into the Socket’s Receive buffer. 

1. Blocking read/2, or 

1. I/O multiplexing system call (select or epoll_wait) to wait on the socket.

1. Kernel to remove data from kernel's receive buffer

1. Copy above data into the buffer supplied to the read/2 system call.

1. If the receive buffer is empty and the user calls read, system call will block until data is available.

1. If the receive buffer is nonempty and user calls read, system call will immediately return with whatever data is available. 

1. A partial read can happen if the amount of data ready in the read queue is less than the size of the user-supplied buffer.  The caller can detect this by checking the return value of read

1. If the receive buffer is full and other end of the TCP connection tries to send additional data, the kernel will refuse to ACK the packets. 

1. There is a timeout happening at the Weblogic level. This would require updates to the following entries. This could explain some of the 502s, but not all of them as the request timelines don't fit the descriptions of timeouts. That said, the following timeout settings will need to be increased to their max or 310 s.

1. Duration: Number of seconds to maintain HTTP keep-alive before timing out the request. 

1. On Weblogic to read data from sockets MUXERS are used. We may need to tweak and use native muxers that use platform-specific native binaries to speed it up. But first attempt will be to confirm slow reads.

1. By default, health condition is not checked by cache before returning the cached connection to the client for use. If the network connection is unstable, the system will need to check the connection's health condition before returning it to the client. 

1. To enable this behavior (checking the health condition), set http.keepAliveCache.socketHealthCheckTimeout system property to a value greater than 0 to tune how a socket connection is returned from cache when keep-alive is enabled.

1. Connection Backlog Buffering

1. Number of File Descriptors

1. Complete Message Timeout

1. File Descriptor: Within the context of this discussion, this refers to the new file descriptor made for an incoming connection as a part of the accept() call. Historically, TCP/IP sockets use file descriptors. A network connection can be seen as being analogous to a file - you can read, write, and close both, and fits well with the Unix idea of "everything is a file".

1. Command to extract monthly frequency of 502s  Collapse source

1. cat access.log | grep "Dec/2018" | awk '($9 ~ /502/)'  | awk '{print $4}' | awk -F '/' '{print $1}' | awk -F '[' '{print $2}' | sort | uniq -c

Bibliography:

1. See the implementation of socket structs in the Linux kernel’s net/sock.h.

1. http://beej.us/guide/bgnet/html/single/bgnet.html#socket

How TCP Sockets work:

How (Already) Established Connection Flow works

A few basics first:

Stream sockets (that TCP uses) are reliable two-way connected communication streams. They maintain 

A socket can be seen as a way to speak to other programs using standard Unix file descriptors. In other words:

For each Socket or TCP file descriptor (tracked by the kernel), there are 

Kernel is notified in 2 possible ways:

Kernel gets the packet from the NIC, it 

Identifies the TCP connection packet is associated with. It uses

Kernel wakes/triggers processes doing/using 

User Space process calls read/2 on the file descriptor. This causes 

Read Semantics: 

Possible Issues:

Socket Reads are too slow. If the Kernel senses that the kernel buffers are not clearing quickly, it may start sending RESET back to the client.

Connection's health condition is bad:

Other Parameters (probably not relevant in this case since the socket and the connection already exist):

Definitions:

1. This section explains how socket system calls interact with kernel data structures, and how kernel interacts with the actual network. It also explains how listen queue overflowswork.
   1. 1. HTTPS Duration:Number of seconds to maintain HTTPS keep-alive before timing out the request.
      2. Complete Message Timeout: The maximum number of seconds that this server waits for a complete message to be received. If you configure network channels for this server, each channel can override this message timeout.
      3. Idle Connection Timeout: The maximum number of seconds that a connection is allowed to be idle before it is closed by the server. The T3 and T3S protocols ignore this attribute. If you configure network channels for this server, each channel can override this idle connection message timeout.