Detecting a Transparent Proxy with Wireshark/Tshark
Recently I got pulled into a debate between two colleagues who were troubleshooting a problem where some users could access a website over SSL, and others couldn't. One person was arguing that the problem was caused by client misconfiguration, and the other was arguing that it wasn't. Following my mantra "when in doubt, capture packets", we captured some traffic and had a look. I'm not going to go through the entire troubleshooting process; rather I'm going to focus on what was ultimately causing the problem. Here's the packet sequence, output from Tshark with some of the TCP details removed to make it fit:
921 13.795492 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [SYN]
926 13.837731 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [SYN, ACK]
927 13.837753 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [ACK]
928 13.838086 10.100.100.192 -> 184.86.133.186 SSL Client Hello
932 13.840253 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
943 13.879563 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [ACK]
944 13.879588 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
945 13.880052 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
946 13.881491 184.86.133.186 -> 10.100.100.192 TLSv1 Server Hello
947 13.881513 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
948 13.881535 184.86.133.186 -> 10.100.100.192 TLSv1 Certificate, Server Hello Done
949 13.881545 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
950 13.881554 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
953 13.882063 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
The key thing to note here is the delta between the SYN and SYN/ACK: about 42ms. When I was viewing this in Wireshark I set a packet time reference on packet #921, using the "Ctrl T" keyboard shortcut; this makes it easier to see the delta values.
I then set another time reference on packet 931, the TLSv1 Client Hello. Immediately following this, less than 1 millisecond later, we see a RST come back from the server. Red flag! Since we already established the probable latency between the hosts as ~42ms using the SYN - SYN/ACK pair, this is extremely suspicious.
A few packets later, we see a TLSv1 Server Hello message inbound, AFTER the RST. The delta? Approximately 42ms, exactly what we'd expect.
I immediately inferred from this that a transparent proxy content filter was spoofing the RST due to something that it deemed to be objectionable content, and for whatever reason it wasn't notifying the user.
I talked to the administrator for the content filter, and indeed, a policy had been incorrectly applied to some users that blocked the content in question.
Another win for packet capture.
921 13.795492 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [SYN]
926 13.837731 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [SYN, ACK]
927 13.837753 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [ACK]
928 13.838086 10.100.100.192 -> 184.86.133.186 SSL Client Hello
932 13.840253 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
943 13.879563 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [ACK]
944 13.879588 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
945 13.880052 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
946 13.881491 184.86.133.186 -> 10.100.100.192 TLSv1 Server Hello
947 13.881513 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
948 13.881535 184.86.133.186 -> 10.100.100.192 TLSv1 Certificate, Server Hello Done
949 13.881545 10.100.100.192 -> 184.86.133.186 TCP 50306 > https [RST]
950 13.881554 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
953 13.882063 184.86.133.186 -> 10.100.100.192 TCP https > 50306 [RST]
The key thing to note here is the delta between the SYN and SYN/ACK: about 42ms. When I was viewing this in Wireshark I set a packet time reference on packet #921, using the "Ctrl T" keyboard shortcut; this makes it easier to see the delta values.
I then set another time reference on packet 931, the TLSv1 Client Hello. Immediately following this, less than 1 millisecond later, we see a RST come back from the server. Red flag! Since we already established the probable latency between the hosts as ~42ms using the SYN - SYN/ACK pair, this is extremely suspicious.
A few packets later, we see a TLSv1 Server Hello message inbound, AFTER the RST. The delta? Approximately 42ms, exactly what we'd expect.
I immediately inferred from this that a transparent proxy content filter was spoofing the RST due to something that it deemed to be objectionable content, and for whatever reason it wasn't notifying the user.
I talked to the administrator for the content filter, and indeed, a policy had been incorrectly applied to some users that blocked the content in question.
Another win for packet capture.