I’ve been watching videos and books to get myself familiar with eBPF tooling, more toward Linux network stack and VFS. This post serves as a note on what I’ve learned.
Brendan Gregg has listed the command line tools that are useful for analyzing a Linux instance if he has to ssh into the instance. They could be preliminary analysis to quickly get a feel on what’s going on on a specific instance.
- dmesg -T | tail
- vmstat 1
- mpstat -P ALL 1
- pidstat 1
- iostat -xz 1
- free -m
- sar -n DEV 1
- sar -n TCP,ETCP 1
To dig deeper into the detail, these eBPF-based tooling would be helpful.
When the information you are trying to get are not there, and
strace is not enough. The following tools could be helpful to peek into the critical paths in the kernel modules.
A nice book to read is a new book authored by Brendan.
While studying for the SRE materials, I also found Cloudflare has many good blog posts that I could learn from.
A blog post on how to optimize for http2 stack gives really good insight on how the Linux network stack works.
net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr net.ipv4.tcp_notsent_lowat = 16384
Their interview questions also makes you start to question yourself if you really know the modern TCP/IP stack.
Archaeology What is the lowest TCP port number? The TCP frame has an URG pointer field, when is it used? Can the RST packet have a payload? When is the "flow" field in IPv6 used? What does the IP_FREEBIND socket option do? Forgotten Quirks What does the PSH flag actually do? The TCP timestamp is implicated in SYN cookies. How? Can a "UDP" packet have a checksum field set to zero? How does TCP simultaneous open work? Does it actually work? Fragmentation and Congestion What is a stupid window syndrome? What are the CWE and ECE flags in TCP header? What is the IP ID field and what does it have to do with DF bit? Why do some packets have a non-zero IP ID and a DF set? Fresh Ideas Can a SYN packet have a payload? (hint: new RFC proposals) Can a SYN+ACK packet have a payload? ICMP Path MTU ICMP packet-too-big messages are returned by routers and contain a part of the original packet in the payload. What is the minimal length of this payload that is accepted by Linux? When an ICMP packet-too-big message is returned by an intermediate router it will have the source IP of that router. In practice though, we often see a source IP of the ICMP message to be identical to the destination IP of the original packet. Why could that happen? Linux Configuration Linux has a "tcp_no_metrics_save" sysctl setting. What does it save and for how long? Linux uses two queues to handle incoming TCP connections: the SYN queue and the accept queue. What is the length of the SYN queue? What happens if the SYN queue grows too large and overflows? Touching the router What are BGP bogons, and why are they less of a problem now? TCP has an extension which adds MD5 checksums to packets. When is it useful? And finally: What are the differences in checksumming algorithms in IPv4 and IPv6?