This is the last day of 2019, it was transformative year. I’ve learned a lot by practicing. I faced challenges and learned to make difficult decisions but make the decision rationally. When I faced challenges I like to do binge reading, part of the reasons is that I have the presumption that there must be smart people have been through the challenges I faced today. When I watched the documentary “Inside Bill’s Brain” I noticed that he also has the binge reading habit. He would lock himself in a room and read everything he can about a topic. The books I have finished in the past several weeks are:
- The man who solved the market
- A Man of All Markets
- Dear Chairman
- Quench your own Thirst
- The Ride of a Lifetime
- Never Split the difference
- My Father’s Business
- 公司的品格 2
I’ve learned to like Audiobooks and I would say it really help a lot on your reading efficiency since it would make you effectively use your time on the road or while you are jogging. This easily adds another hour or two on each of the day. It could also let your eyes to rest, especially for a software engineer where you already use your eyes a lot. I can simply borrow the Audiobook from Singapore library and finish it in one week or two (The deadline of borrowing a book also creates urgency so that you feel like finishing it).
I’ve been watching videos and books to get myself familiar with eBPF tooling, more toward Linux network stack and VFS. This post serves as a note on what I’ve learned.
Brendan Gregg has listed the command line tools that are useful for analyzing a Linux instance if he has to ssh into the instance. They could be preliminary analysis to quickly get a feel on what’s going on on a specific instance.
- dmesg -T | tail
- vmstat 1
- mpstat -P ALL 1
- pidstat 1
- iostat -xz 1
- free -m
- sar -n DEV 1
- sar -n TCP,ETCP 1
To dig deeper into the detail, these eBPF-based tooling would be helpful.
When the information you are trying to get are not there, and
strace is not enough. The following tools could be helpful to peek into the critical paths in the kernel modules.
A nice book to read is a new book authored by Brendan.
While studying for the SRE materials, I also found Cloudflare has many good blog posts that I could learn from.
A blog post on how to optimize for http2 stack gives really good insight on how the Linux network stack works.
net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr net.ipv4.tcp_notsent_lowat = 16384
Their interview questions also makes you start to question yourself if you really know the modern TCP/IP stack.
Archaeology What is the lowest TCP port number? The TCP frame has an URG pointer field, when is it used? Can the RST packet have a payload? When is the "flow" field in IPv6 used? What does the IP_FREEBIND socket option do? Forgotten Quirks What does the PSH flag actually do? The TCP timestamp is implicated in SYN cookies. How? Can a "UDP" packet have a checksum field set to zero? How does TCP simultaneous open work? Does it actually work? Fragmentation and Congestion What is a stupid window syndrome? What are the CWE and ECE flags in TCP header? What is the IP ID field and what does it have to do with DF bit? Why do some packets have a non-zero IP ID and a DF set? Fresh Ideas Can a SYN packet have a payload? (hint: new RFC proposals) Can a SYN+ACK packet have a payload? ICMP Path MTU ICMP packet-too-big messages are returned by routers and contain a part of the original packet in the payload. What is the minimal length of this payload that is accepted by Linux? When an ICMP packet-too-big message is returned by an intermediate router it will have the source IP of that router. In practice though, we often see a source IP of the ICMP message to be identical to the destination IP of the original packet. Why could that happen? Linux Configuration Linux has a "tcp_no_metrics_save" sysctl setting. What does it save and for how long? Linux uses two queues to handle incoming TCP connections: the SYN queue and the accept queue. What is the length of the SYN queue? What happens if the SYN queue grows too large and overflows? Touching the router What are BGP bogons, and why are they less of a problem now? TCP has an extension which adds MD5 checksums to packets. When is it useful? And finally: What are the differences in checksumming algorithms in IPv4 and IPv6?