Tracking, Not Tracing, Linux Thread Activity For Complete System Visibility - Tanel Poder

[@wolpumba4099]

*Tracking, Not Tracing, Linux Thread Activity for Complete System Visibility* * *0:10** Introduction:* Tanel Poder introduces the concept of Linux thread state tracking as a way to gain complete system visibility from a demand-side perspective, focusing on the activities of active threads. * *0:16** Demand-Side Visibility:* The goal is to understand what threads are doing and why they might be off-CPU, providing insight beyond traditional system utilization metrics. * *0:47** Process Snapper & Fact Table Analogy:* Tanel highlights his older tool, Process Snapper, which samples proc files to show thread activity. This data can be considered a fact table in a data warehouse, allowing analysis of thread behavior over time. * *1:09** Xtop & Extended Task State Array:* Xtop, a newer eBPF-based tool, is introduced. It utilizes an "extended task state array" to store custom thread metadata, including application-level context like system calls and stack traces. * *3:41** Decoupled Layers:* The system employs two decoupled layers: eBPF programs that maintain the extended task state array by updating thread state information, and a sampling layer that periodically captures snapshots of this state. * *4:50** Tracking vs. Tracing:* The approach emphasizes "tracking" over "tracing." Instead of logging every event, only the latest state and relevant context are stored, providing an overview without the overhead of full tracing. * *6:34** Benefits of Tracking:* This method offers a balance between system-wide visibility and the ability to drill down into individual threads and their off-CPU behavior, as demonstrated with examples of MySQL threads performing fsync operations. * *7:32** Scheduler Hooking & Latency Analysis:* Tanel discusses the potential for hooking into scheduler events to track thread wake-up relationships and analyze latencies between different components, such as application threads and database or file system operations. * *8:57** Future Plans & Call for Contributions:* Future plans include incorporating distributed tracing IDs and building a V3 of the tool using modern libbpf. A call for contributions is made to help develop the open-source project further. * *9:38** Conclusion:* Tanel encourages viewers to try out the tools, provide feedback, and reach out with any questions. I used gemini-1.5-pro-exp-0801 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.06 Input tokens: 14677 Output tokens: 486