Quick feedback - you are talking too fast.

Thanks nice overview of using eBPF in the kernel of embedded Automotive Grade Linux to improve observability and threat protection in the kernel container/vm sense its seriously powerful.

Simply amazing. Trying this out

Jackson Jennifer Davis Shirley Walker Patricia

Liz, I'm late to this one however, as usual, SUEPR COOL :-) -- Q: do you play those instruments behind you? 🙂

👍

*Security Assessment of the eBPF Verifier* * *0:37** Enhanced Safety with eBPF:* eBPF offers a safer alternative to loading custom kernel modules, minimizing the risk of widespread system crashes caused by program bugs. * *0:58** Runtime Compilation:* eBPF code is compiled into assembly by the kernel at runtime, ensuring near-native execution speed. * *1:01** Kernel Verification for Untrusted Code:* The Linux kernel employs an eBPF verifier to ensure the safety of potentially malicious eBPF programs before execution. * *1:25** Static Verification Process:* The verifier analyzes every possible execution path of an eBPF program, rejecting it if any path is deemed unsafe. * *2:06** Definition of Safety:* Safety encompasses memory safety (preventing invalid pointer dereferences) and other documented and undocumented invariants. For instance, programs must free or release referenced pointers before exiting, as statically checked by the verifier. * *2:48** Soundness and Completeness:* The eBPF verifier is designed to be sound (rejecting all unsafe programs), but not necessarily complete (it may reject some safe programs). * *3:10** Previous Research:* Past security research has revealed over 40 CVEs related to the verifier and highlighted the potential for high-impact vulnerabilities like container escapes and privilege escalation. * *3:32** Formal and Dynamic Approaches:* Some parts of the verifier have undergone formal verification, proving their correctness in isolation. Dynamic testing and fuzzing efforts like Google's buzzer have also been employed. * *4:17** NCC Group's Methodology:* The current security assessment involves manual source code review of the verifier and relevant parts of the x86-64 JIT compiler. It leverages expertise in kernel development and vulnerability research. * *4:34** Invariant Documentation:* A key aspect of the assessment is to identify and comprehensively document the invariants that the verifier must enforce beyond basic memory safety. * *4:51** Public Report Release:* The findings of the security assessment will be published in a detailed report later in the year. I used gemini-1.5-pro-exp-0801 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.05 Input tokens: 13046 Output tokens: 440

*Tracking, Not Tracing, Linux Thread Activity for Complete System Visibility* * *0:10** Introduction:* Tanel Poder introduces the concept of Linux thread state tracking as a way to gain complete system visibility from a demand-side perspective, focusing on the activities of active threads. * *0:16** Demand-Side Visibility:* The goal is to understand what threads are doing and why they might be off-CPU, providing insight beyond traditional system utilization metrics. * *0:47** Process Snapper & Fact Table Analogy:* Tanel highlights his older tool, Process Snapper, which samples proc files to show thread activity. This data can be considered a fact table in a data warehouse, allowing analysis of thread behavior over time. * *1:09** Xtop & Extended Task State Array:* Xtop, a newer eBPF-based tool, is introduced. It utilizes an "extended task state array" to store custom thread metadata, including application-level context like system calls and stack traces. * *3:41** Decoupled Layers:* The system employs two decoupled layers: eBPF programs that maintain the extended task state array by updating thread state information, and a sampling layer that periodically captures snapshots of this state. * *4:50** Tracking vs. Tracing:* The approach emphasizes "tracking" over "tracing." Instead of logging every event, only the latest state and relevant context are stored, providing an overview without the overhead of full tracing. * *6:34** Benefits of Tracking:* This method offers a balance between system-wide visibility and the ability to drill down into individual threads and their off-CPU behavior, as demonstrated with examples of MySQL threads performing fsync operations. * *7:32** Scheduler Hooking & Latency Analysis:* Tanel discusses the potential for hooking into scheduler events to track thread wake-up relationships and analyze latencies between different components, such as application threads and database or file system operations. * *8:57** Future Plans & Call for Contributions:* Future plans include incorporating distributed tracing IDs and building a V3 of the tool using modern libbpf. A call for contributions is made to help develop the open-source project further. * *9:38** Conclusion:* Tanel encourages viewers to try out the tools, provide feedback, and reach out with any questions. I used gemini-1.5-pro-exp-0801 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.06 Input tokens: 14677 Output tokens: 486

*Break All the Things: Leveraging eBPF for Chaos Engineering* * *0:19** Chaos Engineering Background:* Scott introduces chaos engineering, highlighting its origins at Apple and popularization by Netflix. It involves experimenting on systems to enhance their resilience to failures. * *0:34** Traditional Chaos Experiments:* Common methods include disrupting VMs and network connectivity. Tools like Chaos Mesh (sometimes using eBPF) and cloud platforms like AWS facilitate these experiments. * *0:50** Benefits of Chaos Engineering:* It helps identify emergent failures in multi-service systems and weaknesses in individual services under specific failure conditions. * *1:14** Limitations of Traditional Approaches:* Large-scale cloud environments may be unnecessary for testing individual services, and traditional component testing with stubs can be cumbersome. * *1:28** Shifting Left with eBPF:* eBPF offers a "universal language of destruction" for injecting failures directly at the kernel and network level, simplifying testing and allowing earlier detection of issues. * *2:16** eBPF-Based Failure Injection Examples:* * *2:27** Network I/O Disruption:* eBPF programs can intercept and drop network traffic based on criteria like process ID or port, simulating network outages. * *3:01** System Call Failures:* eBPF can intercept system calls (like `openat`) and return error codes, simulating resource access failures. This approach is potentially generalizable across different system calls. * *3:45** Traffic Control (TC) for Packet Loss:* Leveraging the TC subsystem, eBPF can selectively drop outgoing traffic for specific processes, mimicking network packet loss. * *4:23** Future Directions and Potential:* * *4:26** Integration with Test Frameworks:* Scott proposes integrating eBPF-based chaos experiments into common testing frameworks like JUnit or Go test, enhancing developer workflows. * *4:37** Exploring New Probe Points:* The eXpress Data Path (XDP) could be used for coarser network failures, and the Linux Security Modules (LSM) for simulating security breaches. * *4:52** Simulating Resource Exhaustion:* eBPF could be used to manipulate cgroup limits to create resource exhaustion scenarios. * *5:03** Call to Action:* Scott encourages feedback and contributions to the project, providing a link to his GitHub repository with example code. I used gemini-1.5-pro-exp-0801 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.05 Input tokens: 13312 Output tokens: 494

I think from a support perspective we still have to learn and understand whether ebpf is just another tool or it cteates new possibilities to have a functionality we don't have with all the existing tools, being in business for decates . Ebpf is also a thing all admins and software engineers have to be aware of, otherwise it might not be explainable why we see a certain behaviour. Im looking forward to all of the rhings which might be possible with ebpf

netkit@cilium 🎉

Nico. So according to your statement - you recommend using Cillium CLI over HELM.

IT WAS ACTUALLY AMAZING, KEEP ON THE GOOD WORK FROM A STUDENT FROM VIET

*Summary* * *(**0:00** - **1:18**)* *Tetragon Overview:* Tetragon is an open-source, eBPF-based security observability and enforcement tool. It's part of the Cilium project and uses eBPF to hook into kernel events, filter for suspicious activities, and generate alerts or take actions. * *(**1:18** - **17:59**)* *In-Kernel Network Selectors:* Tetragon allows you to filter network events directly within the kernel, minimizing overhead. You can create policies based on source/destination IP, port, and protocol to selectively monitor or block network traffic. This is particularly useful for observing connections to/from the internet while ignoring local traffic. * *(**17:59** - **29:26**)* *Canary Tokens Integration:* Tetragon can trigger Canary Tokens, which are essentially tripwires that alert you when an attacker interacts with a specific resource. You can use these tokens in various ways (e.g., links, files) and Tetragon can be configured to trigger them based on specific events. This provides a quick way to get notified of suspicious activity. * *(**29:26** - **44:01**)* *Binary Matching (Prefix, Exact, Postfix):* Tetragon can match against the full path of executed binaries (not just the user-provided string). It supports exact matching, prefix matching (e.g., anything in /tmp/), and the recently added postfix matching (e.g., anything ending in 'curl'). This ensures accurate identification even when attackers use symbolic links or obfuscation techniques. * *(**44:01** - **59:52**)* *Rate Limiting:* To prevent overwhelming the system with events, Tetragon allows you to rate-limit alerts. You can specify the frequency (e.g., one event every 10 seconds) for a given event type, which is useful for monitoring high-volume activities like UDP datagrams. Summarized by AI model: gemini-1.5-pro-exp-0801 Cost (if I didn't use the free tier): $0.1184 Input tokens: 31205 Output tokens: 872

Help Bangladesh students Save Bangladesh students 😭😭

Which ide is this?

So glad to see something like this. I came to the same conclusion as you but you’ve done something about it!

Why tc hookpoint don't work in python syntax

i want to use generated costimize BPF code(writen in C ) for develope cilium bpf, can I just use cilium tools like cilium-bpf or something like this to integrate C code to kernel(maybe cilium itself can automatically compile and load it to kernel hook)?

I love shedrack

Thanks Riham, it's a great talk helping to understand, what are current challenges with GPU profiling. Would be interesting to know, how the things have changed since then :)

I am waiting :)

what happens if the chained XDP programs return different actions?

nice demo

Great demo of the Cilium network policies! Couple suggestions to make it easier to follow the actions: - move the video feed from the camera into the upper right corner so it does not obscure the terminal input and output - split the screen in two, with one half showing the yaml file and other half where the commands are run

I understand that this a basic implementation, but for a large HTTP request, do we not need to forward all the packets to the same backend ? With this implementation, it will randomly distribute packets among the backends. So we need some stickiness to map a particular HTTP request to a particular backend server.

Hello y’all! Thanks for hosting ✌️😌

This is definitely very interesting and I got a lot of insights on things I did not know about. However I am wondering if eBPF based technologies like Grafana Beyla and/or Pyroscope and/or Agent/Alloy could not be used to compare flame graphs representations of sshd calls to potentially detect zero-day behavior change instead of using the info gathered after tampering was infered. That would be much more powerfull furthermore as I think that considering the fact that this cve was detected due to a timing issue, such flame graphs should really be able to put this into evidence quite easily

Very cool video. From Cilium 1.14 the default value for fqdns min ttl is 0 and not 3600s

amazing!

*Abstract* This video demonstrates how to detect and potentially prevent the exploitation of the "XZ" SSH vulnerability using Tetragon, an eBPF-based security tool. The video covers setting up a reproduction environment with a compromised SSH server, installing Tetragon, and creating a tracing policy to identify the vulnerability. Additionally, the video highlights the benefits of Tetragon's kernel-level operation and showcases a library of example policies for detecting common vulnerabilities and exploits (CVEs). Tetragon can when itself is running on a host and the vulnerability (sshd and xz) is present in a container that is running on the host. *Summary* *Introduction and Events* * 0:15: Introduction and welcome message. * 1:07: Announcement of the "Buzzing through Kubernetes" workshop series for network engineers. * 1:35: Upcoming events: Open Source Summit in Seattle, KCD New York, and Cloud Native Security Con in Seattle. * 2:30: Information on subscribing to the Isovalent newsletter for weekly news updates. *XZ Vulnerability and Detection* * 3:11: Discussion of the "XZ" SSH vulnerability and Isovalent's efforts in understanding and addressing it. * 3:23: Reference to a blog post by Jalal and Jeremy Covin that provides detailed information about the exploit. * 4:31: Introduction of a Tetragon tracing policy example for detecting the vulnerability. * 4:47: Explanation of how the policy works by hooking into the Linux kernel and watching for specific library versions used by the SSH daemon (sshd). * 6:42: Discussion on the potential response actions (notification vs. blocking) and their implications. *Reproduction Environment Setup* * 7:44: Setting up a reproduction environment using Kind to create a local Kubernetes cluster with a vulnerable SSH server. * 12:19: Troubleshooting Docker issues and restarting the cluster creation process. * 17:40: Installing Cilium and Tetragon on the Kind cluster. * 26:31: Addressing technical difficulties and switching to a different host for demonstration purposes. * 35:33: Creating a Kind cluster on a Google Cloud Compute instance. * 42:01: Verifying the initial state of the SSH server and confirming the presence of the vulnerable library. *Vulnerability Exploitation and Detection with Tetragon* * 49:24: Downloading and installing a compromised version of the "XZ" library to create a vulnerable SSH server. * 52:07: Verifying the vulnerability using the provided detection script. * 54:44: Accessing the Tetragon container and using the `tetra` command-line tool to observe events. * 57:09: Demonstrating that Tetragon detects the vulnerability when SSHing into the compromised server. * 59:58: Analyzing the detailed event information provided by Tetragon. * 1:00:12: Highlighting the specific event that triggers the tracing policy and confirms the use of the compromised library. *Conclusion and Resources* * 1:01:03: Introduction to the Tetragon policy library with examples for detecting various CVEs, including the "XZ" vulnerability. * 1:01:37: Encouragement for community contributions to the policy library. * 1:02:03: Closing remarks and thanks to the audience. i used gemini 1.5 pro to summarize the transcript with this prompt: Create an abstract and summarize the following video transcript as a bullet list. Prepend each bullet point with starting timestamp. Do not show the stopping timestamp. Also split the summary into sections and create section titles: Token count 8,196 / 1,048,576

A good point to say and show what task we are trying to do, what problem we’re trying to resolve? Ps: Oh I see, after 20 minutes of presentation 😂

Can we have a link to the slides?

😩 'promo sm'

Hi, thanks for this! Do you have the source code/prototype for this?

Looks like you have only 8 queues for the nic, that's why cilium does not use your 10 cores (perfectly fine if on purpose though) better to have a bit of spare CPU for the rest of the system ;)

Excellent video, packing wealth of information in 17mins.

Very useful video!

Hi, I'd like to clarify if AWS VPC CNI is needed for Prefix Delegation to work ? I thought that it's possible to remove AWS VPC CNI and install cilium witch would handle prefix delegation in the same way as AWS VPC CNI. Is there a way to not have AWS VPC CNI and still have working Prefix Delegation ?

So happy this channel exists!

Very cool, thanks for making this available. Have a use case somewhat similar to Netflix but processing linear video on bare metal that this may be useful for helping to optimize the environment.

Please, where can I find the slides? Thanks.

Nice

absolutely fantastic

Wonderful!

so great!!! helped me a bunch, it is kinda tricky to onboard to ebpf and opentelemetry at once

Was trying to find a link to Dario's blog post referred around 41:42 but to no avail. Any hints as to where to find, it, please?

If this could have been recorded in a slightly higher resolution, that would be fantastic. HD is not easy to read/follow with small font :(

This a great idea, congratulations

How do I get the source code？