Processing Docker Container Logs and Metrics at Scale (DockerCon 2023)

Containerization has transformed the software development landscape, and deploying applications is one part of the journey; the need for continuous monitoring and processing of logs and metrics in large-scale environments has become a critical challenge. We dive deep into how logging works for Docker containerized applications, the pros and cons of filesystem versus network-oriented events under different transports and formats, the complexity of multiline log messages, and how to deal with them. We demonstrate why in-motion log parsing and filtering are essential for scalable environments and the best practices to move that information reliably to different backends for data analysis that considers various strategies to deal with back pressure, network outages, data loss, and general unresponsive systems. From the tooling aspect, we will use Fluent Bit, an open source telemetry agent, to demonstrate the data collection and processing aspects applied to containerized environments and general best practices for a healthy logging layer. The target audience for this session includes DevOps engineers, system administrators, developers, and anyone involved in cloud-native environments looking for a deep understanding of applied telemetry to deal with highly scalable environments. By implementing the strategies and techniques shared in this session, attendees enhance the operational efficiency and stability of their logging layers.
Presentation: Processing Docker containers Logs and Metrics at Scale
Speaker: Eduardo Silva, Co-founder & CEO, Calyptia
Resources:
Highlights from DockerCon 2023: New Docker Local, Cloud, and AI/ML Innovations - www.docker.com/blog/highlight...
Get started with Docker - www.docker.com/get-started/
What are containers? www.docker.com/resources/what...
Try Docker Desktop www.docker.com/products/docke...
Docker 101 Tutorial www.docker.com/101-tutorial/
Join the conversation!
LinkedIn → dockr.ly/LinkedIn
Twitter → dockr.ly/Twitter
Facebook → dockr.ly/Facebook
Instagram → dockr.ly/Instagram
ABOUT DOCKER: Docker provides a suite of development tools, services, trusted content, and automations, used individually or together, to accelerate the delivery of secure applications.
#docker #containers #softwaredevelopment