🌿 E2: Weekly kernel dev gardening: Build system refactor & cleanup 🌿

(detailed timestamps in comments)
#livecoding #softwareengineering #operatingsystem
/ discord
Welcome! This is my chill Operating Systems development "gardening" project.
The "gardening philosophy" is to have fun, above everything else. This means taking baby steps and locking in small wins before building up for more ambitious projects. People also call this "hacking" on something.
My personal goal & vision is to build a minimal, high quality OS foundation which can be used for play, experimentation, and learning.
I'm personally not an expert systems programmer, although I'd like to be someday. Right now, but I'm just an intermediate one - I worked in infosec for 7 years, and music tech for 3 years. I've been exposed to the Linux kernel in both settings, but in highly constrained contexts, and mostly reading code, not writing it.
==
Today, we delete all the random files we no longer need, and then take a baby step towards separating the kernel files from userspace files.
topics touched: C, Makefile, build system, some assembly, includes
(detailed timestamps in comments)
0:00: Start
0:19: 🌿 Chill development session focusing on organizing directory structure and cleaning up build system.
10:08: 🌱 Refactoring and cleaning up the existing OS codebase by removing unnecessary files and rewriting scripts in Python.
18:07: 🧩 Exploring and modifying a build system in OS development, including understanding make files and generating assembly code.
27:08: 🧹 Refactoring and cleaning up unnecessary components in an OS/kernel development project
37:15: 🌿 Refactoring and cleaning up an existing OS project, focusing on removing unnecessary files and maintaining good commit hygiene.
46:02: 🧹 Exploration of boot process components and bootloader functionality in OS development.
53:53: 🔧 Refactoring and cleaning up the build system for OS/kernel development
1:04:59: 🌱 Refactoring the build system by moving kernel files to a separate directory incrementally.
1:16:58: 🧰 Refactoring build system and directory structure for OS/kernel development.
1:29:36: 🌿 Refactoring and cleanup of build system for operating system kernel development
1:42:06: ⚙️ Troubleshooting build system errors and refactoring for kernel and user space separation.
1:55:35: 💻 Development progress on OS kernel: directory restructuring, file movement, and build system update.
2:07:23: 🌿 Concluding a stream with unexpected adventures and plans for next week.
Recap by Tammy AI

[@offlinemark]

Full timestamps & Best moments: 3:10 Last time: We fixed brittle includes, because we want to be good software engineers 5:20 Goal: Show everyday life of systems programmer, which is often not glamorous 16:17 Rewrite a perl script because I don't want to maintain perl 31:33 The gardening philosophy Full timestamps: 0:19: 🌿 Chill development session focusing on organizing directory structure and cleaning up build system. 0:19: Working on an existing basic open-source operating system for fun and learning purposes. 1:51: Planning to clean up messy build system and file organization. 2:30: Previous session involved cleaning up code formatting and fixing issues. 3:39: Goal is to maintain professionalism and use best practices in development. 6:09: Encountered build error and fixed it by including missing files. 8:51: Successfully got the OS to build again after fixing the error. 8:51: Updated topic to 'streaming OS kernel Dev' and focused on build system cleaning. 9:51: Main task for the day is to organize directory structure for better organization. 10:08: 🌱 Refactoring and cleaning up the existing OS codebase by removing unnecessary files and rewriting scripts in Python. 10:08: Planning to add folders for organization in the codebase 10:16: Intending to delete unnecessary code before restructuring the project 11:00: Considering removing random scripts like 'cut H' for simplification 12:16: Emulating real-world software development by inheriting and managing existing codebase 14:23: Setting guiding principles to keep moving forward and have fun during the development process 16:19: Planning to rewrite a Pearl script in Python for better maintainability 17:22: Receiving positive feedback on Discord for the development progress 17:28: Focusing on making the codebase comfortable and familiar to work with 18:07: 🧩 Exploring and modifying a build system in OS development, including understanding make files and generating assembly code. 18:07: Starting from scratch with comfortable tools in OS development. 18:15: Inheriting existing system components and needing to port them. 18:45: Reimplementing a script in Python for textual manipulations. 19:25: Discovering and exploring the vectors.s file in the build system. 21:03: Understanding the concept of targets and dependencies in make files. 23:01: Observing implicit rules in make files for generating object files. 24:27: Discussing the absence of explicit rules for certain build tasks. 26:02: Experimenting with injecting syntax errors in the Pearl script for testing. 26:53: Corrupting the Pearl script to simulate a build task failure. 27:08: 🧹 Refactoring and cleaning up unnecessary components in an OS/kernel development project 27:08: Identified issues with the build system due to bad assembly generation 27:55: Discovered unused flags in the assembly compilation command 28:29: Noticed implicit build rules in the makefile causing confusion 29:05: Exploring and deleting unnecessary files and components 30:42: Goal of understanding and customizing the OS for experimentation 32:20: Considering reimplementing core parts of the OS for learning purposes 33:05: Removing unnecessary programs related to printing and table of contents 36:17: Cleaning up unused files and components to streamline the project 37:02: Engaging with viewers and seeking input on OS development experiences 37:15: 🌿 Refactoring and cleaning up an existing OS project, focusing on removing unnecessary files and maintaining good commit hygiene. 37:15: Project inherited from xv6 project, minimal code modifications made 37:55: Using OBS for streaming instead of Streamlabs for better functionality 38:53: Interest in learning more about ARM architecture and OpenBSD for future OS development 39:07: Comparing x86 and ARM for OS development, aiming to eventually port OS to ARM 41:07: Cleaning up unnecessary legacy files and code from the OS project 43:12: Maintaining good commit hygiene and following best practices in software development 45:00: Limited prior experience in building OS, more familiar with specific areas of Linux kernel 46:02: 🧹 Exploration of boot process components and bootloader functionality in OS development. 46:02: Discussion on the importance of the boot sector in the boot process 46:21: Mention of using Master Boot Record technology for booting x86 PCs 46:35: Expression of interest in learning and exploring more about OS development and bootloaders 49:14: Acknowledgment of the significance of staying healthy while programming 49:31: Recognition of the importance of the discussed file related to the boot process 50:02: Explanation of the role of BIOS in loading the 512-byte boot sector into memory 51:25: Comparison between traditional bootloaders and industrial-strength bootloaders like uboot 52:28: Illustration of the boot process components including BIOS, boot sector, and kernel 53:30: Reference to a complex boot process setup with multiple partitions for boot and file systems 53:53: 🔧 Refactoring and cleaning up the build system for OS/kernel development 53:53: Discussion on the importance of a specific comment related to BIOS boot sector verification 55:17: Deciding to remove 'spinP' tool as it is not needed in the build system 59:09: Considering adding directories like 'kernel' and 'user space' to organize the files better 1:01:31: Creating a new branch named 'adders' for the refactoring task 1:03:27: Expressing apprehension about potentially breaking the build system while trying to understand the make file better 1:04:59: 🌱 Refactoring the build system by moving kernel files to a separate directory incrementally. 1:04:59: Identified confusion between kernel and user space objects in the make file. 1:05:39: Goal to move essential kernel files to a directory while maintaining build functionality. 1:06:10: Experimented with moving files to 'kernel' directory using Vim and Xargs commands. 1:10:19: Received help from chat on using Xargs command for file movement. 1:12:11: Successfully moved kernel-related files to 'kernel' directory to avoid build breakage. 1:15:07: Considered separating kernel and user directories with internal make files for future refactoring. 1:16:09: Opted for incremental approach due to time constraints, focusing on gradual porting of files. 1:16:58: 🧰 Refactoring build system and directory structure for OS/kernel development. 1:16:58: Moved user space to a new directory 'Kern' and kernel to the root of the repository. 1:17:07: Considered options for including files from 'Kern' space to 'Colel' space. 1:19:05: Renamed 'Kernel' directory to 'Kar' to resolve conflicts and build issues. 1:20:33: Addressed the issue of missing headers by moving them to the appropriate directories. 1:27:05: Discussed the challenge of shared headers between kernel and user space, opting for simplicity. 1:28:54: Fixed build system to recognize the new directory structure and file locations. 1:29:25: Acknowledged the need to later address variable handling in make files. 1:29:36: 🌿 Refactoring and cleanup of build system for operating system kernel development 1:29:36: Identified compilation issue with vectors in the code 1:30:36: Troubleshooting build system errors related to header files 1:31:47: Noticed a bug in GCC command with two -I flags 1:36:05: Considered separating CXX flags for user space support 1:39:46: Utilized Vim registers for efficient text manipulation 1:40:44: Added Kernel C Flags to kernel build targets 1:41:58: Acknowledged potential impact on user space build due to changes 1:42:06: ⚙ Troubleshooting build system errors and refactoring for kernel and user space separation. 1:42:06: Attempted to compile kernel file but encountered errors due to missing dependencies. 1:43:20: Realized the shared make file for kernel and user space caused issues with C Flags. 1:46:46: Suggested adding explicit rules for kernel and user space dependencies. 1:50:00: Tried a hacky solution by adding Das I Kern to the list for both spaces. 1:52:03: Encountered confusion and errors during the clean-up process. 1:53:18: Divergence in Fork and fixing errors related to types in x86 H file. 1:54:05: Questioned the formatting of dash I and attempted to define types in x86 H. 1:55:02: Continued troubleshooting errors and uncertainties in the build system. 1:55:35: 💻 Development progress on OS kernel: directory restructuring, file movement, and build system update. 1:55:35: Created 'Kern' directory and relocated kernel files for build system compatibility. 2:04:02: Utilized xargs to efficiently move C and H files into the new directory. 2:06:30: Adjusted build system references to the new 'Kern' directory for kernel object files. 2:07:23: 🌿 Concluding a stream with unexpected adventures and plans for next week. 2:07:23: Deciding to end the stream and make dinner 2:07:33: Reflecting on the stream turning into an interesting adventure 2:07:39: Wishing viewers a great week and announcing return next week Recap by Tammy AI

[@DanelonNicolas]

nice! this time I lost the stream, but I'm seeing it async haha oh my dude! between the 0:20:00 and 0:24:20 the bell of the terminal sound so strong for people with headsets.. :( can u please remove it?

[@offlinemark]

Thanks for the feedback! I’ve unfortunately conditioned myself to ignore the bell sound, and just disabled it 😂

[@offlinemark]

Thanks for watching!!

[@offlinemark]

Btw I made a discord server! happy to chat, feel free to join: discord.gg/EH224mjcsd