Welcome to this comprehensive Linux kernel tutorial designed to transform you from a curious learner into a confident kernel developer. The Linux kernel stands as one of the most remarkable achievements in open-source software engineering—a sophisticated, high-performance operating system core that powers everything from smartphones and embedded devices to supercomputers and cloud infrastructure serving billions of users worldwide.

This Linux kernel tutorial provides a structured, practical journey through kernel architecture, development methodologies, programming techniques, and real-world application. Whether you’re a system programmer seeking to understand how operating systems work at the lowest levels, a developer aiming to contribute to the Linux project, or a computer science student exploring kernel internals, this tutorial delivers the knowledge foundation you need to succeed.

Throughout this Linux kernel tutorial, you’ll discover how the kernel manages hardware resources, implements process scheduling, handles memory allocation, provides file system support, enables network communication, and exposes these capabilities to user-space applications through system calls. More importantly, you’ll learn how to read kernel source code, write kernel modules, debug kernel issues, and participate in the global Linux development community.

Understanding the Linux Kernel: Foundation Concepts

What is the Linux Kernel?

The Linux kernel is the core component of Linux operating systems—the fundamental software layer that mediates between hardware and applications. It oversees communication and the exchange of resources between a computer’s hardware and its processes, functioning as the essential bridge that enables software to utilize physical hardware capabilities.

Created by Linus Torvalds in 1991 as a hobby project while studying at the University of Helsinki, the Linux kernel has evolved into a massive collaborative development effort involving thousands of contributors from around the world. What began as a simple task switcher written in Intel 80386 assembly language has grown into millions of lines of sophisticated C code supporting dozens of processor architectures and countless hardware configurations.

The kernel operates in a protected execution mode called kernel space, distinct from user space where regular applications run. This separation provides critical security and stability—if an application crashes, it doesn’t bring down the entire system. However, kernel code executes with full hardware access, meaning kernel bugs can cause system-wide failures, making kernel development both powerful and responsibility-heavy.

Monolithic Yet Modular Architecture

A unique characteristic explored in this Linux kernel tutorial is that Linux employs a seemingly paradoxical design. The Linux kernel is both monolithic and modular—classified as a monolithic kernel architecturally since the entire OS runs in kernel space, yet modular since it can be assembled from modules that are loaded and unloaded at runtime.

In a monolithic kernel design, all operating system services—including process management, memory management, file systems, device drivers, and network stacks—execute within a single address space in kernel mode. This contrasts with microkernel architectures where many services run as separate user-space processes. The monolithic approach provides performance advantages through direct function calls rather than inter-process communication, but requires careful engineering to maintain stability and security.

The modular aspect means that kernel functionality can be compiled as loadable kernel modules (LKMs) that can be dynamically inserted into or removed from the running kernel without rebooting. This modularity enables: flexibility to load only needed drivers and features, reduced memory footprint by loading modules on-demand, simplified hardware support without kernel recompilation, and easier development and testing of new features.

Kernel Space vs User Space

Understanding the distinction between kernel space and user space is fundamental to this Linux kernel tutorial. Kernel space is the memory area reserved to the kernel while user space is the memory area reserved to a particular user process, with kernel space access protected so user applications cannot access it directly, while user space can be directly accessed from code running in kernel mode.

This separation implements a protection mechanism ensuring system stability and security. User applications execute in user mode with restricted privileges—they cannot directly access hardware, manipulate memory management structures, or interfere with other processes. When user applications need kernel services, they invoke system calls that transition execution into kernel mode where privileged operations can be performed safely under kernel control.

The virtual address space is typically divided with kernel space occupying the upper portion and user space the lower portion, though the exact layout varies by architecture. Security mechanisms prevent user mode code from accessing kernel space addresses, while kernel mode code can access both kernel and user space as needed for implementing system call functionality.

Linux Kernel Architecture: Major Subsystems

Process Management and Scheduling

The process management subsystem handles one of the kernel’s most critical responsibilities: creating, scheduling, and terminating processes. The system manages creation, scheduling, and termination of processes, ensuring that multiple programs can execute concurrently while providing the illusion that each has exclusive access to the processor.

Process Representation: Each process is represented by a task_struct structure containing all information the kernel needs to manage that process—process state, scheduling information, memory management details, open file descriptors, signal handlers, parent/child relationships, and much more. This structure serves as the kernel’s comprehensive view of a process.

Process States: Processes transition through several states during their lifecycle including running (actively executing on a CPU), runnable (ready to execute but waiting for CPU time), sleeping (waiting for an event or resource), stopped (suspended by signal), and zombie (terminated but waiting for parent to collect exit status).

The Completely Fair Scheduler: Modern Linux kernels use the Completely Fair Scheduler (CFS) as the default scheduler for normal priority processes. CFS aims to provide fair CPU time distribution among all runnable processes by tracking virtual runtime—the amount of CPU time each process has received. The scheduler selects the process with the smallest virtual runtime for execution, ensuring fairness over time.

Real-Time Scheduling: Linux also supports real-time scheduling policies (SCHED_FIFO and SCHED_RR) for processes requiring predictable, low-latency response. Real-time processes have priority over normal processes and use different scheduling algorithms optimized for deterministic behavior rather than fairness.

Memory Management

The memory management subsystem provides one of the kernel’s most complex and critical functions—managing physical RAM and virtual address spaces for all processes. Understanding memory management is essential in any comprehensive Linux kernel tutorial.

Physical Memory Management: The kernel divides physical memory into fixed-size pages (typically 4KB on x86 architectures) and tracks the status of each page—whether it’s free, allocated to a specific process, used for kernel data structures, or serving as page cache for file system operations. The buddy allocator manages free pages efficiently, allowing allocation and deallocation of contiguous page blocks in power-of-two sizes.

Virtual Memory: Each process operates in its own virtual address space, providing isolation and simplifying memory management from the application perspective. The Memory Management Unit (MMU) hardware translates virtual addresses to physical addresses using page tables maintained by the kernel. This translation occurs transparently for every memory access.

Page Tables: Multi-level page table structures map virtual addresses to physical pages. On x86-64 systems, four-level page tables enable addressing the vast 64-bit virtual address space while keeping memory overhead manageable. The kernel manages these page tables, updating them during process creation, memory allocation, and page swapping.

The Slab Allocator: For efficiently managing kernel memory allocations smaller than a page, Linux uses slab allocation. The slab allocator creates caches for frequently used object types (like task_struct, inode, dentry structures), pre-allocating objects to avoid allocation overhead and reducing fragmentation.

Page Cache: To accelerate file I/O, the kernel maintains a page cache storing recently accessed file contents in RAM. When applications read files, the kernel serves data from cache when available, avoiding slow disk access. Write operations update the cache, with actual disk writes occurring asynchronously in the background.

Swapping and OOM Killer: When memory becomes scarce, the kernel can swap less-active pages to disk storage, freeing physical memory for more immediate needs. If even swapping cannot satisfy memory demands, the Out-Of-Memory (OOM) killer terminates processes to prevent total system failure, selecting victims based on heuristics designed to maximize availability while minimizing disruption.

Virtual File System (VFS)

The Virtual File System layer provides a crucial abstraction in the kernel architecture, allowing different file system implementations to coexist while presenting a uniform interface to user space.

Unified Interface: Applications interact with files using standard system calls like open(), read(), write(), close() regardless of the underlying file system type—ext4, XFS, Btrfs, NFS, FAT32, or dozens of others. The VFS layer translates these generic operations into file system-specific implementations.

Key VFS Data Structures: The VFS operates using several fundamental structures including superblock (representing a mounted file system), inode (representing a file or directory), dentry (directory entry caching), and file (representing an open file). These abstractions enable consistent file operations across diverse file system types.

File System Operations: Each file system registers operation vectors—function pointers implementing file system-specific behaviors for operations like creating files, reading directories, allocating blocks, and managing metadata. When VFS receives a request, it invokes the appropriate function through these operation vectors.

Path Lookup and Dcache: Translating pathname strings (like /home/user/document.txt) into kernel inode structures involves traversing the directory hierarchy. The directory entry cache (dcache) dramatically accelerates this process by caching recent path lookups, avoiding repeated file system queries.

Device Drivers

Device drivers enable the kernel to communicate with hardware devices, translating generic kernel operations into device-specific commands. The Linux kernel provides interaction with hardware devices such as hard disks, network cards, and peripherals, with a corresponding driver for each device that allows the kernel to communicate with it.

Character Devices: Character devices provide stream-oriented access where data is read and written sequentially in character or byte units. Examples include serial ports, keyboards, mice, and terminals. Character device drivers implement file operations (open, read, write, ioctl, release) that applications invoke through device files in /dev.

Block Devices: Block devices handle data in fixed-size blocks (usually 512 bytes or larger) and support random access. Hard drives, SSDs, and USB storage are block devices. Block device drivers are more complex than character drivers, interfacing with the block I/O layer, request queues, and I/O schedulers.

Network Devices: Network interfaces use a different model than character and block devices. Instead of file operations, network drivers implement transmission and reception functions, interfacing with the network stack. The kernel provides sk_buff structures to represent network packets flowing through the system.

Driver Model and Sysfs: The kernel’s device model organizes devices hierarchically by bus type (PCI, USB, platform), manages driver binding to devices, handles power management, and exports device information through sysfs virtual file system. This infrastructure simplifies driver development and provides consistent device management.

Network Stack

The Linux kernel includes a comprehensive network stack implementing the TCP/IP protocol suite and enabling network communication.

Layered Architecture: The network stack is structured into a layered architecture comprising Link, Internet, Transport, and Application layers, following the standard TCP/IP model. Each layer handles specific aspects of network communication.

Link Layer: The link layer manages interaction with network hardware including Ethernet adapters, wireless devices, and other physical interfaces. It encapsulates higher-layer packets into frames suitable for transmission on the physical network, handling media access control and error detection.

Internet Layer: IP (Internet Protocol) operates at this layer, handling packet routing between networks. Both IPv4 and IPv6 are supported, with the kernel maintaining routing tables, performing fragmentation when necessary, and implementing ICMP for control messages.

Transport Layer: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) provide transport services. TCP offers reliable, ordered, connection-oriented communication with flow control and congestion management. UDP provides lightweight, connectionless datagram delivery without reliability guarantees.

Socket API: Applications access network functionality through the socket API, which provides standardized interfaces for creating connections, sending and receiving data, and managing network communication. The kernel socket layer connects these user-space calls to the appropriate protocol implementations.

Netfilter Framework: The netfilter framework enables packet filtering, network address translation (NAT), and connection tracking. Tools like iptables/nftables configure netfilter rules, allowing implementation of firewalls and complex routing policies.

Getting Started: Setting Up Your Kernel Development Environment

Prerequisites and Required Knowledge

Before diving deep into this Linux kernel tutorial‘s practical aspects, understanding the prerequisites ensures effective learning. Kernel development differs significantly from application programming, requiring specific knowledge and skills.

C Programming Proficiency: The Linux kernel is written in C, so you should have at least a basic understanding of C before diving into kernel work. You need comfort with pointers, structures, memory management, function pointers, and bitwise operations. Kernel code uses GNU C extensions beyond standard C, including statement expressions, typeof operator, and computed goto.

Linux User Experience: Familiarity with Linux command-line interfaces, system administration tasks, and shell scripting helps tremendously. Understanding how user-space applications interact with the kernel through system calls provides valuable context for kernel development.

Git Version Control: Knowing git is not actually required but can really help you since you can dig through changelogs and search for information you’ll need. The Linux kernel development workflow revolves around git repositories, patches, and email-based collaboration. Basic git operations—cloning, committing, branching, and generating patches—are essential skills.

Hardware Understanding: While not requiring electrical engineering expertise, basic understanding of computer architecture, CPU operation, memory hierarchies, I/O devices, and hardware buses helps immensely when working with kernel code that directly manipulates hardware.

Installing Development Tools

Setting up a proper development environment forms the foundation for successful kernel work.

Compiler and Build Tools: The GNU Compiler Collection (GCC) compiles the Linux kernel. Install gcc, make, binutils, and related development tools. Recent kernels can also be built with Clang/LLVM for those preferring that toolchain.

# For Debian/Ubuntu systems
sudo apt-get install build-essential libncurses-dev bison flex \
  libssl-dev libelf-dev gcc make

# For Red Hat/Fedora/CentOS systems
sudo dnf groupinstall "Development Tools"
sudo dnf install ncurses-devel bison flex elfutils-libelf-devel openssl-devel

Kernel Source Code: Obtain the kernel source from kernel.org or clone the git repository. The mainline repository maintained by Linus Torvalds serves as the official source.

# Download specific kernel version
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.tar.xz
tar xvf linux-6.6.tar.xz
cd linux-6.6

# Or clone git repository
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux

Documentation and References: The kernel source includes extensive documentation in the Documentation/ directory. Reading Documentation/process/howto.rst and related files provides valuable guidance on kernel development workflows and community practices.

Virtual Machine Setup for Safe Testing

Testing kernel code on physical hardware risks system crashes and data loss. Virtual machines provide safe, isolated environments for kernel development and testing.

QEMU/KVM: QEMU provides full system emulation supporting multiple architectures. With KVM (Kernel-based Virtual Machine), QEMU achieves near-native performance by leveraging hardware virtualization.

# Install QEMU
sudo apt-get install qemu-system-x86

# Create minimal root filesystem
# (You can use buildroot, busybox, or existing distribution images)

# Boot custom kernel in QEMU
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
  -hda rootfs.ext4 \
  -append "root=/dev/sda console=ttyS0" \
  -m 2G \
  -smp 2 \
  -nographic

VirtualBox and VMware: Commercial virtualization solutions like VirtualBox and VMware Workstation offer user-friendly interfaces and good Linux support. While slightly less flexible than QEMU for kernel development, they provide reliable environments for testing.

Container-Based Development: While containers don’t provide full kernel isolation (they share the host kernel), they offer useful environments for compiling kernel source and developing kernel modules that can then be tested in VMs.

Building Your First Kernel

Kernel Configuration

Kernel configuration determines which features, drivers, and architectures are included in the compiled kernel. The configuration system uses Kconfig language to define options and dependencies.

Configuration Methods: Several interfaces access the configuration system:

# Text-based menu interface (ncurses)
make menuconfig

# Graphical Qt-based interface
make xconfig

# Graphical GTK-based interface
make gconfig

# Use existing configuration
make oldconfig

# Use distribution's configuration as starting point
cp /boot/config-$(uname -r) .config
make oldconfig

Key Configuration Areas: Essential configuration categories include:

• General setup: Core kernel features, namespace support, control groups
• Processor type: Target CPU architecture and optimizations
• Power management: ACPI, CPU frequency scaling, suspend/resume
• Networking support: Protocol stacks, netfilter, wireless
• Device drivers: Hardware support for storage, network, graphics, USB, etc.
• File systems: Supported file system types
• Kernel hacking: Debug options, tracing, testing features

For learning purposes, enable debug options under “Kernel hacking” including debug symbols, lockdep, KASAN (memory error detector), and various warning flags. These features slow the kernel but catch bugs early.

Compilation Process

Once configured, build the kernel using make. The build system (Kbuild) handles dependencies, compiling thousands of source files and linking them into kernel images and modules.

# Compile kernel with all available CPU cores
make -j$(nproc)

# This compiles:
# - vmlinux: uncompressed kernel image
# - bzImage: compressed bootable kernel (arch/x86/boot/bzImage)
# - Kernel modules: drivers compiled as loadable modules

# Compile only specific targets
make bzImage        # Just the bootable kernel
make modules        # Just the modules
make Module.symvers # Symbol versions for modules

Understanding Build Output: The compilation produces several important files:

• vmlinux: Uncompressed kernel ELF image with debug symbols
• arch/x86/boot/bzImage: Compressed bootable kernel image
• System.map: Kernel symbol table mapping addresses to names
• Module.symvers: Symbol version information for modules
• .ko files: Compiled kernel modules scattered through the source tree

Installation: After successful compilation, install the kernel and modules:

# Install modules to /lib/modules/$(uname -r)
sudo make modules_install

# Install kernel image to /boot
sudo make install

# Update bootloader (grub)
sudo update-grub  # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg  # Fedora/RHEL

Booting Custom Kernel

After installation, reboot and select your custom kernel from the bootloader menu. Modern systems use GRUB2 which automatically detects new kernels during make install.

Troubleshooting Boot Issues: If the custom kernel fails to boot:

• Check kernel configuration includes necessary drivers (disk controller, root filesystem type)
• Verify initramfs/initrd was generated with required modules
• Review bootloader configuration for correct kernel parameters
• Use GRUB’s edit feature to modify boot parameters at runtime
• Keep previous working kernel available for fallback

Kernel Boot Parameters: The kernel accepts parameters during boot modifying behavior:

# Common parameters
root=/dev/sda1              # Root filesystem device
ro                          # Mount root read-only initially
console=ttyS0,115200       # Serial console for debugging
debug                       # Enable debug output
single                     # Boot to single-user mode

Writing Your First Kernel Module

Hello World Kernel Module

Loadable Kernel Modules (LKMs) provide the most accessible entry point for kernel programming. This Linux kernel tutorial section demonstrates creating a simple module.

Basic Module Structure: Every kernel module requires certain elements:

// hello.c - Simple Hello World Kernel Module
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("A simple Hello World module");
MODULE_VERSION("1.0");

// Module initialization function
static int __init hello_init(void)
{
    printk(KERN_INFO "Hello, World! Module loaded.\n");
    return 0;  // Return 0 indicates successful loading
}

// Module cleanup function
static void __exit hello_exit(void)
{
    printk(KERN_INFO "Goodbye, World! Module unloaded.\n");
}

// Register init and exit functions
module_init(hello_init);
module_exit(hello_exit);

Key Components Explained:

• #include <linux/*.h>: Include kernel headers, not standard C library headers
• MODULE_LICENSE(): Declares module licensing (must be GPL-compatible for many kernel features)
• __init and __exit: Macros marking initialization and cleanup code for memory optimization
• printk(): Kernel’s equivalent to printf, logs to kernel message buffer
• module_init() and module_exit(): Macros registering initialization and cleanup functions

Compiling Kernel Modules

Kernel modules require special Makefiles integrating with the kernel build system (Kbuild).

Makefile for External Module:

# Makefile for hello module
obj-m += hello.o

# Get kernel build directory
KDIR := /lib/modules/$(shell uname -r)/build

# Default target
all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules

clean:
	$(MAKE) -C $(KDIR) M=$(PWD) clean

# Install module
install:
	$(MAKE) -C $(KDIR) M=$(PWD) modules_install

Build and Load Module:

# Compile the module
make

# This produces hello.ko (kernel object)

# Load module into running kernel
sudo insmod hello.ko

# Verify module loaded
lsmod | grep hello

# View kernel log messages
dmesg | tail

# Remove module
sudo rmmod hello

# View unload message
dmesg | tail

Module Parameters

Modules can accept parameters during loading, enabling runtime configuration without recompilation.

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/moduleparam.h>

MODULE_LICENSE("GPL");

// Define module parameters
static int number = 42;
static char *name = "default";

// Register parameters
module_param(number, int, S_IRUGO);
MODULE_PARM_DESC(number, "An integer parameter");

module_param(name, charp, S_IRUGO);
MODULE_PARM_DESC(name, "A string parameter");

static int __init param_init(void)
{
    printk(KERN_INFO "Module loaded with number=%d, name=%s\n", 
           number, name);
    return 0;
}

static void __exit param_exit(void)
{
    printk(KERN_INFO "Module unloaded\n");
}

module_init(param_init);
module_exit(param_exit);

Using Parameters:

# Load with custom parameters
sudo insmod param_module.ko number=100 name="test"

# Parameters also visible in sysfs
cat /sys/module/param_module/parameters/number
cat /sys/module/param_module/parameters/name

Understanding Kernel Logging

The printk() function logs messages with priority levels indicated by log level macros:

printk(KERN_EMERG "System is unusable");       // Emergency (0)
printk(KERN_ALERT "Action required");          // Alert (1)
printk(KERN_CRIT "Critical condition");        // Critical (2)
printk(KERN_ERR "Error condition");            // Error (3)
printk(KERN_WARNING "Warning condition");      // Warning (4)
printk(KERN_NOTICE "Normal but significant");  // Notice (5)
printk(KERN_INFO "Informational");             // Info (6)
printk(KERN_DEBUG "Debug-level messages");     // Debug (7)

Messages are stored in the kernel ring buffer, viewable through dmesg or /proc/kmsg. The console displays messages above a configurable log level threshold.

Also Read : WorkFusion Tutorial

Device Driver Development

Character Device Driver Basics

Character device drivers provide one of the most common kernel programming tasks. This Linux kernel tutorial section covers creating a simple character driver.

Driver Structure:

#include <linux/init.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/uaccess.h>

#define DEVICE_NAME "mychardev"
#define CLASS_NAME "mychar"

MODULE_LICENSE("GPL");

static int major_number;
static struct class *chardev_class = NULL;
static struct device *chardev_device = NULL;

// Device file operations
static int dev_open(struct inode *, struct file *);
static int dev_release(struct inode *, struct file *);
static ssize_t dev_read(struct file *, char __user *, size_t, loff_t *);
static ssize_t dev_write(struct file *, const char __user *, size_t, loff_t *);

// File operations structure
static struct file_operations fops = {
    .open = dev_open,
    .read = dev_read,
    .write = dev_write,
    .release = dev_release,
};

static int __init chardev_init(void)
{
    // Register character device
    major_number = register_chrdev(0, DEVICE_NAME, &fops);
    if (major_number < 0) {
        printk(KERN_ALERT "Failed to register device\n");
        return major_number;
    }

    // Create device class
    chardev_class = class_create(THIS_MODULE, CLASS_NAME);
    if (IS_ERR(chardev_class)) {
        unregister_chrdev(major_number, DEVICE_NAME);
        return PTR_ERR(chardev_class);
    }

    // Create device file
    chardev_device = device_create(chardev_class, NULL,
                                   MKDEV(major_number, 0),
                                   NULL, DEVICE_NAME);
    if (IS_ERR(chardev_device)) {
        class_destroy(chardev_class);
        unregister_chrdev(major_number, DEVICE_NAME);
        return PTR_ERR(chardev_device);
    }

    printk(KERN_INFO "Device registered: major=%d\n", major_number);
    return 0;
}

static void __exit chardev_exit(void)
{
    device_destroy(chardev_class, MKDEV(major_number, 0));
    class_unregister(chardev_class);
    class_destroy(chardev_class);
    unregister_chrdev(major_number, DEVICE_NAME);
    printk(KERN_INFO "Device unregistered\n");
}

static int dev_open(struct inode *inodep, struct file *filep)
{
    printk(KERN_INFO "Device opened\n");
    return 0;
}

static int dev_release(struct inode *inodep, struct file *filep)
{
    printk(KERN_INFO "Device closed\n");
    return 0;
}

static ssize_t dev_read(struct file *filep, char __user *buffer,
                        size_t len, loff_t *offset)
{
    char *message = "Hello from kernel!\n";
    int message_len = strlen(message);
    int bytes_read = 0;

    if (*offset >= message_len)
        return 0;

    bytes_read = message_len - *offset;
    if (bytes_read > len)
        bytes_read = len;

    if (copy_to_user(buffer, message + *offset, bytes_read))
        return -EFAULT;

    *offset += bytes_read;
    return bytes_read;
}

static ssize_t dev_write(struct file *filep, const char __user *buffer,
                         size_t len, loff_t *offset)
{
    printk(KERN_INFO "Received %zu bytes from user\n", len);
    return len;
}

module_init(chardev_init);
module_exit(chardev_exit);

Key Concepts:

• register_chrdev(): Registers character device, kernel assigns major number
• class_create() and device_create(): Create sysfs entries, udev automatically creates /dev/mychardev
• file_operations structure: Defines callbacks for file operations
• copy_to_user() and copy_from_user(): Safe data transfer between kernel and user space

Testing the Character Device

# Load the module
sudo insmod mychardev.ko

# Verify device file created
ls -l /dev/mychardev

# Read from device
cat /dev/mychardev

# Write to device
echo "Hello" > /dev/mychardev

# Check kernel log
dmesg | tail

# Unload module
sudo rmmod mychardev

Kernel Debugging Techniques

Using printk for Debugging

The simplest debugging technique involves strategic printk() statements logging execution flow and variable values.

printk(KERN_DEBUG "Function %s called with value=%d\n", __func__, value);
printk(KERN_DEBUG "Pointer address: %p, value: %d\n", ptr, *ptr);

Dynamic Debug: The kernel’s dynamic debug feature allows enabling/disabling debug messages at runtime without recompiling:

# Enable debug messages for specific file
echo 'file mydriver.c +p' > /sys/kernel/debug/dynamic_debug/control

# Enable for specific function
echo 'func my_function +p' > /sys/kernel/debug/dynamic_debug/control

# Disable
echo 'file mydriver.c -p' > /sys/kernel/debug/dynamic_debug/control

Using kgdb and gdb

KGDB enables remote debugging of the kernel using GDB, providing breakpoints, single-stepping, and variable inspection.

Setup:

# Boot kernel with kgdb parameters
kgdboc=ttyS0,115200 kgdbwait

# Connect from another machine
gdb vmlinux
(gdb) target remote /dev/ttyS0
(gdb) continue

Kernel Crash Analysis

When kernels crash, they produce oops messages containing register dumps, stack traces, and other diagnostic information.

Analyzing Oops:

# Decode oops with symbols
./scripts/decode_stacktrace.sh < oops.txt vmlinux

# Use addr2line for specific addresses
addr2line -e vmlinux address

Tracing Tools

Ftrace: Function tracer built into the kernel:

# Enable function tracing
echo function > /sys/kernel/debug/tracing/current_tracer

# Trace specific function
echo my_function > /sys/kernel/debug/tracing/set_ftrace_filter

# View trace
cat /sys/kernel/debug/tracing/trace

perf: Performance analysis tool:

# Record kernel events
sudo perf record -a -g sleep 10

# Analyze report
sudo perf report

Contributing to the Linux Kernel

Understanding the Development Process

Linux kernel development follows established processes ensuring quality and stability. This Linux kernel tutorial section covers community participation.

Development Cycle: The Linux kernel constantly evolves, with new versions released approximately every 9-10 weeks. Each release cycle includes:

1. Merge window (2 weeks): Major changes merged into mainline
2. RC phase (~7 weeks): Release candidates with bug fixes only
3. Release: Final version tagged, cycle repeats

Mailing Lists: Kernel development occurs primarily through email on linux-kernel@vger.kernel.org and subsystem-specific lists. KernelNewbies provides helpful mailing lists where you can ask almost any type of basic kernel development question.

Creating and Submitting Patches

Patch Format: Kernel patches follow specific formatting requirements:

# Make changes to kernel source
# Commit with descriptive message
git commit -s

# Generate patch
git format-patch -1

# Check patch formatting
./scripts/checkpatch.pl 0001-your-patch.patch

Patch Content Requirements:

• Subject line: Brief description (50-75 characters)
• Body: Detailed explanation of what and why
• Signed-off-by: Developer’s certification of origin
• Follow coding style (Documentation/process/coding-style.rst)

Submission Process:

# Send patch via email
git send-email --to=maintainer@email.com \
               --cc=linux-kernel@vger.kernel.org \
               0001-your-patch.patch

Coding Style Guidelines

The Linux kernel enforces strict coding style ensuring consistency across millions of lines of code.

Key Style Rules:

• Use tabs (8 spaces wide), not spaces for indentation
• Line length: 80 characters preferred, 100 maximum
• Braces: Opening brace on same line (except functions)
• Variable names: descriptive, lowercase with underscores
• Function names: lowercase, descriptive
• Comments: /* */ style, not // (except special cases)

Check Your Code:

# Run checkpatch script
./scripts/checkpatch.pl --file mydriver.c

# Check spelling
./scripts/checkpatch.pl --codespell mydriver.c

Advanced Kernel Programming Topics

Synchronization and Locking

Kernel code often runs concurrently on multiple CPUs or can be interrupted, requiring synchronization primitives to protect shared data.

Spinlocks: For short critical sections, spinlocks provide efficient mutual exclusion:

#include <linux/spinlock.h>

static DEFINE_SPINLOCK(my_lock);

void critical_section(void)
{
    unsigned long flags;
    
    spin_lock_irqsave(&my_lock, flags);
    // Critical section - interrupts disabled
    // Access shared data here
    spin_unlock_irqrestore(&my_lock, flags);
}

Mutexes: For longer critical sections that can sleep:

#include <linux/mutex.h>

static DEFINE_MUTEX(my_mutex);

void sleepable_critical_section(void)
{
    mutex_lock(&my_mutex);
    // Can sleep here if needed
    // Access shared data
    mutex_unlock(&my_mutex);
}

Semaphores: Counting semaphores allow multiple concurrent accessors:

#include <linux/semaphore.h>

static DEFINE_SEMAPHORE(my_sem);

void limited_access(void)
{
    down(&my_sem);  // Acquire
    // Limited concurrent access
    up(&my_sem);    // Release
}

Read-Write Locks: Optimize for read-heavy workloads:

#include <linux/rwlock.h>

static DEFINE_RWLOCK(my_rwlock);

void reader(void)
{
    read_lock(&my_rwlock);
    // Multiple readers allowed simultaneously
    read_unlock(&my_rwlock);
}

void writer(void)
{
    write_lock(&my_rwlock);
    // Exclusive access for writers
    write_unlock(&my_rwlock);
}

Memory Allocation in Kernel

Kernel memory allocation differs significantly from user space due to operating in fixed memory without paging.

kmalloc: General-purpose allocation for small objects:

#include <linux/slab.h>

void *buffer;

// Allocate memory
buffer = kmalloc(size, GFP_KERNEL);
if (!buffer) {
    printk(KERN_ERR "Memory allocation failed\n");
    return -ENOMEM;
}

// Use buffer

// Free memory
kfree(buffer);

GFP Flags:

• GFP_KERNEL: Standard allocation, can sleep
• GFP_ATOMIC: Cannot sleep, use in interrupt context
• GFP_DMA: Allocate DMA-capable memory
• __GFP_ZERO: Zero-initialize allocated memory

vmalloc: For large virtually contiguous allocations:

#include <linux/vmalloc.h>

void *large_buffer = vmalloc(large_size);
if (!large_buffer) {
    return -ENOMEM;
}

// Use buffer
vfree(large_buffer);

get_free_pages: Direct page allocation:

unsigned long page = __get_free_pages(GFP_KERNEL, order);
// order determines size: 2^order pages allocated

free_pages(page, order);

Interrupt Handling

Device drivers must handle hardware interrupts, requiring special programming considerations.

Registering Interrupt Handler:

#include <linux/interrupt.h>

static irqreturn_t my_interrupt_handler(int irq, void *dev_id)
{
    // Quick processing in interrupt context
    // Schedule bottom half for longer work
    
    return IRQ_HANDLED;
}

static int __init device_init(void)
{
    int ret;
    
    ret = request_irq(irq_number,
                      my_interrupt_handler,
                      IRQF_SHARED,
                      "mydevice",
                      dev_id);
    if (ret) {
        printk(KERN_ERR "Cannot register IRQ\n");
        return ret;
    }
    
    return 0;
}

static void __exit device_exit(void)
{
    free_irq(irq_number, dev_id);
}

Top Half vs Bottom Half: Interrupt handlers should execute quickly. Long operations belong in bottom halves implemented via workqueues or tasklets.

Workqueues:

#include <linux/workqueue.h>

static struct workqueue_struct *my_wq;
static DECLARE_WORK(my_work, work_handler);

static void work_handler(struct work_struct *work)
{
    // Lengthy processing in process context
    // Can sleep here
}

// Schedule work from interrupt
queue_work(my_wq, &my_work);

Kernel Timers

Timers schedule function execution after specified delays.

#include <linux/timer.h>

static struct timer_list my_timer;

static void timer_callback(struct timer_list *t)
{
    printk(KERN_INFO "Timer expired\n");
    
    // Reschedule if needed
    mod_timer(&my_timer, jiffies + msecs_to_jiffies(5000));
}

static int __init timer_init(void)
{
    timer_setup(&my_timer, timer_callback, 0);
    
    // Schedule timer 5 seconds from now
    mod_timer(&my_timer, jiffies + msecs_to_jiffies(5000));
    
    return 0;
}

static void __exit timer_exit(void)
{
    del_timer(&my_timer);
}

Kernel Threads

Long-running kernel tasks execute in kernel threads.

#include <linux/kthread.h>

static struct task_struct *thread;

static int thread_function(void *data)
{
    while (!kthread_should_stop()) {
        // Do work
        printk(KERN_INFO "Kernel thread running\n");
        
        // Sleep to avoid consuming CPU
        msleep(1000);
    }
    
    return 0;
}

static int __init kthread_init(void)
{
    thread = kthread_run(thread_function, NULL, "my_thread");
    if (IS_ERR(thread)) {
        printk(KERN_ERR "Failed to create thread\n");
        return PTR_ERR(thread);
    }
    
    return 0;
}

static void __exit kthread_exit(void)
{
    kthread_stop(thread);
}

Kernel Data Structures

Linked Lists

The kernel provides a generic doubly-linked list implementation used throughout the codebase.

#include <linux/list.h>

struct my_data {
    int value;
    struct list_head list;
};

static LIST_HEAD(my_list);

// Add entry
struct my_data *new_entry = kmalloc(sizeof(*new_entry), GFP_KERNEL);
new_entry->value = 42;
list_add_tail(&new_entry->list, &my_list);

// Iterate
struct my_data *entry;
list_for_each_entry(entry, &my_list, list) {
    printk(KERN_INFO "Value: %d\n", entry->value);
}

// Remove entry
list_del(&entry->list);
kfree(entry);

Hash Tables

For efficient lookups, the kernel provides hash table implementations.

#include <linux/hashtable.h>

DEFINE_HASHTABLE(my_hash, 10);  // 2^10 buckets

struct my_entry {
    int key;
    int value;
    struct hlist_node node;
};

// Add entry
struct my_entry *e = kmalloc(sizeof(*e), GFP_KERNEL);
e->key = 123;
e->value = 456;
hash_add(my_hash, &e->node, e->key);

// Lookup
struct my_entry *found;
hash_for_each_possible(my_hash, found, node, key) {
    if (found->key == key) {
        printk(KERN_INFO "Found value: %d\n", found->value);
        break;
    }
}

// Remove
hash_del(&found->node);
kfree(found);

Red-Black Trees

Self-balancing binary search trees for ordered data.

#include <linux/rbtree.h>

struct my_node {
    int key;
    struct rb_node node;
};

static struct rb_root my_tree = RB_ROOT;

// Insert
void insert_node(struct rb_root *root, struct my_node *new)
{
    struct rb_node **link = &root->rb_node;
    struct rb_node *parent = NULL;
    
    while (*link) {
        struct my_node *entry = rb_entry(*link, struct my_node, node);
        parent = *link;
        
        if (new->key < entry->key)
            link = &(*link)->rb_left;
        else
            link = &(*link)->rb_right;
    }
    
    rb_link_node(&new->node, parent, link);
    rb_insert_color(&new->node, root);
}

// Search
struct my_node *search(struct rb_root *root, int key)
{
    struct rb_node *node = root->rb_node;
    
    while (node) {
        struct my_node *entry = rb_entry(node, struct my_node, node);
        
        if (key < entry->key)
            node = node->rb_left;
        else if (key > entry->key)
            node = node->rb_right;
        else
            return entry;
    }
    
    return NULL;
}

System Calls

Understanding System Call Interface

System calls provide the interface through which user-space applications request kernel services. This critical boundary separates unprivileged user code from privileged kernel operations.

System Call Mechanism: When applications invoke system calls like read(), write(), or open(), the C library translates these into architecture-specific instructions triggering a transition from user mode to kernel mode. On x86-64, this uses the syscall instruction.

System Call Table: The kernel maintains a system call table mapping system call numbers to handler functions. When system calls execute, the kernel looks up the appropriate handler and invokes it with provided arguments.

Adding a Custom System Call

While not commonly done, understanding how to add system calls provides deep insight into the kernel-user space interface.

Define System Call:

// In kernel/sys.c or similar location
SYSCALL_DEFINE1(mysyscall, int, value)
{
    printk(KERN_INFO "Custom syscall called with value=%d\n", value);
    return value * 2;
}

Register System Call Number: Add entry to system call table for your architecture (e.g., arch/x86/entry/syscalls/syscall_64.tbl).

User Space Invocation:

#include <unistd.h>
#include <sys/syscall.h>

#define __NR_mysyscall 440  // Your syscall number

int main()
{
    long result = syscall(__NR_mysyscall, 21);
    printf("Result: %ld\n", result);
    return 0;
}

Conclusion: Your Kernel Development Journey

This comprehensive Linux kernel tutorial has guided you through the fundamental concepts, practical techniques, and advanced topics essential for kernel development. You’ve explored kernel architecture, built custom kernels, written loadable modules, created device drivers, and learned debugging techniques that professional kernel developers use daily.

The Linux kernel represents one of computing’s most sophisticated and successful collaborative projects. Its millions of lines of code, contributed by thousands of developers worldwide, power billions of devices from smartphones to supercomputers. By understanding kernel internals and development practices, you join this global community of systems programmers advancing the state of operating system technology.

Key Takeaways from This Tutorial:

• The kernel mediates between hardware and applications, managing resources and providing services
• Understanding kernel subsystems (process management, memory management, VFS, device drivers, networking) reveals how modern operating systems work
• Kernel modules provide a practical entry point for kernel programming without requiring full kernel compilation
• Device drivers connect the kernel to hardware, implementing standardized interfaces for diverse devices
• Synchronization primitives, memory allocation, and interrupt handling require careful attention due to the concurrent, resource-constrained kernel environment
• Contributing to the Linux kernel involves following community processes, coding standards, and communication protocols

Continuing Your Learning:

The kernel is vast—this tutorial provides foundation, but deep expertise requires ongoing learning and practice:

• Read Kernel Code: The source code is the authoritative documentation. Browse areas that interest you, trace code paths, understand implementation choices
• Join the Community: Subscribe to linux-kernel mailing list, participate in discussions, attend conferences
• Start Small: Begin with simple bug fixes or documentation improvements before tackling major features
• Focus on a Subsystem: Specialize in an area like networking, filesystems, or specific hardware
• Keep Building: Regular practice through personal projects, contributions, and experimentation builds expertise

Resources for Further Learning:

• Linux Kernel Documentation (Documentation/ directory in source)
• KernelNewbies wiki and resources
• Linux Device Drivers book by Corbet, Rubini, and Kroah-Hartman
• Linux Kernel Development by Robert Love
• Understanding the Linux Kernel by Bovet and Cesati
• Linux kernel source code itself

The journey from kernel tutorial beginner to confident kernel developer requires patience, persistence, and curiosity. Embrace challenges as learning opportunities, celebrate small victories, and contribute to the incredible collaborative achievement that is the Linux kernel. Welcome to the world of systems programming!