Skip to content
Code Inside Out

AAOS - Build a Kernel Module

Vendor modules are not included in the kernel. They are built separately, then included into the system based on the vendor configurations.

For general Android Kernel, refer to https://source.android.com/docs/setup/build/building-kernels.

For Cuttlefish Device, refer to https://source.android.com/docs/devices/cuttlefish/kernel-dev.

Prebuilt Kernel


The AOSP tree contains only prebuilt kernel binaries. When and AVD image is built, it copies the prebuilt kernel image to the output folder.

Start from the main makefile:

mgrep "aosp_cf_x86_64_auto"
./device/google/cuttlefish/AndroidProducts.mk:32: aosp_cf_x86_64_auto:$(LOCAL_DIR)/vsoc_x86_64_only/auto/aosp_cf.mk \

This makefile inherates from other makefiles:

device/google/cuttlefish/vsoc_x86_64_only/auto/aosp_cf.mk
$(call inherit-product, $(SRC_TARGET_DIR)/product/core_64_bit_only.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_generic_system.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_system_ext.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_product.mk)
$(call inherit-product, device/google/cuttlefish/shared/auto/device_vendor.mk)
$(call inherit-product, device/google/cuttlefish/vsoc_x86_64/kernel.mk)
$(call inherit-product, device/google/cuttlefish/vsoc_x86_64/bootloader.mk)

Kernel is defined in the makefile:

device/google/cuttlefish/shared/BoardConfig.mk
TARGET_KERNEL_USE ?= 6.1
TARGET_KERNEL_ARCH ?= $(TARGET_ARCH)
SYSTEM_DLKM_SRC ?= kernel/prebuilts/$(TARGET_KERNEL_USE)/$(TARGET_KERNEL_ARCH)
TARGET_KERNEL_PATH ?= $(SYSTEM_DLKM_SRC)/kernel-$(TARGET_KERNEL_USE)
KERNEL_MODULES_PATH ?= \
kernel/prebuilts/common-modules/virtual-device/$(TARGET_KERNEL_USE)/$(subst _,-,$(TARGET_KERNEL_ARCH))
PRODUCT_COPY_FILES += $(TARGET_KERNEL_PATH):kernel

The variable TARGET_KERNEL_PATH is finally expanded to kernel/prebuilts/6.1/x86_64/kernel-6.1:

file kernel/prebuilts/6.1/x86_64/kernel-6.1
Linux kernel x86 boot executable bzImage, version 6.1.68-android14-11-ga7f647f49daf-ab11484633 (build-user@build-host) #1 SMP PREEMPT Wed Feb 21 00:52:15 UTC 2024, RO-rootFS, swap_dev 0X11, Normal VGA

Compile Kernel


Android Kernel is hosted in a different repo from the AOSP project. Check the branches and their build systems at https://source.android.com/docs/setup/reference/bazel-support.

  • Android 13 introduced building kernels with Bazel (kleaf) - it is super fast
  • Older Android versions use build.sh, which is quite slow
  • From Android 11, Vendor modules are recommended to built separately in common-modules

For Cuttlefish Device, refer to https://source.android.com/docs/devices/cuttlefish/kernel-dev.

Cuttlefish supports the following kernel manifests on aosp-main repo,
branches: common-android14-6.1, common-android14-5.15, common-android-mainline

Define macros:

export AOSP_HOME="$HOME/aosp"
export KERNEL_BRANCH="common-android14-6.1"

Initialize the repo:

mkdir -p ${AOSP_HOME}/${KERNEL_BRANCH}
cd ${AOSP_HOME}/${KERNEL_BRANCH}
repo init --depth=1 \
-u https://android.googlesource.com/kernel/manifest \
-b ${KERNEL_BRANCH}

Download source code:

repo sync -c -j$(nproc)

Cuttlefish Device is a kernel build named virtual_device_x86_64 which is based on kernel_x86_64:

common-modules/virtual-device/BUILD.bazel
kernel_build(
name = "virtual_device_x86_64",
srcs = [":virtual_device_x86_64_common_sources"],
outs = [],
arch = "x86_64",
base_kernel = "//common:kernel_x86_64",
build_config = "build.config.virtual_device.x86_64",
make_goals = [
"modules",
],
module_outs = _VIRT_COMMON_MODULES + [
# keep sorted
"dummy-cpufreq.ko",
"nd_virtio.ko",
"test_meminit.ko",
"virtio_pmem.ko",
],
)

Build Virtual Device Kernel and Modules:

tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist
[dist] INFO: Copying to /home/trongvq/aosp/kernel-common-android14-6.1/out/virtual_device_x86_64/dist

All modules will be included in InitRamFS.

Check the Kernel image:

file out/virtual_device_x86_64/dist/bzImage
Linux kernel x86 boot executable bzImage, version 6.1.99-android14-11-maybe-dirty (build-user@build-host) #1 SMP PREEMPT Thu Jan 1 00:00:00 UTC 1970, RO-rootFS, swap_dev 0X11, Normal VGA

Run with new Kernel


Cuttlefish Device


It is easy to tell Cuttlefish to use new kernel and initramfs (included all virtual kernel modules):

export AOSP_HOME="$HOME/aosp"
export KERNEL_BRANCH="common-android14-6.1"
launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img

Cuttlefish

Android Virtual Device


This needs to be rebuilt to run with new kernel:

device/generic/goldfish/board/kernel/x86_64.mk
TARGET_KERNEL_USE ?= 6.1
KERNEL_ARTIFACTS_PATH := /home/trongvq/aosp/common-android14-6.1/out/virtual_device_x86_64/dist
VIRTUAL_DEVICE_KERNEL_MODULES_PATH := $(KERNEL_ARTIFACTS_PATH)
EMULATOR_KERNEL_FILE := $(KERNEL_ARTIFACTS_PATH)/bzImage

then run emulator as usual.

Emulator

Compilation may fail due to a missing kernel module, remove it from makefile and build again.

Kernel Modules


Modules are pieces of code that can be loaded and unloaded into the kernel upon demand. HAL implementations are packaged into modules and loaded by the Android system at the appropriate time. A kernel module can be a device driver that handles or manages a hardware

As part of the module kernel requirements introduced in Android 8.0, all system-on-chip (SoC) kernels must support loadable kernel modules.

To support loadable kernel modules, android-base.cfg in all common kernels includes the following kernel-config options:

CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y

All device kernels must enable these options. Kernel modules should also support unloading and reloading whenever possible.

Kernel Menu Config


Refer to: https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/.

Below command will run a menu to configure internel modules in kernel itself. External/Vendor modules do not listed in Kernel Menu Config.

tools/bazel run //common:kernel_x86_64_config -- menuconfig

Vendor Modules


Refer to: https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/docs/ddk/main.md.

From Android 11, Vendor Modules are recommended to built separately. One part of Kleaf is the Driver Development Kit (DDK) which is used to build external modules

Here is a simple vendor module structure:

  • Directorycommon-android14-6.1
    • Directoryvendor
      • Directorytrongvq # vendor name
        • BUILD.bazel # build file
        • Directoryhello_world # vendor module
          • hello_world.c

Build a Module

  1. Create a folder for vendor packages:

    mkdir vendor/trongvq
  2. Create a folder for venfor module and its source code:

    mkdir vendor/trongvq/hello_world
    vendor/trongvq/hello_world/hello_world.c
    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/init.h>
    static int hello_world_init(void) {
    printk(KERN_ALERT "trongvq: Hello World!\n");
    return 0;
    }
    static void hello_world_exit(void) {
    printk(KERN_ALERT "trongvq: Goodbye World!\n");
    }
    module_init(hello_world_init);
    module_exit(hello_world_exit);
    MODULE_LICENSE("GPL");
  3. Create a ddk_module so system will know that there is a module there,
    this module is built for kernel named virtual_device_x86_64:

    vendor/trongvq/hello_world/BUILD.bazel
    load("//build/kernel/kleaf:kernel.bzl", "ddk_module")
    filegroup(
    name = "hello_world_sources",
    srcs = glob([
    "hello_world/*.c",
    ]),
    )
    ddk_module(
    name = "hello_world",
    srcs = [":hello_world_sources"],
    out = "hello_world.ko",
    kernel_build = "//common-modules/virtual-device:virtual_device_x86_64",
    deps = [
    "//common:all_headers_x86_64",
    ],
    )
  4. Compile the vendor module:

    tools/bazel build //vendor/trongvq:hello_world
    Output: bazel-bin/vendor/trongvq/hello_world/hello_world.ko
  5. Load and test module on device:

    adb push bazel-bin/vendor/trongvq/hello_world/hello_world.ko /data/local/tmp/
    adb root
    adb shell insmod /data/local/tmp/hello_world.ko
    adb shell dmesg | grep trongvq
    trongvq: Hello World!
    adb shell rmmod /data/local/tmp/hello_world.ko
    adb shell dmesg | grep trongvq
    trongvq: Hello World!
    trongvq: Goodbye World!

Integrate Module to Kernel

  1. Create a kernel_module_group and make it visible as a package __pkg__
    for the target kernel build //common-modules/virtual-device:

    load("//build/kernel/kleaf:kernel.bzl", "ddk_module")
    load("//build/kernel/kleaf:kernel.bzl", "kernel_module_group")
    filegroup(
    name = "hello_world_sources",
    srcs = glob([
    "hello_world/*.c",
    ]),
    )
    ddk_module(
    name = "hello_world",
    srcs = [":hello_world_sources"],
    out = "hello_world.ko",
    kernel_build = "//common-modules/virtual-device:virtual_device_x86_64",
    deps = [
    "//common:all_headers_x86_64",
    ],
    )
    kernel_module_group(
    name = "kernel_modules",
    srcs = [
    ":hello_world"
    ],
    visibility = ["//common-modules/virtual-device:__pkg__"],
    )
  2. In the target kernel build, make kernel_build visible for vendor packages,
    and under a kernel_modules_install where it lists kernel modules,
    add the vendor kernel_module_group into the list:

    common-modules/virtual-device/BUILD.bazel
    kernel_build(
    name = "virtual_device_x86_64",
    ...
    visibility = [
    "//vendor/trongvq:__pkg__",
    ],
    )
    kernel_modules_install(
    name = "virtual_device_x86_64_modules_install",
    kernel_build = ":virtual_device_x86_64",
    kernel_modules = [
    ":virtual_device_x86_64_external_modules",
    "//vendor/trongvq:kernel_modules"
    ],
    )
  3. Compile the target kernel again:

    tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist

    Now, the vendor module should be copied into out/virtual_device_x86_64/dist/hello_world.ko

  4. Load Cuttlefish again:

    export AOSP_HOME="$HOME/aosp"
    export KERNEL_BRANCH="common-android14-6.1"
    launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img
  5. Check that vendor module is loaded automatically:

    adb root
    adb shell lsmode | grep hello_world
    hello_world 16384 0

    Note that module is loaded in InitRamFS, there is no init message printed in dmesg.

  6. Check that vendor module is included in filesystem:

    adb shell find vendor* -name hello*
    vendor_dlkm/lib/modules/extra/vendor/trongvq/hello_world.ko

Some notes here:

  • the kerner target //common:kernel_x86_64 is a base build,
    the target kernel //common-modules/virtual-device:virtual_device_x86_64 is defined with addition modules
  • if the vendor kernel is build for //common:kernel_x86_64, the need changes in the build for /common/BUILD.bazel in the section which defines kernel_x86_64, but it needs more work.
  • for older Android version, please refer to this guide: https://vuquangtrong.github.io/blog/android/kernel-module.

Example: virtio_rtc Module


This is an example of a character device which mimics a virtual Real Time Clock (RTC) with only get, set, and tick functions.

  • Directorycommon-android14-6.1
    • Directorycommon-modules/virtual-device # externa modules
      • BUILD.bazek # cuttlefish kernel target
    • Directoryvendor/trongvq # vendor name
      • BUILD.bazel # vendor build file
      • Directoryvirtio_rtc # module name
        • virtio_rtc.c # module implementation

Main functions implementation:

virtio_rtc.c
#define DEVICE_NAME "virtio_rtc"
#define CLASS_NAME "virtio"
#define BUFFER_SIZE 16
...
static long seconds = 0; // unprotected shared data!
static char str_seconds[BUFFER_SIZE]; // unprotected shared data!
static int thread_func(void *arg) {
while (!kthread_should_stop()) {
ssleep(1);
seconds++;
}
return 0;
}
...
static ssize_t dev_read(struct file *filep, char *buffer, size_t len, loff_t *offset) {
snprintf(str_seconds, BUFFER_SIZE, "%ld\n", seconds);
return simple_read_from_buffer(buffer, len, offset, str_seconds, BUFFER_SIZE);
}
static ssize_t dev_write(struct file *filep, const char *buffer, size_t len, loff_t *offset) {
ssize_t s = simple_write_to_buffer(str_seconds, BUFFER_SIZE, offset, buffer, len);
if (kstrtol(str_seconds, 10, &seconds)) {
return -EINVAL;
}
return s;
}
...
static int __init virtio_rtc_init(void) {
...
task = kthread_run(thread_func, NULL, "virtio_rtc_thread");
if (IS_ERR(task)) {
printk(KERN_ALERT DEVICE_NAME "Failed to create the thread\n");
return PTR_ERR(task);
}
}
...

Rebuild kernel with vendor modules:

tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist

Start Cuttlefish device:

export AOSP_HOME="$HOME/aosp"
export KERNEL_BRANCH="common-android14-6.1"
source build/envsetup.sh
lunch aosp_cf_x86_64_auto-trunk_staging-userdebug
launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img

Access to the device’s shell as root:

adb shell
su

Read from and Write to the /dev/virtio_rtc device:

cat /dev/virtio_rtc
echo 100 > /dev/virtio_rtc