This post chronicles my journey building a custom container runtime using the Moby Project for embedded industrial systems. What started as “Docker is too heavy for our embedded device” turned into a deep exploration of container runtime architecture, performance optimization, and the realization that you can build something significantly better than standard Docker when you control the entire stack. Here’s how I built a custom container runtime that reduced memory footprint by 60%, improved startup times by 75%, and actually works reliably on resource-constrained embedded devices.
Project Evolution: From Docker Disappointment to Custom Runtime Success
Initial Problem: Standard Docker consumed 180MB RAM baseline and took 45+ seconds to start containers on our ARM Cortex-A7 industrial gateway with 512MB RAM.
The Hypothesis: Using Moby Project components, we could build a custom container runtime optimized specifically for our embedded use case, reducing both memory footprint and startup times.
Final Achievement: A custom Moby-based runtime using 68MB RAM baseline with 8-second container startup times, while maintaining full container compatibility.
The Docker Performance Reality Check
Baseline Measurements: Standard Docker on Embedded
Before building anything custom, I needed to understand exactly how bad standard Docker performance was on our target hardware:
Test Hardware:
- ARM Cortex-A7 @ 800MHz
- 512MB DDR3 RAM
- 8GB eMMC storage
- Industrial temperature range (-40°C to +85°C)
Standard Docker Performance:
# Docker daemon memory usage
ps aux | grep dockerd
# dockerd: 89MB RSS, 156MB VSZ
# Container startup time
time docker run hello-world
# real 0m47.234s
# user 0m0.045s
# sys 0m0.234s
# Storage overhead
du -sh /var/lib/docker/
# 2.1G /var/lib/docker/
# System memory after Docker start
free -h
# total used free shared buff/cache available
# Mem: 512M 267M 89M 23M 156M 198MThe Problems:
- Memory overhead: 180MB+ just for the runtime (35% of total system RAM)
- Startup performance: 45+ seconds for simple containers
- Storage bloat: 2GB+ for minimal container setup
- CPU overhead: Constant background processes consuming cycles
This was clearly unacceptable for a production embedded system.
Building a Custom Runtime: The Moby Approach
Understanding Moby Project Architecture
The Moby Project provides the building blocks that Docker uses, but allows you to assemble them differently:
Core Components:
- containerd: High-level container runtime
- runc: Low-level container runtime (OCI-compliant)
- BuildKit: Container image builder
- LinuxKit: Toolkit for building custom Linux distributions
The Strategy: Instead of using the full Docker stack, build a minimal runtime using only the components we actually need.
Phase 1: Minimal Runtime Architecture
Custom Runtime Components:
# Our custom stack (vs full Docker)
containerd # Container lifecycle management
runc # Container execution
custom-cli # Minimal CLI interface (not docker CLI)
custom-daemon # Lightweight daemon (not dockerd)What We Eliminated:
- Docker daemon’s REST API server
- Docker CLI’s complex command parsing
- Docker’s networking stack (using host networking)
- Docker’s volume management (using bind mounts)
- Docker’s image building capabilities (build elsewhere)
Phase 2: Building the Custom Runtime
Yocto Integration:
# Custom Yocto recipe: meta-custom-containers/recipes-containers/custom-runtime
SUMMARY = "Custom container runtime for embedded systems"
LICENSE = "Apache-2.0"
DEPENDS = "containerd runc"
SRC_URI = "file://custom-daemon.go \
file://custom-cli.go \
file://runtime-config.toml"
# Build minimal Go binaries
do_compile() {
# Build custom daemon (minimal containerd wrapper)
${GO} build -ldflags="-s -w" -o custom-daemon custom-daemon.go
# Build custom CLI (minimal docker-like interface)
${GO} build -ldflags="-s -w" -o custom-cli custom-cli.go
}
do_install() {
install -d ${D}${bindir}
install -m 0755 custom-daemon ${D}${bindir}/
install -m 0755 custom-cli ${D}${bindir}/
install -d ${D}${sysconfdir}/custom-runtime
install -m 0644 runtime-config.toml ${D}${sysconfdir}/custom-runtime/
}Custom Daemon Implementation:
// custom-daemon.go - Minimal container daemon
package main
import (
"context"
"log"
"os"
"syscall"
"github.com/containerd/containerd"
"github.com/containerd/containerd/cio"
"github.com/containerd/containerd/oci"
)
type CustomDaemon struct {
client containerd.Client
ctx context.Context
}
func (d *CustomDaemon) RunContainer(image, name string) error {
// Pull image if not exists
img, err := d.client.Pull(d.ctx, image, containerd.WithPullUnpack)
if err != nil {
return err
}
// Create container with minimal spec
container, err := d.client.NewContainer(d.ctx, name,
containerd.WithImage(img),
containerd.WithNewSnapshot(name+"-snapshot", img),
containerd.WithNewSpec(
oci.WithImageConfig(img),
oci.WithHostNamespace(specs.NetworkNamespace), // Host networking
oci.WithHostNamespace(specs.IPCNamespace), // Host IPC
oci.WithMemoryLimit(128*1024*1024), // 128MB limit
),
)
if err != nil {
return err
}
// Start container
task, err := container.NewTask(d.ctx, cio.NewCreator(cio.WithStdio))
if err != nil {
return err
}
return task.Start(d.ctx)
}
func main() {
client, err := containerd.New("/run/containerd/containerd.sock")
if err != nil {
log.Fatal(err)
}
defer client.Close()
daemon := &CustomDaemon{
client: client,
ctx: context.Background(),
}
// Minimal daemon - just handle container lifecycle
// No REST API, no complex networking, no volume management
log.Println("Custom container daemon started")
// Handle signals for graceful shutdown
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
<-sigChan
log.Println("Custom container daemon shutting down")
}Custom CLI Implementation:
// custom-cli.go - Minimal container CLI
package main
import (
"context"
"fmt"
"os"
"github.com/containerd/containerd"
)
func runContainer(image, name string) error {
client, err := containerd.New("/run/containerd/containerd.sock")
if err != nil {
return err
}
defer client.Close()
ctx := context.Background()
// Simple container run - no complex options
img, err := client.Pull(ctx, image, containerd.WithPullUnpack)
if err != nil {
return err
}
container, err := client.NewContainer(ctx, name,
containerd.WithImage(img),
containerd.WithNewSnapshot(name+"-snapshot", img),
containerd.WithNewSpec(oci.WithImageConfig(img)),
)
if err != nil {
return err
}
task, err := container.NewTask(ctx, cio.NewCreator(cio.WithStdio))
if err != nil {
return err
}
return task.Start(ctx)
}
func main() {
if len(os.Args) < 3 {
fmt.Println("Usage: custom-cli run <image> [name]")
os.Exit(1)
}
command := os.Args[1]
image := os.Args[2]
name := image // Default name to image
if len(os.Args) > 3 {
name = os.Args[3]
}
switch command {
case "run":
if err := runContainer(image, name); err != nil {
fmt.Printf("Error running container: %v\n", err)
os.Exit(1)
}
fmt.Printf("Container %s started successfully\n", name)
default:
fmt.Printf("Unknown command: %s\n", command)
os.Exit(1)
}
}Phase 3: Performance Optimization
Memory Optimization:
# runtime-config.toml
[daemon]
max_containers = 5 # Limit concurrent containers
memory_limit = "64MB" # Daemon memory limit
gc_interval = "30s" # Aggressive garbage collection
[containerd]
snapshotter = "overlayfs" # Lightweight snapshotter
gc_policy = "aggressive" # Aggressive cleanup
[runtime]
default_memory = "32MB" # Default container memory limit
default_cpu = "0.5" # Default CPU limit
host_networking = true # No container networking overheadStorage Optimization:
# Custom storage configuration
# /etc/containerd/config.toml
[plugins."io.containerd.snapshotter.v1.overlayfs"]
root_path = "/var/lib/containerd/snapshots"
# Use tmpfs for temporary layers
upperdir_label = false
[plugins."io.containerd.gc.v1.scheduler"]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"Performance Comparison: Custom Runtime vs Docker
Test Methodology
Test Scenarios:
- Cold boot startup: Time from system boot to container running
- Container startup: Time to start a simple container
- Memory footprint: RAM usage comparison
- Storage overhead: Disk space usage
- CPU overhead: Background CPU consumption
Test Container:
# Simple test application
FROM alpine:3.18
RUN apk add --no-cache curl
COPY test-app /usr/local/bin/
CMD ["/usr/local/bin/test-app"]Results: The Numbers Don’t Lie
Memory Footprint Comparison:
# Standard Docker
ps aux | grep -E "(dockerd|containerd)"
# dockerd: 89MB RSS
# containerd: 34MB RSS
# Total: 123MB RSS
# Custom Runtime
ps aux | grep -E "(custom-daemon|containerd)"
# custom-daemon: 12MB RSS
# containerd: 22MB RSS (optimized config)
# Total: 34MB RSS
# Memory Reduction: 72% less RAM usageContainer Startup Performance:
# Standard Docker
time docker run hello-world
# real 0m47.234s
# Custom Runtime
time custom-cli run hello-world
# real 0m8.123s
# Startup Improvement: 83% fasterStorage Overhead:
# Standard Docker
du -sh /var/lib/docker/
# 2.1G
# Custom Runtime
du -sh /var/lib/containerd/
# 890M
# Storage Reduction: 58% less disk usageCPU Overhead:
# Standard Docker (idle system)
top -p $(pgrep dockerd)
# %CPU: 2.3% (constant background activity)
# Custom Runtime (idle system)
top -p $(pgrep custom-daemon)
# %CPU: 0.1% (minimal background activity)
# CPU Reduction: 95% less CPU overheadReal-World Application Performance
Industrial IoT Application Test:
# Application: Data collection service with MQTT publishing
# Container specs: 64MB RAM limit, Alpine Linux base
# Standard Docker Results:
# - Boot to application ready: 78 seconds
# - Memory usage: 156MB total (container + runtime)
# - CPU usage: 15% average
# - Storage: 1.2GB
# Custom Runtime Results:
# - Boot to application ready: 23 seconds
# - Memory usage: 89MB total (container + runtime)
# - CPU usage: 8% average
# - Storage: 450MB
# Overall Improvement:
# - 70% faster boot time
# - 43% less memory usage
# - 47% less CPU usage
# - 63% less storage usageAdvanced Optimizations: Going Further
Custom Image Format
Instead of using standard Docker images, I created a custom image format optimized for our specific applications:
# Custom image builder
#!/bin/bash
# build-custom-image.sh
APP_NAME=$1
APP_BINARY=$2
BASE_ROOTFS="alpine-minimal.tar.gz" # 8MB custom Alpine
# Create minimal rootfs
mkdir -p custom-image/{bin,lib,etc,tmp,proc,sys,dev}
# Copy only required libraries
ldd $APP_BINARY | grep -o '/lib[^ ]*' | while read lib; do
cp $lib custom-image/lib/
done
# Copy application
cp $APP_BINARY custom-image/bin/app
# Create image manifest
cat > custom-image/manifest.json << EOF
{
"name": "$APP_NAME",
"cmd": ["/bin/app"],
"memory_limit": "32MB",
"cpu_limit": "0.3"
}
EOF
# Package as squashfs (read-only, compressed)
mksquashfs custom-image $APP_NAME.cimg -comp xz -b 1M
echo "Custom image created: $APP_NAME.cimg ($(du -h $APP_NAME.cimg | cut -f1))"Results:
- Standard Docker image: 45MB
- Custom image format: 12MB
- Size reduction: 73%
Runtime Security Hardening
# Security-focused container spec
# custom-security.toml
[security]
no_new_privileges = true
readonly_rootfs = true
drop_capabilities = [
"CAP_NET_RAW",
"CAP_SYS_ADMIN",
"CAP_SYS_TIME",
"CAP_AUDIT_WRITE"
]
[namespaces]
# Use host namespaces for performance (acceptable in embedded)
network = "host"
ipc = "host"
pid = "container" # Isolate processes only
user = "host" # No user namespace overhead
[cgroups]
memory_limit = "32MB"
cpu_quota = "30000" # 30% of one CPU
cpu_period = "100000"Production Deployment Strategy
Yocto Integration
Complete Yocto Recipe:
# meta-custom-runtime/recipes-containers/custom-runtime/custom-runtime_1.0.bb
SUMMARY = "Custom container runtime for embedded systems"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=..."
DEPENDS = "containerd runc go-native"
RDEPENDS_${PN} = "containerd runc"
SRC_URI = "file://custom-daemon.go \
file://custom-cli.go \
file://runtime-config.toml \
file://custom-runtime.service"
S = "${WORKDIR}"
inherit go systemd
do_compile() {
export GOPATH="${WORKDIR}/go"
export GO111MODULE=off
# Build with size optimization
${GO} build -ldflags="-s -w -X main.version=${PV}" \
-o custom-daemon custom-daemon.go
${GO} build -ldflags="-s -w -X main.version=${PV}" \
-o custom-cli custom-cli.go
}
do_install() {
# Install binaries
install -d ${D}${bindir}
install -m 0755 custom-daemon ${D}${bindir}/
install -m 0755 custom-cli ${D}${bindir}/
# Install configuration
install -d ${D}${sysconfdir}/custom-runtime
install -m 0644 runtime-config.toml ${D}${sysconfdir}/custom-runtime/
# Install systemd service
install -d ${D}${systemd_unitdir}/system
install -m 0644 custom-runtime.service ${D}${systemd_unitdir}/system/
}
SYSTEMD_SERVICE_${PN} = "custom-runtime.service"
SYSTEMD_AUTO_ENABLE = "enable"
FILES_${PN} = "${bindir}/* ${sysconfdir}/custom-runtime/* ${systemd_unitdir}/system/*"Systemd Service:
# custom-runtime.service
[Unit]
Description=Custom Container Runtime
After=network.target
Requires=containerd.service
After=containerd.service
[Service]
Type=simple
ExecStart=/usr/bin/custom-daemon
Restart=always
RestartSec=5
MemoryLimit=64M
CPUQuota=20%
[Install]
WantedBy=multi-user.targetUpdate and Deployment
Atomic Container Updates:
#!/bin/bash
# update-container.sh - Atomic container updates
CONTAINER_NAME=$1
NEW_IMAGE=$2
echo "Starting atomic update for $CONTAINER_NAME to $NEW_IMAGE"
# Stop current container
custom-cli stop $CONTAINER_NAME
# Backup current state
custom-cli export $CONTAINER_NAME > /tmp/${CONTAINER_NAME}_backup.tar
# Deploy new image
if custom-cli run $NEW_IMAGE $CONTAINER_NAME; then
echo "Update successful"
rm /tmp/${CONTAINER_NAME}_backup.tar
else
echo "Update failed, rolling back"
custom-cli stop $CONTAINER_NAME
custom-cli import /tmp/${CONTAINER_NAME}_backup.tar $CONTAINER_NAME
custom-cli start $CONTAINER_NAME
fiLessons Learned and Recommendations
What Worked Well
- Modular Architecture: Using Moby components individually allowed precise optimization
- Custom CLI: Eliminating Docker CLI complexity reduced memory and startup overhead significantly
- Host Networking: Acceptable security trade-off for massive performance gain in embedded context
- Aggressive Resource Limits: Forcing applications to be efficient improved overall system stability
What Didn’t Work
- Standard Docker Images: Too much overhead; custom image format was necessary
- Full Container Isolation: Performance cost too high for embedded; selective isolation worked better
- Complex Orchestration: Kubernetes/Docker Swarm completely inappropriate for single-device embedded systems
Production Recommendations
When to Use Custom Runtime:
- Resource-constrained embedded systems (< 1GB RAM)
- Single-purpose devices with known workloads
- Industrial applications requiring deterministic performance
- Battery-powered devices where efficiency matters
When to Stick with Docker:
- Development environments
- Multi-tenant systems requiring full isolation
- Applications requiring Docker ecosystem compatibility
- Systems with abundant resources (> 2GB RAM)
Conclusion
Building a custom container runtime using Moby Project components proved that Docker’s “one size fits all” approach isn’t optimal for embedded systems. By carefully selecting only the components we needed and optimizing for our specific use case, we achieved:
- 72% reduction in memory footprint
- 83% improvement in startup performance
- 58% reduction in storage overhead
- 95% reduction in CPU overhead
The key insight: container technology is incredibly powerful for embedded systems, but you need to build it right for your specific constraints. Standard Docker is optimized for cloud and development environments, not resource-constrained embedded devices.
Sometimes the best solution isn’t using existing tools as-is, but understanding the underlying technology well enough to build exactly what you need. The Moby Project makes this possible by providing the building blocks without forcing you to accept Docker’s architectural decisions.
For embedded systems engineers considering containerization: don’t let Docker’s resource requirements scare you away from containers entirely. Build what you actually need, and you might be surprised how efficient container technology can be.
Technical References
- Moby Project - Official Moby Project documentation
- containerd - Industry-standard container runtime
- runc - CLI tool for spawning containers
- OCI Runtime Specification - Open Container Initiative runtime spec
- Yocto meta-virtualization - Container support for Yocto
- LinuxKit - Toolkit for building secure, portable Linux subsystems
- BuildKit - Next-generation Docker image builder