home-kubernetes/experiments/e2b/dev-vm-deployment-plan.md

E2B Dev VM Setup on KVM Homelab

Context

The user wants to run the E2B infrastructure dev stack on a KVM virtual machine in their homelab. E2B uses Firecracker microVMs (which need /dev/kvm), so the guest VM needs nested virtualization (KVM-in-KVM). This guide covers VM creation, OS configuration, toolchain installation, and running the full dev stack.

Note from upstream: DEV-LOCAL.md says "Linux is required. This is a work in progress. Not everything will function as expected."

---

Phase 1: Create the KVM VM on the Hypervisor Host

1.1 Enable nested virtualization on the host

Intel:

cat /sys/module/kvm_intel/parameters/nested   # check
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel nested=1
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf

AMD:

cat /sys/module/kvm_amd/parameters/nested
sudo modprobe -r kvm_amd && sudo modprobe kvm_amd nested=1
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf

# get details about loaded kernel module
systool -v -m kvm_amd

1.2 Create the VM

virt-install \
  --name e2b-dev \
  --ram 16384 \
  --vcpus 8 \
  --cpu host-passthrough \
  --os-variant ubuntu24.04 \
  --disk path=/var/lib/libvirt/images/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
  --network bridge=virbr0,model=virtio \
  --graphics none \
  --console pty,target_type=serial \
  --cdrom /path/to/ubuntu-24.04-live-server-amd64.iso \
  --extra-args 'console=ttyS0,115200n8'

Why these specs:

Resource	Value	Rationale
RAM	16 GB (24-32 better)	4 GB for huge pages (2048 x 2MB), ~4 GB for 10 Docker containers, rest for Go services + Firecracker VMs
vCPUs	8	Firecracker VMs consume vCPUs; Go services are concurrent
Disk	100 GB	Docker images, FC binaries, kernels, rootfs, Go cache, build artifacts
CPU	host-passthrough	Mandatory -- exposes VMX/SVM to guest so /dev/kvm works inside the VM

1.3 Alternative: libvirt XML

If you manage VMs declaratively, the critical part is:

<cpu mode='host-passthrough' check='none' migratable='off'/>

1.4 Verify nested KVM works (after OS install)

ls -la /dev/kvm                                    # must exist
lsmod | grep kvm                                   # kvm + kvm_intel/kvm_amd
grep -cE '(vmx|svm)' /proc/cpuinfo                # must be > 0

If /dev/kvm is missing, go back to 1.1.

---

Phase 2: OS Configuration (inside the guest VM)

Recommended OS: Ubuntu 24.04 LTS Server (matches CI, well-tested with Firecracker)

2.1 Base packages

sudo apt update && sudo apt upgrade -y
sudo apt install -y \
  build-essential git curl wget unzip jq make gcc pkg-config \
  iptables iproute2 net-tools ca-certificates gnupg \
  lsb-release software-properties-common gettext-base

2.2 Kernel modules

# Load now
sudo modprobe nbd nbds_max=64
sudo modprobe kvm
sudo modprobe kvm_intel   # or kvm_amd
sudo modprobe tun
sudo modprobe veth
sudo modprobe nf_tables
sudo modprobe nft_nat

# Persist across reboots
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
nbd
kvm
kvm_intel
tun
veth
nf_tables
nft_nat
EOF

echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf

2.3 Sysctl

cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
vm.nr_hugepages=2048
vm.max_map_count=1048576
vm.swappiness=10
vm.vfs_cache_pressure=50
net.ipv4.ip_forward=1
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF

sudo sysctl --system

2.4 Udev rules (suppress NBD inotify noise)

cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
KERNEL=="nbd*", OPTIONS+="nowatch"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger

2.5 File descriptor limits

cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
*    soft    nofile    1048576
*    hard    nofile    1048576
root soft    nofile    1048576
root hard    nofile    1048576
EOF

---

Phase 3: Install Toolchain

3.1 Docker

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in
docker --version && docker compose version

3.2 mise (manages all tools from .tool-versions)

curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc

3.3 Install project tools

The repo's .tool-versions pins:

Tool	Version
golang	1.25.4
buf	1.28.1
bun	1.3.2
protoc	29.3
protoc-gen-go	1.28.1
protoc-gen-go-grpc	1.6.1
protoc-gen-connect-go	1.18.1
golangci-lint	2.8.0
terraform	1.5.7
packer	1.13.1
python	3.13.11
gcloud	534.0.0

cd ~/e2b-infra   # after cloning
mise install     # installs everything from .tool-versions

Note: gcloud is only needed for gsutil to download prebuilt artifacts. If you don't have GCP credentials, you can download them via HTTPS instead (see Phase 5).

---

Phase 4: Clone Repository

git clone https://github.com/e2b-dev/infra.git ~/e2b-infra
cd ~/e2b-infra
go work sync

---

Phase 5: Download Prebuilt Artifacts

Firecracker binaries and Linux kernels must be downloaded from the public GCS bucket.

With gsutil (if gcloud is configured):

make download-public-kernels
make download-public-firecrackers

Without GCP credentials (HTTPS alternative):

The bucket e2b-prod-public-builds is publicly accessible. You'll need to browse/list it to find available versions, then download manually:

# Install gsutil standalone (no GCP project needed for public buckets)
# OR use curl/wget against:
#   https://storage.googleapis.com/e2b-prod-public-builds/
# to discover and download kernel and firecracker builds

The simplest path is to install just the gcloud CLI (via mise) and run the make targets -- no GCP project or auth is needed for public bucket reads.

---

Phase 6: Prepare Local Environment

6.1 Start infrastructure containers

make local-infra
# Starts: PostgreSQL 17.4, Redis 7.4.2, ClickHouse 25.4.5.24,
#          Grafana 12.0.0, Loki 3.4.1, Tempo 2.8.2, Mimir 2.17.1,
#          OTEL Collector 0.146.0, Vector, Memcached 1.6.38

Wait for all containers to be healthy:

docker compose -f packages/local-dev/docker-compose.yaml ps

6.2 Initialize databases

make -C packages/db migrate-local              # PostgreSQL
make -C packages/clickhouse migrate-local      # ClickHouse

6.3 Build envd (in-VM daemon)

make -C packages/envd build

6.4 Seed database with dev credentials

make -C packages/local-dev seed-database

Creates test user, team, API key, and access token for local development.

---

Phase 7: Run the Dev Stack

Each service runs in the foreground. Use tmux, screen, or separate SSH sessions.

Terminal	Command	Listens on
1	make local-infra	(Docker containers)
2	make -C packages/api run-local	:3000
3	make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local	:5008
4	make -C packages/client-proxy run-local	:3002

The orchestrator requires sudo -- Firecracker needs root for /dev/kvm, network namespaces, veth pairs, nftables rules, and NBD devices.

---

Phase 8: Verify

8.1 Health checks

curl -s http://localhost:3000/health              # API
curl -s -o /dev/null -w "%{http_code}" http://localhost:53000   # Grafana (expect 302)
curl -s 'http://localhost:8123/?query=SELECT%201'  # ClickHouse
redis-cli -h localhost -p 6379 ping                # Redis

8.2 Build the base template

make -C packages/shared/scripts local-build-base-template

8.3 Test with E2B client SDK

export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
export E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
export E2B_API_URL=http://localhost:3000
export E2B_SANDBOX_URL=http://localhost:3002
# Use E2B SDK/CLI to create a sandbox

---

Phase 9: Access from Host (Optional)

SSH port forwarding

ssh -N \
  -L 3000:localhost:3000 \
  -L 3002:localhost:3002 \
  -L 53000:localhost:53000 \
  -L 5432:localhost:5432 \
  -L 8123:localhost:8123 \
  user@<vm-ip>

Or use bridged networking

If the VM has a routable IP on your LAN, services are directly accessible (Docker binds to 0.0.0.0, Go services listen on all interfaces).

---

Service Endpoints Reference

Service	URL
E2B API	http://localhost:3000
E2B Client Proxy	http://localhost:3002
E2B Orchestrator	http://localhost:5008
Grafana	http://localhost:53000
PostgreSQL	postgres://postgres:postgres@localhost:5432
ClickHouse (HTTP)	http://localhost:8123
ClickHouse (native)	localhost:9000
Redis	localhost:6379
OTEL Collector (gRPC)	localhost:4317
OTEL Collector (HTTP)	localhost:4318
Loki	http://localhost:3100
Vector	localhost:30006

---

Troubleshooting

Problem	Fix
/dev/kvm missing in guest	Enable nested virt on host (Phase 1.1), use host-passthrough CPU
modprobe nbd fails	Check kernel has NBD support: modinfo nbd
Orchestrator permission errors	Must run with sudo
Huge pages < 2048	Not enough contiguous memory; increase VM RAM or set earlier in boot
Docker containers won't start	Check systemctl status docker, port conflicts with ss -tlnp
gsutil not found	Install via mise install gcloud or download artifacts via HTTPS

---

After VM Reboot Checklist

# 1. Verify kernel modules and sysctl (should be persistent)
lsmod | grep nbd
cat /proc/sys/vm/nr_hugepages   # expect 2048

# 2. Start infra
cd ~/e2b-infra && make local-infra

# 3. Start services (separate terminals)
make -C packages/api run-local
sudo make -C packages/orchestrator run-local
make -C packages/client-proxy run-local