misc: zot registry, k8s OIDC, server configs, sandbox experiments, and notes

- docker-30/zot: add Zot OCI registry with on-demand sync to docker.io,
  registry.k8s.io, ghcr.io, quay.io
- kubernetes-kvm-terraform: wire Kanidm OIDC via structured
  AuthenticationConfiguration; add reference apiserver manifest and
  join-node-02 helper
- servers: reorganize shadow/ under servers/, add saint vhost config and
  utility-101 VM definition, add shadow hrajfrisbee.cz vhost and
  storage-23 notes
- experiments: add notes and configs for e2b dev VM, kata + firecracker
  on kube, microsandbox, orb-stack k3s (terraform + cloud-init), rke2
- vms/docker: document tailscale + node-exporter setup
- blog: stub post on Gateway API
- chore: gitignore tmp/, smtp_password, and the two local-only
  credential caches; add per-project .claude/settings.json

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-01 18:12:38 +02:00
parent 5ca27a832b
commit 80d0cc1168
34 changed files with 2814 additions and 1 deletions

View File

@@ -0,0 +1,391 @@
# E2B Dev VM Setup on KVM Homelab
## Context
The user wants to run the E2B infrastructure dev stack on a KVM virtual machine in their homelab. E2B uses Firecracker microVMs (which need `/dev/kvm`), so the guest VM needs **nested virtualization** (KVM-in-KVM). This guide covers VM creation, OS configuration, toolchain installation, and running the full dev stack.
> **Note from upstream**: [DEV-LOCAL.md](DEV-LOCAL.md) says "Linux is required. This is a work in progress. Not everything will function as expected."
---
## Phase 1: Create the KVM VM on the Hypervisor Host
### 1.1 Enable nested virtualization on the host
**Intel:**
```bash
cat /sys/module/kvm_intel/parameters/nested # check
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel nested=1
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
```
**AMD:**
```bash
cat /sys/module/kvm_amd/parameters/nested
sudo modprobe -r kvm_amd && sudo modprobe kvm_amd nested=1
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
# get details about loaded kernel module
systool -v -m kvm_amd
```
### 1.2 Create the VM
```bash
virt-install \
--name e2b-dev \
--ram 16384 \
--vcpus 8 \
--cpu host-passthrough \
--os-variant ubuntu24.04 \
--disk path=/var/lib/libvirt/images/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
--network bridge=virbr0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--cdrom /path/to/ubuntu-24.04-live-server-amd64.iso \
--extra-args 'console=ttyS0,115200n8'
```
**Why these specs:**
| Resource | Value | Rationale |
|----------|-------|-----------|
| RAM | 16 GB (24-32 better) | 4 GB for huge pages (2048 x 2MB), ~4 GB for 10 Docker containers, rest for Go services + Firecracker VMs |
| vCPUs | 8 | Firecracker VMs consume vCPUs; Go services are concurrent |
| Disk | 100 GB | Docker images, FC binaries, kernels, rootfs, Go cache, build artifacts |
| CPU | `host-passthrough` | **Mandatory** -- exposes VMX/SVM to guest so `/dev/kvm` works inside the VM |
### 1.3 Alternative: libvirt XML
If you manage VMs declaratively, the critical part is:
```xml
<cpu mode='host-passthrough' check='none' migratable='off'/>
```
### 1.4 Verify nested KVM works (after OS install)
```bash
ls -la /dev/kvm # must exist
lsmod | grep kvm # kvm + kvm_intel/kvm_amd
grep -cE '(vmx|svm)' /proc/cpuinfo # must be > 0
```
If `/dev/kvm` is missing, go back to 1.1.
---
## Phase 2: OS Configuration (inside the guest VM)
**Recommended OS:** Ubuntu 24.04 LTS Server (matches CI, well-tested with Firecracker)
### 2.1 Base packages
```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
build-essential git curl wget unzip jq make gcc pkg-config \
iptables iproute2 net-tools ca-certificates gnupg \
lsb-release software-properties-common gettext-base
```
### 2.2 Kernel modules
```bash
# Load now
sudo modprobe nbd nbds_max=64
sudo modprobe kvm
sudo modprobe kvm_intel # or kvm_amd
sudo modprobe tun
sudo modprobe veth
sudo modprobe nf_tables
sudo modprobe nft_nat
# Persist across reboots
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
nbd
kvm
kvm_intel
tun
veth
nf_tables
nft_nat
EOF
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
```
### 2.3 Sysctl
```bash
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
vm.nr_hugepages=2048
vm.max_map_count=1048576
vm.swappiness=10
vm.vfs_cache_pressure=50
net.ipv4.ip_forward=1
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF
sudo sysctl --system
```
### 2.4 Udev rules (suppress NBD inotify noise)
```bash
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
KERNEL=="nbd*", OPTIONS+="nowatch"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger
```
### 2.5 File descriptor limits
```bash
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
EOF
```
---
## Phase 3: Install Toolchain
### 3.1 Docker
```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in
docker --version && docker compose version
```
### 3.2 mise (manages all tools from `.tool-versions`)
```bash
curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc
```
### 3.3 Install project tools
The repo's [.tool-versions](.tool-versions) pins:
| Tool | Version |
| --- | --- |
| golang | 1.25.4 |
| buf | 1.28.1 |
| bun | 1.3.2 |
| protoc | 29.3 |
| protoc-gen-go | 1.28.1 |
| protoc-gen-go-grpc | 1.6.1 |
| protoc-gen-connect-go | 1.18.1 |
| golangci-lint | 2.8.0 |
| terraform | 1.5.7 |
| packer | 1.13.1 |
| python | 3.13.11 |
| gcloud | 534.0.0 |
```bash
cd ~/e2b-infra # after cloning
mise install # installs everything from .tool-versions
```
> **Note:** `gcloud` is only needed for `gsutil` to download prebuilt artifacts. If you don't have GCP credentials, you can download them via HTTPS instead (see Phase 5).
---
## Phase 4: Clone Repository
```bash
git clone https://github.com/e2b-dev/infra.git ~/e2b-infra
cd ~/e2b-infra
go work sync
```
---
## Phase 5: Download Prebuilt Artifacts
Firecracker binaries and Linux kernels must be downloaded from the public GCS bucket.
**With gsutil (if gcloud is configured):**
```bash
make download-public-kernels
make download-public-firecrackers
```
**Without GCP credentials (HTTPS alternative):**
The bucket `e2b-prod-public-builds` is publicly accessible. You'll need to browse/list it to find available versions, then download manually:
```bash
# Install gsutil standalone (no GCP project needed for public buckets)
# OR use curl/wget against:
# https://storage.googleapis.com/e2b-prod-public-builds/
# to discover and download kernel and firecracker builds
```
The simplest path is to install just the `gcloud` CLI (via mise) and run the make targets -- no GCP project or auth is needed for public bucket reads.
---
## Phase 6: Prepare Local Environment
### 6.1 Start infrastructure containers
```bash
make local-infra
# Starts: PostgreSQL 17.4, Redis 7.4.2, ClickHouse 25.4.5.24,
# Grafana 12.0.0, Loki 3.4.1, Tempo 2.8.2, Mimir 2.17.1,
# OTEL Collector 0.146.0, Vector, Memcached 1.6.38
```
Wait for all containers to be healthy:
```bash
docker compose -f packages/local-dev/docker-compose.yaml ps
```
### 6.2 Initialize databases
```bash
make -C packages/db migrate-local # PostgreSQL
make -C packages/clickhouse migrate-local # ClickHouse
```
### 6.3 Build envd (in-VM daemon)
```bash
make -C packages/envd build
```
### 6.4 Seed database with dev credentials
```bash
make -C packages/local-dev seed-database
```
Creates test user, team, API key, and access token for local development.
---
## Phase 7: Run the Dev Stack
Each service runs in the foreground. Use **tmux**, **screen**, or separate SSH sessions.
| Terminal | Command | Listens on |
| --- | --- | --- |
| 1 | `make local-infra` | (Docker containers) |
| 2 | `make -C packages/api run-local` | `:3000` |
| 3 | `make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local` | `:5008` |
| 4 | `make -C packages/client-proxy run-local` | `:3002` |
> The orchestrator **requires sudo** -- Firecracker needs root for `/dev/kvm`, network namespaces, veth pairs, nftables rules, and NBD devices.
---
## Phase 8: Verify
### 8.1 Health checks
```bash
curl -s http://localhost:3000/health # API
curl -s -o /dev/null -w "%{http_code}" http://localhost:53000 # Grafana (expect 302)
curl -s 'http://localhost:8123/?query=SELECT%201' # ClickHouse
redis-cli -h localhost -p 6379 ping # Redis
```
### 8.2 Build the base template
```bash
make -C packages/shared/scripts local-build-base-template
```
### 8.3 Test with E2B client SDK
```bash
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
export E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
export E2B_API_URL=http://localhost:3000
export E2B_SANDBOX_URL=http://localhost:3002
# Use E2B SDK/CLI to create a sandbox
```
---
## Phase 9: Access from Host (Optional)
### SSH port forwarding
```bash
ssh -N \
-L 3000:localhost:3000 \
-L 3002:localhost:3002 \
-L 53000:localhost:53000 \
-L 5432:localhost:5432 \
-L 8123:localhost:8123 \
user@<vm-ip>
```
### Or use bridged networking
If the VM has a routable IP on your LAN, services are directly accessible (Docker binds to `0.0.0.0`, Go services listen on all interfaces).
---
## Service Endpoints Reference
| Service | URL |
| --- | --- |
| E2B API | `http://localhost:3000` |
| E2B Client Proxy | `http://localhost:3002` |
| E2B Orchestrator | `http://localhost:5008` |
| Grafana | `http://localhost:53000` |
| PostgreSQL | `postgres://postgres:postgres@localhost:5432` |
| ClickHouse (HTTP) | `http://localhost:8123` |
| ClickHouse (native) | `localhost:9000` |
| Redis | `localhost:6379` |
| OTEL Collector (gRPC) | `localhost:4317` |
| OTEL Collector (HTTP) | `localhost:4318` |
| Loki | `http://localhost:3100` |
| Vector | `localhost:30006` |
---
## Troubleshooting
| Problem | Fix |
| --- | --- |
| `/dev/kvm` missing in guest | Enable nested virt on host (Phase 1.1), use `host-passthrough` CPU |
| `modprobe nbd` fails | Check kernel has NBD support: `modinfo nbd` |
| Orchestrator permission errors | Must run with `sudo` |
| Huge pages < 2048 | Not enough contiguous memory; increase VM RAM or set earlier in boot |
| Docker containers won't start | Check `systemctl status docker`, port conflicts with `ss -tlnp` |
| `gsutil` not found | Install via `mise install gcloud` or download artifacts via HTTPS |
---
## After VM Reboot Checklist
```bash
# 1. Verify kernel modules and sysctl (should be persistent)
lsmod | grep nbd
cat /proc/sys/vm/nr_hugepages # expect 2048
# 2. Start infra
cd ~/e2b-infra && make local-infra
# 3. Start services (separate terminals)
make -C packages/api run-local
sudo make -C packages/orchestrator run-local
make -C packages/client-proxy run-local
```

View File

@@ -0,0 +1,95 @@
## dev-vm on beelink
```bash
virt-install \
--name e2b-dev \
--ram 16384 \
--vcpus 8 \
--cpu host-passthrough \
--os-variant ubuntu24.04 \
--disk path=/srv/vms/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
--network bridge=br0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--location /srv/vms/isos/ubuntu-24.04.3-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
--extra-args 'console=ttyS0,115200n8'
# base packages
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
build-essential git curl wget unzip jq make gcc pkg-config \
iptables iproute2 net-tools ca-certificates gnupg \
lsb-release software-properties-common gettext-base
# kernel modules
# Load now
sudo modprobe nbd nbds_max=64
sudo modprobe kvm
sudo modprobe kvm_amd # or kvm_amd
sudo modprobe tun
sudo modprobe veth
sudo modprobe nf_tables
sudo modprobe nft_nat
# Persist across reboots
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
nbd
kvm
kvm_amd
tun
veth
nf_tables
nft_nat
EOF
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
# sysctl
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
vm.nr_hugepages=2048
vm.max_map_count=1048576
vm.swappiness=10
vm.vfs_cache_pressure=50
net.ipv4.ip_forward=1
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF
sudo sysctl --system
# udev rules
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
KERNEL=="nbd*", OPTIONS+="nowatch"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger
# file descriptor limits
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
EOF
```
## install toolchain
```bash
# docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in
docker --version && docker compose version
# mise
curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc
```
```

View File

@@ -0,0 +1,280 @@
# E2B Sandbox Usage Guide
A practical guide for creating, observing, and using E2B sandboxes via the API.
## Prerequisites
### Local Dev Setup
Seed the database to create a test team and API key:
```bash
make -C packages/local-dev seed-database
```
### Environment Variables
```bash
E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
E2B_API_URL=http://localhost:3000
E2B_SANDBOX_URL=http://localhost:3002
```
All examples below use `$E2B_API_URL` and `$E2B_API_KEY` — export them in your shell for convenience:
```bash
export E2B_API_URL=http://localhost:3000
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
```
---
## 1. Create a Sandbox
```bash
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300
}'
```
**Response:**
```json
{
"sandboxID": "i1234567890abcdef",
"templateID": "base",
"envdVersion": "0.5.8",
"domain": "i1234567890abcdef.your-domain"
}
```
Save the `sandboxID` and `domain` from the response for subsequent commands.
### Optional Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `secure` | bool | Returns an `envdAccessToken` for authenticated envd access |
| `envVars` | object | Inject environment variables into the VM |
| `metadata` | object | Tag the sandbox for filtering (e.g. `{"purpose": "test"}`) |
| `autoPause` | bool | Pause instead of kill on timeout |
| `allow_internet_access` | bool | Allow internet egress from the VM |
| `network.allowPublicTraffic` | bool | Allow inbound public traffic |
| `network.allowOut` | array | Allowed egress destinations (IPs, CIDRs, domains) |
| `network.denyOut` | array | Denied egress destinations (IPs/CIDRs only) |
| `volumeMounts` | array | Mount persistent volumes (`{"name": "vol", "path": "/data"}`) |
**Example with options:**
```bash
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300,
"secure": true,
"envVars": {"MY_VAR": "hello"},
"metadata": {"purpose": "demo"}
}'
```
---
## 2. Observe Sandbox State
### Get details of a specific sandbox
```bash
curl $E2B_API_URL/sandboxes/{sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
```
Returns state (`running`/`paused`), CPU/memory/disk config, network settings, and TTL (`endAt`).
### List all sandboxes
```bash
# Running only
curl "$E2B_API_URL/v2/sandboxes?state=running" \
-H "X-API-Key: $E2B_API_KEY"
# Running and paused
curl "$E2B_API_URL/v2/sandboxes?state=running&state=paused" \
-H "X-API-Key: $E2B_API_KEY"
# Filter by metadata
curl "$E2B_API_URL/v2/sandboxes?metadata=purpose%3Ddemo" \
-H "X-API-Key: $E2B_API_KEY"
```
### Get resource metrics (CPU, memory, disk)
```bash
curl "$E2B_API_URL/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
-H "X-API-Key: $E2B_API_KEY"
```
**Response:**
```json
[
{
"timestampUnix": 1234567890,
"cpuCount": 2,
"cpuUsedPct": 25.5,
"memUsed": 268435456,
"memTotal": 536870912,
"diskUsed": 1073741824,
"diskTotal": 5368709120
}
]
```
### Get sandbox logs
```bash
curl "$E2B_API_URL/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
-H "X-API-Key: $E2B_API_KEY"
```
Optional query parameters: `cursor` (ms timestamp), `level` (min log level), `search` (substring match).
---
## 3. Use the Sandbox
The in-VM daemon (**envd**) runs on port 49983 inside each sandbox, exposed via the sandbox's `domain`.
### Upload a file
```bash
curl -X POST https://{domain}/files \
-H "Content-Type: multipart/form-data" \
-F "file=@script.py" \
-F "path=/home/user/script.py"
```
### Using the E2B Python SDK
Point the SDK at your local API:
```python
from e2b import Sandbox
sbx = Sandbox("base", api_url="http://localhost:3000")
# Run a command
result = sbx.commands.run("echo 'Hello from Firecracker VM!'")
print(result.stdout)
# Write and execute a file
sbx.files.write("/home/user/hello.py", "print('Hello world')")
result = sbx.commands.run("python3 /home/user/hello.py")
print(result.stdout)
# List files
files = sbx.files.list("/home/user")
for f in files:
print(f.name)
sbx.kill()
```
### Envd Connect RPC API
The envd daemon exposes Connect RPC services for programmatic access:
**Process Service:**
- `Start(ProcessConfig)` — start a new process
- `List()` — list running processes
- `Connect(ProcessSelector)` — connect to process stdio
- `Signal(ProcessSelector, Signal)` — send signal to process
**Filesystem Service:**
- `ListDir(Path)` — list directory contents
- `Stat(Path)` — get file metadata
- `WatchDir(Path)` — watch for changes
- `Move(Source, Dest)` — move/rename file
- `RemoveDir(Path)` — remove directory
---
## 4. Lifecycle Management
### Extend timeout
```bash
# Add 60 seconds to the TTL
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/refreshes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"duration": 60}'
# Or set an absolute timeout (seconds from now)
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/timeout \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"timeout": 120}'
```
### Pause (snapshot to disk)
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/pause \
-H "X-API-Key: $E2B_API_KEY"
```
### Resume / reconnect
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/connect \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"timeout": 300}'
```
Returns `200` if already running, `201` if resumed from paused state.
### Kill
```bash
curl -X DELETE $E2B_API_URL/sandboxes/{sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
```
### Create a snapshot (template from running sandbox)
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/snapshots \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "my-snapshot"}'
```
---
## 5. Network Configuration
Update network rules on a running sandbox:
```bash
curl -X PUT $E2B_API_URL/sandboxes/{sandboxID}/network \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allowOut": ["8.8.8.8", "example.com"],
"denyOut": ["0.0.0.0/0"]
}'
```
**Rules:**
- If `allowOut` contains domain names, `denyOut` must include `0.0.0.0/0`
- Domain names are not supported in `denyOut` (IPs/CIDRs only)
- `allowOut` entries take precedence over `denyOut`
- Omitting both fields clears all rules

View File

@@ -0,0 +1,46 @@
## VMs
```bash
E2B_API_URL=http://192.168.0.61:3000
# create vm
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300
}'
# response
{"alias":"base","clientID":"6532622b","domain":null,"envdVersion":"0.5.8","sandboxID":"in3h60s6h0ie3kigrcls4","templateID":"pz1l1owhmy0w84e12eqv","trafficAccessToken":null}
sandboxID=in3h60s6h0ie3kigrcls4
# create sandboxVM and capture id
sandboxID=$(curl -s -X POST http://localhost:3000/sandboxes -H "X-API-Key: $E2B_API_KEY" -H "Content-Type: application/json" -d '{
"templateID": "base",
"timeout": 300
}' | jq -r .sandboxID
)
# get state
curl http://localhost:3000/sandboxes/${sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
# list running sandboxes
curl "http://localhost:3000/v2/sandboxes?state=running" \
-H "X-API-Key: $E2B_API_KEY"
# sandbox resource metrics
curl "http://localhost:3000/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
-H "X-API-Key: $E2B_API_KEY"
# sandbox logs
curl "http://localhost:3000/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
-H "X-API-Key: $E2B_API_KEY"
```