misc: zot registry, k8s OIDC, server configs, sandbox experiments, and notes
- docker-30/zot: add Zot OCI registry with on-demand sync to docker.io, registry.k8s.io, ghcr.io, quay.io - kubernetes-kvm-terraform: wire Kanidm OIDC via structured AuthenticationConfiguration; add reference apiserver manifest and join-node-02 helper - servers: reorganize shadow/ under servers/, add saint vhost config and utility-101 VM definition, add shadow hrajfrisbee.cz vhost and storage-23 notes - experiments: add notes and configs for e2b dev VM, kata + firecracker on kube, microsandbox, orb-stack k3s (terraform + cloud-init), rke2 - vms/docker: document tailscale + node-exporter setup - blog: stub post on Gateway API - chore: gitignore tmp/, smtp_password, and the two local-only credential caches; add per-project .claude/settings.json Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
391
experiments/e2b/dev-vm-deployment-plan.md
Normal file
391
experiments/e2b/dev-vm-deployment-plan.md
Normal file
@@ -0,0 +1,391 @@
|
||||
# E2B Dev VM Setup on KVM Homelab
|
||||
|
||||
## Context
|
||||
|
||||
The user wants to run the E2B infrastructure dev stack on a KVM virtual machine in their homelab. E2B uses Firecracker microVMs (which need `/dev/kvm`), so the guest VM needs **nested virtualization** (KVM-in-KVM). This guide covers VM creation, OS configuration, toolchain installation, and running the full dev stack.
|
||||
|
||||
> **Note from upstream**: [DEV-LOCAL.md](DEV-LOCAL.md) says "Linux is required. This is a work in progress. Not everything will function as expected."
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Create the KVM VM on the Hypervisor Host
|
||||
|
||||
### 1.1 Enable nested virtualization on the host
|
||||
|
||||
**Intel:**
|
||||
|
||||
```bash
|
||||
cat /sys/module/kvm_intel/parameters/nested # check
|
||||
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel nested=1
|
||||
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
|
||||
```
|
||||
|
||||
**AMD:**
|
||||
|
||||
```bash
|
||||
cat /sys/module/kvm_amd/parameters/nested
|
||||
sudo modprobe -r kvm_amd && sudo modprobe kvm_amd nested=1
|
||||
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
|
||||
|
||||
# get details about loaded kernel module
|
||||
systool -v -m kvm_amd
|
||||
```
|
||||
|
||||
### 1.2 Create the VM
|
||||
|
||||
```bash
|
||||
virt-install \
|
||||
--name e2b-dev \
|
||||
--ram 16384 \
|
||||
--vcpus 8 \
|
||||
--cpu host-passthrough \
|
||||
--os-variant ubuntu24.04 \
|
||||
--disk path=/var/lib/libvirt/images/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
|
||||
--network bridge=virbr0,model=virtio \
|
||||
--graphics none \
|
||||
--console pty,target_type=serial \
|
||||
--cdrom /path/to/ubuntu-24.04-live-server-amd64.iso \
|
||||
--extra-args 'console=ttyS0,115200n8'
|
||||
```
|
||||
|
||||
**Why these specs:**
|
||||
| Resource | Value | Rationale |
|
||||
|----------|-------|-----------|
|
||||
| RAM | 16 GB (24-32 better) | 4 GB for huge pages (2048 x 2MB), ~4 GB for 10 Docker containers, rest for Go services + Firecracker VMs |
|
||||
| vCPUs | 8 | Firecracker VMs consume vCPUs; Go services are concurrent |
|
||||
| Disk | 100 GB | Docker images, FC binaries, kernels, rootfs, Go cache, build artifacts |
|
||||
| CPU | `host-passthrough` | **Mandatory** -- exposes VMX/SVM to guest so `/dev/kvm` works inside the VM |
|
||||
|
||||
### 1.3 Alternative: libvirt XML
|
||||
|
||||
If you manage VMs declaratively, the critical part is:
|
||||
|
||||
```xml
|
||||
<cpu mode='host-passthrough' check='none' migratable='off'/>
|
||||
```
|
||||
|
||||
### 1.4 Verify nested KVM works (after OS install)
|
||||
|
||||
```bash
|
||||
ls -la /dev/kvm # must exist
|
||||
lsmod | grep kvm # kvm + kvm_intel/kvm_amd
|
||||
grep -cE '(vmx|svm)' /proc/cpuinfo # must be > 0
|
||||
```
|
||||
|
||||
If `/dev/kvm` is missing, go back to 1.1.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: OS Configuration (inside the guest VM)
|
||||
|
||||
**Recommended OS:** Ubuntu 24.04 LTS Server (matches CI, well-tested with Firecracker)
|
||||
|
||||
### 2.1 Base packages
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
sudo apt install -y \
|
||||
build-essential git curl wget unzip jq make gcc pkg-config \
|
||||
iptables iproute2 net-tools ca-certificates gnupg \
|
||||
lsb-release software-properties-common gettext-base
|
||||
```
|
||||
|
||||
### 2.2 Kernel modules
|
||||
|
||||
```bash
|
||||
# Load now
|
||||
sudo modprobe nbd nbds_max=64
|
||||
sudo modprobe kvm
|
||||
sudo modprobe kvm_intel # or kvm_amd
|
||||
sudo modprobe tun
|
||||
sudo modprobe veth
|
||||
sudo modprobe nf_tables
|
||||
sudo modprobe nft_nat
|
||||
|
||||
# Persist across reboots
|
||||
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
|
||||
nbd
|
||||
kvm
|
||||
kvm_intel
|
||||
tun
|
||||
veth
|
||||
nf_tables
|
||||
nft_nat
|
||||
EOF
|
||||
|
||||
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
|
||||
```
|
||||
|
||||
### 2.3 Sysctl
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
|
||||
vm.nr_hugepages=2048
|
||||
vm.max_map_count=1048576
|
||||
vm.swappiness=10
|
||||
vm.vfs_cache_pressure=50
|
||||
net.ipv4.ip_forward=1
|
||||
net.core.somaxconn=65535
|
||||
net.core.netdev_max_backlog=65535
|
||||
net.ipv4.tcp_max_syn_backlog=65535
|
||||
EOF
|
||||
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
### 2.4 Udev rules (suppress NBD inotify noise)
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
|
||||
KERNEL=="nbd*", OPTIONS+="nowatch"
|
||||
EOF
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
### 2.5 File descriptor limits
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
|
||||
* soft nofile 1048576
|
||||
* hard nofile 1048576
|
||||
root soft nofile 1048576
|
||||
root hard nofile 1048576
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Install Toolchain
|
||||
|
||||
### 3.1 Docker
|
||||
|
||||
```bash
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in
|
||||
docker --version && docker compose version
|
||||
```
|
||||
|
||||
### 3.2 mise (manages all tools from `.tool-versions`)
|
||||
|
||||
```bash
|
||||
curl https://mise.run | sh
|
||||
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
### 3.3 Install project tools
|
||||
|
||||
The repo's [.tool-versions](.tool-versions) pins:
|
||||
|
||||
| Tool | Version |
|
||||
| --- | --- |
|
||||
| golang | 1.25.4 |
|
||||
| buf | 1.28.1 |
|
||||
| bun | 1.3.2 |
|
||||
| protoc | 29.3 |
|
||||
| protoc-gen-go | 1.28.1 |
|
||||
| protoc-gen-go-grpc | 1.6.1 |
|
||||
| protoc-gen-connect-go | 1.18.1 |
|
||||
| golangci-lint | 2.8.0 |
|
||||
| terraform | 1.5.7 |
|
||||
| packer | 1.13.1 |
|
||||
| python | 3.13.11 |
|
||||
| gcloud | 534.0.0 |
|
||||
|
||||
```bash
|
||||
cd ~/e2b-infra # after cloning
|
||||
mise install # installs everything from .tool-versions
|
||||
```
|
||||
|
||||
> **Note:** `gcloud` is only needed for `gsutil` to download prebuilt artifacts. If you don't have GCP credentials, you can download them via HTTPS instead (see Phase 5).
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Clone Repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/e2b-dev/infra.git ~/e2b-infra
|
||||
cd ~/e2b-infra
|
||||
go work sync
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Download Prebuilt Artifacts
|
||||
|
||||
Firecracker binaries and Linux kernels must be downloaded from the public GCS bucket.
|
||||
|
||||
**With gsutil (if gcloud is configured):**
|
||||
|
||||
```bash
|
||||
make download-public-kernels
|
||||
make download-public-firecrackers
|
||||
```
|
||||
|
||||
**Without GCP credentials (HTTPS alternative):**
|
||||
|
||||
The bucket `e2b-prod-public-builds` is publicly accessible. You'll need to browse/list it to find available versions, then download manually:
|
||||
|
||||
```bash
|
||||
# Install gsutil standalone (no GCP project needed for public buckets)
|
||||
# OR use curl/wget against:
|
||||
# https://storage.googleapis.com/e2b-prod-public-builds/
|
||||
# to discover and download kernel and firecracker builds
|
||||
```
|
||||
|
||||
The simplest path is to install just the `gcloud` CLI (via mise) and run the make targets -- no GCP project or auth is needed for public bucket reads.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Prepare Local Environment
|
||||
|
||||
### 6.1 Start infrastructure containers
|
||||
|
||||
```bash
|
||||
make local-infra
|
||||
# Starts: PostgreSQL 17.4, Redis 7.4.2, ClickHouse 25.4.5.24,
|
||||
# Grafana 12.0.0, Loki 3.4.1, Tempo 2.8.2, Mimir 2.17.1,
|
||||
# OTEL Collector 0.146.0, Vector, Memcached 1.6.38
|
||||
```
|
||||
|
||||
Wait for all containers to be healthy:
|
||||
|
||||
```bash
|
||||
docker compose -f packages/local-dev/docker-compose.yaml ps
|
||||
```
|
||||
|
||||
### 6.2 Initialize databases
|
||||
|
||||
```bash
|
||||
make -C packages/db migrate-local # PostgreSQL
|
||||
make -C packages/clickhouse migrate-local # ClickHouse
|
||||
```
|
||||
|
||||
### 6.3 Build envd (in-VM daemon)
|
||||
|
||||
```bash
|
||||
make -C packages/envd build
|
||||
```
|
||||
|
||||
### 6.4 Seed database with dev credentials
|
||||
|
||||
```bash
|
||||
make -C packages/local-dev seed-database
|
||||
```
|
||||
|
||||
Creates test user, team, API key, and access token for local development.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Run the Dev Stack
|
||||
|
||||
Each service runs in the foreground. Use **tmux**, **screen**, or separate SSH sessions.
|
||||
|
||||
| Terminal | Command | Listens on |
|
||||
| --- | --- | --- |
|
||||
| 1 | `make local-infra` | (Docker containers) |
|
||||
| 2 | `make -C packages/api run-local` | `:3000` |
|
||||
| 3 | `make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local` | `:5008` |
|
||||
| 4 | `make -C packages/client-proxy run-local` | `:3002` |
|
||||
|
||||
> The orchestrator **requires sudo** -- Firecracker needs root for `/dev/kvm`, network namespaces, veth pairs, nftables rules, and NBD devices.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Verify
|
||||
|
||||
### 8.1 Health checks
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:3000/health # API
|
||||
curl -s -o /dev/null -w "%{http_code}" http://localhost:53000 # Grafana (expect 302)
|
||||
curl -s 'http://localhost:8123/?query=SELECT%201' # ClickHouse
|
||||
redis-cli -h localhost -p 6379 ping # Redis
|
||||
```
|
||||
|
||||
### 8.2 Build the base template
|
||||
|
||||
```bash
|
||||
make -C packages/shared/scripts local-build-base-template
|
||||
```
|
||||
|
||||
### 8.3 Test with E2B client SDK
|
||||
|
||||
```bash
|
||||
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
|
||||
export E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
|
||||
export E2B_API_URL=http://localhost:3000
|
||||
export E2B_SANDBOX_URL=http://localhost:3002
|
||||
# Use E2B SDK/CLI to create a sandbox
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Access from Host (Optional)
|
||||
|
||||
### SSH port forwarding
|
||||
|
||||
```bash
|
||||
ssh -N \
|
||||
-L 3000:localhost:3000 \
|
||||
-L 3002:localhost:3002 \
|
||||
-L 53000:localhost:53000 \
|
||||
-L 5432:localhost:5432 \
|
||||
-L 8123:localhost:8123 \
|
||||
user@<vm-ip>
|
||||
```
|
||||
|
||||
### Or use bridged networking
|
||||
|
||||
If the VM has a routable IP on your LAN, services are directly accessible (Docker binds to `0.0.0.0`, Go services listen on all interfaces).
|
||||
|
||||
---
|
||||
|
||||
## Service Endpoints Reference
|
||||
|
||||
| Service | URL |
|
||||
| --- | --- |
|
||||
| E2B API | `http://localhost:3000` |
|
||||
| E2B Client Proxy | `http://localhost:3002` |
|
||||
| E2B Orchestrator | `http://localhost:5008` |
|
||||
| Grafana | `http://localhost:53000` |
|
||||
| PostgreSQL | `postgres://postgres:postgres@localhost:5432` |
|
||||
| ClickHouse (HTTP) | `http://localhost:8123` |
|
||||
| ClickHouse (native) | `localhost:9000` |
|
||||
| Redis | `localhost:6379` |
|
||||
| OTEL Collector (gRPC) | `localhost:4317` |
|
||||
| OTEL Collector (HTTP) | `localhost:4318` |
|
||||
| Loki | `http://localhost:3100` |
|
||||
| Vector | `localhost:30006` |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Fix |
|
||||
| --- | --- |
|
||||
| `/dev/kvm` missing in guest | Enable nested virt on host (Phase 1.1), use `host-passthrough` CPU |
|
||||
| `modprobe nbd` fails | Check kernel has NBD support: `modinfo nbd` |
|
||||
| Orchestrator permission errors | Must run with `sudo` |
|
||||
| Huge pages < 2048 | Not enough contiguous memory; increase VM RAM or set earlier in boot |
|
||||
| Docker containers won't start | Check `systemctl status docker`, port conflicts with `ss -tlnp` |
|
||||
| `gsutil` not found | Install via `mise install gcloud` or download artifacts via HTTPS |
|
||||
|
||||
---
|
||||
|
||||
## After VM Reboot Checklist
|
||||
|
||||
```bash
|
||||
# 1. Verify kernel modules and sysctl (should be persistent)
|
||||
lsmod | grep nbd
|
||||
cat /proc/sys/vm/nr_hugepages # expect 2048
|
||||
|
||||
# 2. Start infra
|
||||
cd ~/e2b-infra && make local-infra
|
||||
|
||||
# 3. Start services (separate terminals)
|
||||
make -C packages/api run-local
|
||||
sudo make -C packages/orchestrator run-local
|
||||
make -C packages/client-proxy run-local
|
||||
```
|
||||
95
experiments/e2b/installation.md
Normal file
95
experiments/e2b/installation.md
Normal file
@@ -0,0 +1,95 @@
|
||||
## dev-vm on beelink
|
||||
|
||||
```bash
|
||||
virt-install \
|
||||
--name e2b-dev \
|
||||
--ram 16384 \
|
||||
--vcpus 8 \
|
||||
--cpu host-passthrough \
|
||||
--os-variant ubuntu24.04 \
|
||||
--disk path=/srv/vms/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
|
||||
--network bridge=br0,model=virtio \
|
||||
--graphics none \
|
||||
--console pty,target_type=serial \
|
||||
--location /srv/vms/isos/ubuntu-24.04.3-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
|
||||
--extra-args 'console=ttyS0,115200n8'
|
||||
|
||||
|
||||
# base packages
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
sudo apt install -y \
|
||||
build-essential git curl wget unzip jq make gcc pkg-config \
|
||||
iptables iproute2 net-tools ca-certificates gnupg \
|
||||
lsb-release software-properties-common gettext-base
|
||||
|
||||
|
||||
# kernel modules
|
||||
# Load now
|
||||
sudo modprobe nbd nbds_max=64
|
||||
sudo modprobe kvm
|
||||
sudo modprobe kvm_amd # or kvm_amd
|
||||
sudo modprobe tun
|
||||
sudo modprobe veth
|
||||
sudo modprobe nf_tables
|
||||
sudo modprobe nft_nat
|
||||
|
||||
# Persist across reboots
|
||||
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
|
||||
nbd
|
||||
kvm
|
||||
kvm_amd
|
||||
tun
|
||||
veth
|
||||
nf_tables
|
||||
nft_nat
|
||||
EOF
|
||||
|
||||
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
|
||||
|
||||
# sysctl
|
||||
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
|
||||
vm.nr_hugepages=2048
|
||||
vm.max_map_count=1048576
|
||||
vm.swappiness=10
|
||||
vm.vfs_cache_pressure=50
|
||||
net.ipv4.ip_forward=1
|
||||
net.core.somaxconn=65535
|
||||
net.core.netdev_max_backlog=65535
|
||||
net.ipv4.tcp_max_syn_backlog=65535
|
||||
EOF
|
||||
|
||||
sudo sysctl --system
|
||||
|
||||
# udev rules
|
||||
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
|
||||
KERNEL=="nbd*", OPTIONS+="nowatch"
|
||||
EOF
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
|
||||
# file descriptor limits
|
||||
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
|
||||
* soft nofile 1048576
|
||||
* hard nofile 1048576
|
||||
root soft nofile 1048576
|
||||
root hard nofile 1048576
|
||||
EOF
|
||||
```
|
||||
|
||||
## install toolchain
|
||||
|
||||
```bash
|
||||
# docker
|
||||
curl -fsSL https://get.docker.com | sh
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in
|
||||
docker --version && docker compose version
|
||||
|
||||
# mise
|
||||
curl https://mise.run | sh
|
||||
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
|
||||
```
|
||||
|
||||
280
experiments/e2b/sanbox-usage-generated-from-infra-repo.md
Normal file
280
experiments/e2b/sanbox-usage-generated-from-infra-repo.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# E2B Sandbox Usage Guide
|
||||
|
||||
A practical guide for creating, observing, and using E2B sandboxes via the API.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Local Dev Setup
|
||||
|
||||
Seed the database to create a test team and API key:
|
||||
|
||||
```bash
|
||||
make -C packages/local-dev seed-database
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
|
||||
E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
|
||||
E2B_API_URL=http://localhost:3000
|
||||
E2B_SANDBOX_URL=http://localhost:3002
|
||||
```
|
||||
|
||||
All examples below use `$E2B_API_URL` and `$E2B_API_KEY` — export them in your shell for convenience:
|
||||
|
||||
```bash
|
||||
export E2B_API_URL=http://localhost:3000
|
||||
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Create a Sandbox
|
||||
|
||||
```bash
|
||||
curl -X POST $E2B_API_URL/sandboxes \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"templateID": "base",
|
||||
"timeout": 300
|
||||
}'
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"sandboxID": "i1234567890abcdef",
|
||||
"templateID": "base",
|
||||
"envdVersion": "0.5.8",
|
||||
"domain": "i1234567890abcdef.your-domain"
|
||||
}
|
||||
```
|
||||
|
||||
Save the `sandboxID` and `domain` from the response for subsequent commands.
|
||||
|
||||
### Optional Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `secure` | bool | Returns an `envdAccessToken` for authenticated envd access |
|
||||
| `envVars` | object | Inject environment variables into the VM |
|
||||
| `metadata` | object | Tag the sandbox for filtering (e.g. `{"purpose": "test"}`) |
|
||||
| `autoPause` | bool | Pause instead of kill on timeout |
|
||||
| `allow_internet_access` | bool | Allow internet egress from the VM |
|
||||
| `network.allowPublicTraffic` | bool | Allow inbound public traffic |
|
||||
| `network.allowOut` | array | Allowed egress destinations (IPs, CIDRs, domains) |
|
||||
| `network.denyOut` | array | Denied egress destinations (IPs/CIDRs only) |
|
||||
| `volumeMounts` | array | Mount persistent volumes (`{"name": "vol", "path": "/data"}`) |
|
||||
|
||||
**Example with options:**
|
||||
|
||||
```bash
|
||||
curl -X POST $E2B_API_URL/sandboxes \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"templateID": "base",
|
||||
"timeout": 300,
|
||||
"secure": true,
|
||||
"envVars": {"MY_VAR": "hello"},
|
||||
"metadata": {"purpose": "demo"}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Observe Sandbox State
|
||||
|
||||
### Get details of a specific sandbox
|
||||
|
||||
```bash
|
||||
curl $E2B_API_URL/sandboxes/{sandboxID} \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
Returns state (`running`/`paused`), CPU/memory/disk config, network settings, and TTL (`endAt`).
|
||||
|
||||
### List all sandboxes
|
||||
|
||||
```bash
|
||||
# Running only
|
||||
curl "$E2B_API_URL/v2/sandboxes?state=running" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
# Running and paused
|
||||
curl "$E2B_API_URL/v2/sandboxes?state=running&state=paused" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
# Filter by metadata
|
||||
curl "$E2B_API_URL/v2/sandboxes?metadata=purpose%3Ddemo" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
### Get resource metrics (CPU, memory, disk)
|
||||
|
||||
```bash
|
||||
curl "$E2B_API_URL/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"timestampUnix": 1234567890,
|
||||
"cpuCount": 2,
|
||||
"cpuUsedPct": 25.5,
|
||||
"memUsed": 268435456,
|
||||
"memTotal": 536870912,
|
||||
"diskUsed": 1073741824,
|
||||
"diskTotal": 5368709120
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Get sandbox logs
|
||||
|
||||
```bash
|
||||
curl "$E2B_API_URL/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
Optional query parameters: `cursor` (ms timestamp), `level` (min log level), `search` (substring match).
|
||||
|
||||
---
|
||||
|
||||
## 3. Use the Sandbox
|
||||
|
||||
The in-VM daemon (**envd**) runs on port 49983 inside each sandbox, exposed via the sandbox's `domain`.
|
||||
|
||||
### Upload a file
|
||||
|
||||
```bash
|
||||
curl -X POST https://{domain}/files \
|
||||
-H "Content-Type: multipart/form-data" \
|
||||
-F "file=@script.py" \
|
||||
-F "path=/home/user/script.py"
|
||||
```
|
||||
|
||||
### Using the E2B Python SDK
|
||||
|
||||
Point the SDK at your local API:
|
||||
|
||||
```python
|
||||
from e2b import Sandbox
|
||||
|
||||
sbx = Sandbox("base", api_url="http://localhost:3000")
|
||||
|
||||
# Run a command
|
||||
result = sbx.commands.run("echo 'Hello from Firecracker VM!'")
|
||||
print(result.stdout)
|
||||
|
||||
# Write and execute a file
|
||||
sbx.files.write("/home/user/hello.py", "print('Hello world')")
|
||||
result = sbx.commands.run("python3 /home/user/hello.py")
|
||||
print(result.stdout)
|
||||
|
||||
# List files
|
||||
files = sbx.files.list("/home/user")
|
||||
for f in files:
|
||||
print(f.name)
|
||||
|
||||
sbx.kill()
|
||||
```
|
||||
|
||||
### Envd Connect RPC API
|
||||
|
||||
The envd daemon exposes Connect RPC services for programmatic access:
|
||||
|
||||
**Process Service:**
|
||||
- `Start(ProcessConfig)` — start a new process
|
||||
- `List()` — list running processes
|
||||
- `Connect(ProcessSelector)` — connect to process stdio
|
||||
- `Signal(ProcessSelector, Signal)` — send signal to process
|
||||
|
||||
**Filesystem Service:**
|
||||
- `ListDir(Path)` — list directory contents
|
||||
- `Stat(Path)` — get file metadata
|
||||
- `WatchDir(Path)` — watch for changes
|
||||
- `Move(Source, Dest)` — move/rename file
|
||||
- `RemoveDir(Path)` — remove directory
|
||||
|
||||
---
|
||||
|
||||
## 4. Lifecycle Management
|
||||
|
||||
### Extend timeout
|
||||
|
||||
```bash
|
||||
# Add 60 seconds to the TTL
|
||||
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/refreshes \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"duration": 60}'
|
||||
|
||||
# Or set an absolute timeout (seconds from now)
|
||||
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/timeout \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"timeout": 120}'
|
||||
```
|
||||
|
||||
### Pause (snapshot to disk)
|
||||
|
||||
```bash
|
||||
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/pause \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
### Resume / reconnect
|
||||
|
||||
```bash
|
||||
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/connect \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"timeout": 300}'
|
||||
```
|
||||
|
||||
Returns `200` if already running, `201` if resumed from paused state.
|
||||
|
||||
### Kill
|
||||
|
||||
```bash
|
||||
curl -X DELETE $E2B_API_URL/sandboxes/{sandboxID} \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
```
|
||||
|
||||
### Create a snapshot (template from running sandbox)
|
||||
|
||||
```bash
|
||||
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/snapshots \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name": "my-snapshot"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Network Configuration
|
||||
|
||||
Update network rules on a running sandbox:
|
||||
|
||||
```bash
|
||||
curl -X PUT $E2B_API_URL/sandboxes/{sandboxID}/network \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"allowOut": ["8.8.8.8", "example.com"],
|
||||
"denyOut": ["0.0.0.0/0"]
|
||||
}'
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
- If `allowOut` contains domain names, `denyOut` must include `0.0.0.0/0`
|
||||
- Domain names are not supported in `denyOut` (IPs/CIDRs only)
|
||||
- `allowOut` entries take precedence over `denyOut`
|
||||
- Omitting both fields clears all rules
|
||||
46
experiments/e2b/using-e2b-infra.md
Normal file
46
experiments/e2b/using-e2b-infra.md
Normal file
@@ -0,0 +1,46 @@
|
||||
## VMs
|
||||
|
||||
```bash
|
||||
|
||||
E2B_API_URL=http://192.168.0.61:3000
|
||||
|
||||
# create vm
|
||||
curl -X POST $E2B_API_URL/sandboxes \
|
||||
-H "X-API-Key: $E2B_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"templateID": "base",
|
||||
"timeout": 300
|
||||
}'
|
||||
|
||||
# response
|
||||
{"alias":"base","clientID":"6532622b","domain":null,"envdVersion":"0.5.8","sandboxID":"in3h60s6h0ie3kigrcls4","templateID":"pz1l1owhmy0w84e12eqv","trafficAccessToken":null}
|
||||
|
||||
sandboxID=in3h60s6h0ie3kigrcls4
|
||||
|
||||
# create sandboxVM and capture id
|
||||
sandboxID=$(curl -s -X POST http://localhost:3000/sandboxes -H "X-API-Key: $E2B_API_KEY" -H "Content-Type: application/json" -d '{
|
||||
"templateID": "base",
|
||||
"timeout": 300
|
||||
}' | jq -r .sandboxID
|
||||
)
|
||||
|
||||
|
||||
# get state
|
||||
curl http://localhost:3000/sandboxes/${sandboxID} \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
# list running sandboxes
|
||||
curl "http://localhost:3000/v2/sandboxes?state=running" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
# sandbox resource metrics
|
||||
curl "http://localhost:3000/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
# sandbox logs
|
||||
curl "http://localhost:3000/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
|
||||
-H "X-API-Key: $E2B_API_KEY"
|
||||
|
||||
|
||||
```
|
||||
@@ -0,0 +1,22 @@
|
||||
# Disable the default all-in-one installation
|
||||
shims:
|
||||
disableAll: true
|
||||
firecracker:
|
||||
enabled: true
|
||||
|
||||
# In this chart version, 'env' must be a map, not a list.
|
||||
# The template converts these keys into Environment Variables for you.
|
||||
env:
|
||||
kataArtifacts: "firecracker"
|
||||
multiInstallSuffix: "" # This fixes the specific error you saw
|
||||
installArtifactsOnly: false
|
||||
|
||||
# Automatically create the RuntimeClass
|
||||
runtimeClass:
|
||||
create: true
|
||||
name: kata-fc
|
||||
handler: kata-fc
|
||||
|
||||
# Set Firecracker as the default for amd64 nodes
|
||||
defaultShim:
|
||||
amd64: firecracker
|
||||
@@ -0,0 +1,14 @@
|
||||
apiVersion: node.k8s.io/v1
|
||||
kind: RuntimeClass
|
||||
metadata:
|
||||
name: kata-fc
|
||||
handler: kata-fc
|
||||
scheduling:
|
||||
nodeSelector:
|
||||
# Ensure these pods only land on your specialized node
|
||||
runtime: kata-fc
|
||||
tolerations:
|
||||
- key: "dedicated"
|
||||
operator: "Equal"
|
||||
value: "kata-fc"
|
||||
effect: "NoSchedule"
|
||||
198
experiments/kata_and_fc_on_kube/readme.md
Normal file
198
experiments/kata_and_fc_on_kube/readme.md
Normal file
@@ -0,0 +1,198 @@
|
||||
## Node: kube-node-34
|
||||
|
||||
```bash
|
||||
apt install cpu-checker
|
||||
|
||||
# check kvm availability
|
||||
kvm-ok
|
||||
cat /sys/module/kvm_amd/parameters/nested
|
||||
lsmod | grep kvm
|
||||
|
||||
# install kata containers
|
||||
# using kata-deploy ?
|
||||
|
||||
export VERSION=$(curl -sSL https://api.github.com/repos/kata-containers/kata-containers/releases/latest | jq .tag_name | tr -d '"')
|
||||
export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy"
|
||||
|
||||
helm upgrade --install kata-deploy "${CHART}" --version "${VERSION}" --values experiments/kata_and_fc_on_kube/manifests/kata-deploy-values.yaml
|
||||
|
||||
# label only specific nodes to be used with kata-fc
|
||||
kubectl label node kube-node-34 runtime=kata-fc
|
||||
# taint node with NoSchedule
|
||||
kubectl taint nodes kube-node-34 dedicated=kata-fc:NoSchedule
|
||||
|
||||
# configure runtime class
|
||||
k apply -f experiments/kata_and_fc_on_kube/manifests//runtime-class_kata-fc.yaml
|
||||
|
||||
# devmapper configuration for firecracker in containerd
|
||||
ctr plugins ls | grep devmapper
|
||||
|
||||
# create data and metadata files
|
||||
# might want to replace this with LVM !!!!
|
||||
# sudo mkdir -p /var/lib/containerd/devmapper
|
||||
# sudo truncate -s 10G /var/lib/containerd/devmapper/data
|
||||
# sudo truncate -s 1G /var/lib/containerd/devmapper/init_metadata
|
||||
|
||||
# # Associate the files with Loop Devices
|
||||
# sudo losetup /dev/loop10 /var/lib/containerd/devmapper/data
|
||||
# sudo losetup /dev/loop11 /var/lib/containerd/devmapper/init_metadata
|
||||
|
||||
# # Use dmsetup to create the pool:
|
||||
# # This command creates the mapping.
|
||||
# # The numbers '0 20971520' represent the size in 512-byte sectors (for a 10GB file).
|
||||
# sudo dmsetup create containerd-pool --table "0 20971520 thin-pool /dev/loop11 /dev/loop10 128 32768 1"
|
||||
|
||||
lvcreate -L 10G -T kata-vg/kata-pool
|
||||
|
||||
# Add the configuration to /etc/containerd/config.toml:
|
||||
# Find the [plugins."io.containerd.snapshotter.v1.devmapper"] section and update it:
|
||||
|
||||
[plugins."io.containerd.snapshotter.v1.devmapper"]
|
||||
# LVM uses a specific naming convention in /dev/mapper/
|
||||
# It is VolumeGroupName-LogicalVolumeName
|
||||
pool_name = "kata--vg-kata--pool"
|
||||
root_path = "/var/lib/containerd/devmapper"
|
||||
base_image_size = "10GB"
|
||||
discard_blocks = true
|
||||
|
||||
|
||||
# restart containerd
|
||||
systemctl restart containerd
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
|
||||
### kube node preparation & deployment
|
||||
|
||||
```bash
|
||||
# ---------------------------------------------------------------------------
|
||||
# Packages
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
qemu-guest-agent \
|
||||
openssh-server \
|
||||
apt-transport-https \
|
||||
ca-certificates \
|
||||
curl \
|
||||
gnupg \
|
||||
nvme-cli
|
||||
|
||||
systemctl enable --now qemu-guest-agent
|
||||
systemctl enable --now ssh
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# nvme-tcp
|
||||
# ---------------------------------------------------------------------------
|
||||
apt-get install -y linux-modules-extra-$(uname -r)
|
||||
modprobe nvme-tcp
|
||||
echo "nvme-tcp" >> /etc/modules-load.d/nvme-tcp.conf
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Kernel modules for Kubernetes
|
||||
# ---------------------------------------------------------------------------
|
||||
cat > /etc/modules-load.d/k8s.conf <<'EOF'
|
||||
overlay
|
||||
br_netfilter
|
||||
EOF
|
||||
|
||||
modprobe overlay
|
||||
modprobe br_netfilter
|
||||
|
||||
cat > /etc/sysctl.d/k8s.conf <<'EOF'
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
net.ipv4.ip_forward = 1
|
||||
EOF
|
||||
|
||||
sysctl --system
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# containerd
|
||||
# ---------------------------------------------------------------------------
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
|
||||
> /etc/apt/sources.list.d/docker.list
|
||||
apt-get update && apt-get install -y containerd.io
|
||||
|
||||
cat > /etc/containerd/config.toml <<'EOF'
|
||||
version = 2
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
|
||||
runtime_type = "io.containerd.runc.v2"
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
|
||||
SystemdCgroup = true
|
||||
[plugins."io.containerd.grpc.v1.cri".registry]
|
||||
config_path = "/etc/containerd/certs.d"
|
||||
EOF
|
||||
|
||||
# Registry mirrors pointing to Zot at 192.168.0.30:5000
|
||||
mkdir -p \
|
||||
/etc/containerd/certs.d/docker.io \
|
||||
/etc/containerd/certs.d/registry.k8s.io \
|
||||
/etc/containerd/certs.d/ghcr.io \
|
||||
/etc/containerd/certs.d/quay.io
|
||||
|
||||
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'EOF'
|
||||
server = "https://registry-1.docker.io"
|
||||
[host."http://192.168.0.30:5000/v2/docker.io"]
|
||||
capabilities = ["pull", "resolve"]
|
||||
skip_verify = true
|
||||
override_path = true
|
||||
EOF
|
||||
|
||||
cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml <<'EOF'
|
||||
server = "https://registry.k8s.io"
|
||||
[host."http://192.168.0.30:5000/v2/registry.k8s.io"]
|
||||
capabilities = ["pull", "resolve"]
|
||||
skip_verify = true
|
||||
override_path = true
|
||||
EOF
|
||||
|
||||
cat > /etc/containerd/certs.d/ghcr.io/hosts.toml <<'EOF'
|
||||
server = "https://ghcr.io"
|
||||
[host."http://192.168.0.30:5000/v2/ghcr.io"]
|
||||
capabilities = ["pull", "resolve"]
|
||||
skip_verify = true
|
||||
override_path = true
|
||||
EOF
|
||||
|
||||
cat > /etc/containerd/certs.d/quay.io/hosts.toml <<'EOF'
|
||||
server = "https://quay.io"
|
||||
[host."http://192.168.0.30:5000/v2/quay.io"]
|
||||
capabilities = ["pull", "resolve"]
|
||||
skip_verify = true
|
||||
override_path = true
|
||||
EOF
|
||||
|
||||
systemctl restart containerd
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# kubelet systemd drop-in
|
||||
# ---------------------------------------------------------------------------
|
||||
mkdir -p /etc/systemd/system/kubelet.service.d
|
||||
cat > /etc/systemd/system/kubelet.service.d/10-containerd.conf <<'EOF'
|
||||
[Unit]
|
||||
After=containerd.service
|
||||
Requires=containerd.service
|
||||
|
||||
[Service]
|
||||
ExecStartPre=/bin/bash -c 'until [ -S /var/run/containerd/containerd.sock ]; do sleep 1; done'
|
||||
ExecStartPre=/usr/bin/crictl info
|
||||
EOF
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# kubeadm / kubelet / kubectl v1.32
|
||||
# ---------------------------------------------------------------------------
|
||||
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
|
||||
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
|
||||
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" \
|
||||
> /etc/apt/sources.list.d/kubernetes.list
|
||||
apt-get update && apt-get install -y kubelet kubeadm kubectl
|
||||
apt-mark hold kubelet kubeadm kubectl
|
||||
|
||||
|
||||
# join kube cluster
|
||||
JOIN_COMMAND="kubeadm join 192.168.0.31:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>"
|
||||
14
experiments/microsandbox/install.md
Normal file
14
experiments/microsandbox/install.md
Normal file
@@ -0,0 +1,14 @@
|
||||
```bash
|
||||
virt-install \
|
||||
--name microsandbox \
|
||||
--ram 8192\
|
||||
--vcpus 4 \
|
||||
--cpu host-passthrough \
|
||||
--os-variant ubuntu24.04 \
|
||||
--disk path=/srv/vms/microsandbox.qcow2,size=100,format=qcow2,bus=virtio \
|
||||
--network bridge=br0,model=virtio \
|
||||
--graphics none \
|
||||
--console pty,target_type=serial \
|
||||
--location /srv/vms/isos/ubuntu-24.04.3-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
|
||||
--extra-args 'console=ttyS0,115200n8'
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
#cloud-config
|
||||
hostname: ${hostname}
|
||||
manage_etc_hosts: true
|
||||
|
||||
# Kernel modules for container networking
|
||||
write_files:
|
||||
- path: /etc/modules-load.d/k8s.conf
|
||||
content: |
|
||||
overlay
|
||||
br_netfilter
|
||||
- path: /etc/sysctl.d/k8s.conf
|
||||
content: |
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
net.ipv4.ip_forward = 1
|
||||
|
||||
packages:
|
||||
- curl
|
||||
- gpg
|
||||
- apt-transport-https
|
||||
|
||||
runcmd:
|
||||
- modprobe overlay
|
||||
- modprobe br_netfilter
|
||||
- sysctl --system
|
||||
- swapoff -a
|
||||
- sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
|
||||
74
experiments/orb-stack/modules/base-template/main.tf
Normal file
74
experiments/orb-stack/modules/base-template/main.tf
Normal file
@@ -0,0 +1,74 @@
|
||||
terraform {
|
||||
required_providers {
|
||||
orbstack = {
|
||||
source = "robertdebock/orbstack"
|
||||
version = "~> 3.0"
|
||||
}
|
||||
cloudinit = {
|
||||
source = "hashicorp/cloudinit"
|
||||
version = "~> 2.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
variable "node_count" {
|
||||
description = "Number of nodes to deploy"
|
||||
type = number
|
||||
default = 3
|
||||
}
|
||||
|
||||
variable "name" {
|
||||
description = "Base name for the machines"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "distro" {
|
||||
description = "OS distribution"
|
||||
type = string
|
||||
default = "ubuntu"
|
||||
}
|
||||
|
||||
variable "distro_version" {
|
||||
description = "OS distribution version/codename"
|
||||
type = string
|
||||
default = "noble"
|
||||
}
|
||||
|
||||
variable "extra_cloud_init_parts" {
|
||||
description = "Additional cloud-init parts to layer on top of the base config"
|
||||
type = list(object({ content = string, content_type = string }))
|
||||
default = []
|
||||
}
|
||||
|
||||
data "cloudinit_config" "this" {
|
||||
count = var.node_count
|
||||
|
||||
part {
|
||||
content_type = "text/cloud-config"
|
||||
content = templatefile("${path.module}/cloud-init-base.yaml", {
|
||||
hostname = "${var.name}-${count.index + 1}"
|
||||
})
|
||||
}
|
||||
|
||||
dynamic "part" {
|
||||
for_each = var.extra_cloud_init_parts
|
||||
content {
|
||||
content_type = part.value.content_type
|
||||
content = part.value.content
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "orbstack_machine" "this" {
|
||||
count = var.node_count
|
||||
|
||||
name = "${var.name}-${count.index + 1}"
|
||||
distro = var.distro
|
||||
region = var.distro_version
|
||||
|
||||
user_data = data.cloudinit_config.this[count.index].rendered
|
||||
}
|
||||
|
||||
output "machines" {
|
||||
value = { for m in orbstack_machine.this : m.name => m.ip_address }
|
||||
}
|
||||
11
experiments/orb-stack/stacks/k3s/cloud-init-cp.yaml
Normal file
11
experiments/orb-stack/stacks/k3s/cloud-init-cp.yaml
Normal file
@@ -0,0 +1,11 @@
|
||||
#cloud-config
|
||||
merge_how:
|
||||
- name: list
|
||||
settings: [append]
|
||||
- name: dict
|
||||
settings: [recurse_array]
|
||||
|
||||
runcmd:
|
||||
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" K3S_TOKEN="${k3s_token}" sh -
|
||||
- until kubectl get nodes; do sleep 2; done
|
||||
- cp /etc/rancher/k3s/k3s.yaml /root/kubeconfig.yaml
|
||||
9
experiments/orb-stack/stacks/k3s/cloud-init-worker.yaml
Normal file
9
experiments/orb-stack/stacks/k3s/cloud-init-worker.yaml
Normal file
@@ -0,0 +1,9 @@
|
||||
#cloud-config
|
||||
merge_how:
|
||||
- name: list
|
||||
settings: [append]
|
||||
- name: dict
|
||||
settings: [recurse_array]
|
||||
|
||||
runcmd:
|
||||
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" K3S_URL="https://${cp_ip}:6443" K3S_TOKEN="${join_token}" sh -
|
||||
39
experiments/orb-stack/stacks/k3s/main.tf
Normal file
39
experiments/orb-stack/stacks/k3s/main.tf
Normal file
@@ -0,0 +1,39 @@
|
||||
variable "k3s_token" {
|
||||
description = "Shared secret for k3s cluster join (set via TF_VAR or tfvars)"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
module "control_plane" {
|
||||
source = "../../modules/base-template"
|
||||
name = "k3s-cp"
|
||||
node_count = 1
|
||||
|
||||
extra_cloud_init_parts = [{
|
||||
content_type = "text/cloud-config"
|
||||
content = templatefile("${path.module}/cloud-init-cp.yaml", {
|
||||
k3s_token = var.k3s_token
|
||||
})
|
||||
}]
|
||||
}
|
||||
|
||||
module "workers" {
|
||||
source = "../../modules/base-template"
|
||||
name = "k3s-worker"
|
||||
node_count = 2
|
||||
|
||||
extra_cloud_init_parts = [{
|
||||
content_type = "text/cloud-config"
|
||||
content = templatefile("${path.module}/cloud-init-worker.yaml", {
|
||||
cp_ip = values(module.control_plane.machines)[0]
|
||||
join_token = var.k3s_token
|
||||
})
|
||||
}]
|
||||
}
|
||||
|
||||
output "cluster" {
|
||||
value = {
|
||||
control_plane = module.control_plane.machines
|
||||
workers = module.workers.machines
|
||||
}
|
||||
}
|
||||
219
experiments/rke2/rke2-basics.md
Normal file
219
experiments/rke2/rke2-basics.md
Normal file
@@ -0,0 +1,219 @@
|
||||
## install
|
||||
|
||||
```bash
|
||||
# master node
|
||||
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.32.12+rke2r1 sh -
|
||||
systemctl enable rke2-server.service
|
||||
systemctl start rke2-server.service
|
||||
journalctl -u rke2-server -f
|
||||
|
||||
# open firewalld
|
||||
sudo firewall-cmd --permanent --add-port=9345/tcp
|
||||
sudo firewall-cmd --permanent --add-port=6443/tcp
|
||||
sudo firewall-cmd --permanent --add-port=10250/tcp # Kubelet
|
||||
sudo firewall-cmd --reload
|
||||
|
||||
# install nerdctl
|
||||
# Set the version
|
||||
VERSION="2.2.1" # Check GitHub for the latest version
|
||||
|
||||
# Download the tarball
|
||||
wget https://github.com/containerd/nerdctl/releases/download/v${VERSION}/nerdctl-${VERSION}-linux-arm64.tar.gz
|
||||
|
||||
# Extract to your path
|
||||
sudo tar -C /usr/local/bin -xzvf nerdctl-${VERSION}-linux-arm64.tar.gz nerdctl
|
||||
|
||||
# configure nerdctl
|
||||
sudo mkdir -p /etc/nerdctl
|
||||
sudo tee /etc/nerdctl/nerdctl.toml <<EOF
|
||||
address = "unix:///run/k3s/containerd/containerd.sock"
|
||||
namespace = "k8s.io"
|
||||
EOF
|
||||
|
||||
# install buildkit
|
||||
# Set current stable version
|
||||
BK_VER="0.28.0"
|
||||
|
||||
# Download arm64 binary
|
||||
wget https://github.com/moby/buildkit/releases/download/v${BK_VER}/buildkit-v${BK_VER}.linux-arm64.tar.gz
|
||||
|
||||
# Extract only the binaries to /usr/local/bin
|
||||
sudo tar -C /usr/local/bin -xzvf buildkit-v${BK_VER}.linux-arm64.tar.gz --strip-components=1 bin/
|
||||
|
||||
# Create the service file
|
||||
sudo tee /etc/systemd/system/buildkit.service <<EOF
|
||||
[Unit]
|
||||
Description=BuildKit
|
||||
Documentation=https://github.com/moby/buildkit
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/buildkitd --addr unix:///run/buildkit/buildkitd.sock
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# Reload and Start
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now buildkit
|
||||
|
||||
# ---------------------------------------------
|
||||
|
||||
|
||||
# agent/worker node
|
||||
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_VERSION=v1.32.12+rke2r1 sh -
|
||||
systemctl enable rke2-agent.service
|
||||
mkdir -p /etc/rancher/rke2/
|
||||
# token from master node
|
||||
# cat /var/lib/rancher/rke2/server/node-token
|
||||
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
|
||||
server: https://192.168.64.3:9345
|
||||
token: K107618960f87b9efb3a3255ce00a9743d29f1db9376820c9144cb85fa3c554dc69::server:06b2effdf0c9ce3952efc8a5d80bf084
|
||||
EOF
|
||||
systemctl start rke2-agent.service
|
||||
journalctl -u rke2-agent -f
|
||||
|
||||
|
||||
# Set up kubectl on the server node
|
||||
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
|
||||
echo 'export PATH=$PATH:/var/lib/rancher/rke2/bin' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
|
||||
|
||||
## build and deploy application
|
||||
|
||||
```bash
|
||||
# build container with nerdctl
|
||||
nerdctl --namespace k8s.io build --tag hello-world:latest .
|
||||
|
||||
# export image as tar on master node
|
||||
nerdctl save hello-world:latest -o hello-world.tar
|
||||
# copy it over to worker node
|
||||
scp hello-world.tar novakj@192.168.64.4:~/
|
||||
# import image on the agent node
|
||||
sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import hello-world.tar
|
||||
|
||||
kubectl create namespace rke2-apps
|
||||
|
||||
cat <<EOF > deployment.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hello-world-deployment
|
||||
namespace: rke2-apps
|
||||
labels:
|
||||
type: staticwebapp
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
type: staticwebapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
type: staticwebapp
|
||||
spec:
|
||||
containers:
|
||||
- name: staticwebapp
|
||||
image: hello-world:latest
|
||||
imagePullPolicy: Never
|
||||
ports:
|
||||
- containerPort: 80
|
||||
resources:
|
||||
requests:
|
||||
memory: "32Mi"
|
||||
cpu: "200m"
|
||||
limits:
|
||||
memory: "64Mi"
|
||||
cpu: "300m"
|
||||
EOF
|
||||
|
||||
kubectl create -f deployment.yaml
|
||||
|
||||
# expose deployment
|
||||
kubectl expose deployment hello-world-deployment --name hello-world-service --port=8080 --target-port=80 -n rke2-apps
|
||||
|
||||
# install ingress-nginx (even though i thought that there is ingress controller already deployed)
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.2/deploy/static/provider/cloud/deploy.yaml
|
||||
|
||||
# create ingress with "localhost" as host
|
||||
kubectl create ingress hello-world-ingress --class=nginx --rule="test-host/*=hello-world-service:8080" -n rke2-apps
|
||||
kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 8081:80
|
||||
|
||||
|
||||
|
||||
# incomplete completion configuration ;-)
|
||||
dnf install bash-completion -y
|
||||
alias 'k=kubectl'
|
||||
|
||||
# ~/.bashrc
|
||||
|
||||
# 1. Load the main bash-completion package first
|
||||
# On Rocky/RHEL, it's usually at this path:
|
||||
[[ -r "/usr/share/bash-completion/bash_completion" ]] && . "/usr/share/bash-completion/bash_completion"
|
||||
|
||||
# Enable kubectl bash completion
|
||||
source <(kubectl completion bash)
|
||||
|
||||
# Set up the alias
|
||||
alias k=kubectl
|
||||
|
||||
# Link the kubectl completion logic to the 'k' alias
|
||||
complete -o default -F __start_kubectl k
|
||||
```
|
||||
|
||||
|
||||
|
||||
## upgrading RKE2
|
||||
|
||||
```bash
|
||||
# install upgrade controller
|
||||
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.9.1/system-upgrade-controller.yaml
|
||||
|
||||
# server upgrade
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: upgrade.cattle.io/v1
|
||||
kind: Plan
|
||||
metadata:
|
||||
name: rke2-server-upgrade
|
||||
namespace: system-upgrade
|
||||
spec:
|
||||
concurrency: 1
|
||||
cordon: true
|
||||
nodeSelector:
|
||||
matchExpressions:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: In
|
||||
values: ["true"]
|
||||
serviceAccountName: system-upgrade
|
||||
upgrade:
|
||||
image: rancher/rke2-upgrade
|
||||
version: v1.33.9+rke2r1
|
||||
EOF
|
||||
|
||||
# agent upgrade
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: upgrade.cattle.io/v1
|
||||
kind: Plan
|
||||
metadata:
|
||||
name: rke2-agent-upgrade
|
||||
namespace: system-upgrade
|
||||
spec:
|
||||
concurrency: 1
|
||||
cordon: true
|
||||
nodeSelector:
|
||||
matchExpressions:
|
||||
- key: node-role.kubernetes.io/control-plane
|
||||
operator: DoesNotExist
|
||||
prepare:
|
||||
# Logic: "Don't start workers until servers are done"
|
||||
args: ["wait-for-plan", "rke2-server-upgrade"]
|
||||
image: rancher/rke2-upgrade
|
||||
serviceAccountName: system-upgrade
|
||||
upgrade:
|
||||
image: rancher/rke2-upgrade
|
||||
version: v1.33.9+rke2r1
|
||||
EOF
|
||||
```
|
||||
215
experiments/rke2/step-by-step.md
Normal file
215
experiments/rke2/step-by-step.md
Normal file
@@ -0,0 +1,215 @@
|
||||
## VMS creation
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
# hypervisor: beelink (192.168.0.6)
|
||||
|
||||
vms_path=/srv/vms/images
|
||||
isos_path=/srv/vms/isos
|
||||
cd $isos_path
|
||||
# Grab Ubuntu 24.04 cloud image
|
||||
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
||||
|
||||
# Create a base disk from the cloud image (repeat per node)
|
||||
for NODE in rke2-server rke2-agent1 rke2-agent2; do
|
||||
DISK_SIZE="30G"
|
||||
[[ "$NODE" == rke2-agent* ]] && DISK_SIZE="50G"
|
||||
|
||||
qemu-img create -f qcow2 -F qcow2 -b $isos_path/noble-server-cloudimg-amd64.img $vms_path/${NODE}.qcow2
|
||||
qemu-img resize $vms_path/${NODE}.qcow2 ${DISK_SIZE}
|
||||
done
|
||||
```
|
||||
|
||||
|
||||
```bash
|
||||
# prepare cloud-init/user-data
|
||||
apt install cloud-image-utils
|
||||
|
||||
# prepare cloud-init and launch VMs
|
||||
declare -A nodes=(
|
||||
[rke2-server]="192.168.0.51"
|
||||
[rke2-agent1]="192.168.0.52"
|
||||
[rke2-agent2]="192.168.0.53"
|
||||
)
|
||||
|
||||
for node in "${!nodes[@]}"; do
|
||||
ip="${nodes[$node]}"
|
||||
|
||||
# user-data
|
||||
cat <<EOF > user-data-${node}
|
||||
#cloud-config
|
||||
hostname: ${node}
|
||||
manage_etc_hosts: false
|
||||
users:
|
||||
- name: sre
|
||||
sudo: ALL=(ALL) NOPASSWD:ALL
|
||||
shell: /bin/bash
|
||||
ssh_authorized_keys:
|
||||
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV jan.novak@Jans-MacBook-Air.local
|
||||
write_files:
|
||||
- path: /etc/hosts
|
||||
append: true
|
||||
content: |
|
||||
192.168.0.51 rke2-server
|
||||
192.168.0.52 rke2-agent1
|
||||
192.168.0.53 rke2-agent2
|
||||
- path: /etc/modules-load.d/rke2.conf
|
||||
content: |
|
||||
br_netfilter
|
||||
overlay
|
||||
- path: /etc/sysctl.d/99-rke2.conf
|
||||
content: |
|
||||
net.bridge.bridge-nf-call-iptables = 1
|
||||
net.bridge.bridge-nf-call-ip6tables = 1
|
||||
net.ipv4.ip_forward = 1
|
||||
swap:
|
||||
filename: /swap.img
|
||||
size: 0
|
||||
maxsize: 0
|
||||
runcmd:
|
||||
- swapoff -a
|
||||
- sed -i '/swap/d' /etc/fstab
|
||||
- modprobe br_netfilter
|
||||
- modprobe overlay
|
||||
- sysctl --system
|
||||
package_update: true
|
||||
packages:
|
||||
- qemu-guest-agent
|
||||
- nfs-common
|
||||
- open-iscsi
|
||||
power_state:
|
||||
mode: reboot
|
||||
EOF
|
||||
|
||||
# network config
|
||||
cat <<EOF > network-config-${node}
|
||||
network:
|
||||
version: 2
|
||||
ethernets:
|
||||
eth0:
|
||||
match:
|
||||
driver: "virtio_net"
|
||||
addresses:
|
||||
- "${ip}/24"
|
||||
nameservers:
|
||||
addresses:
|
||||
- 8.8.8.8
|
||||
routes:
|
||||
- to: "default"
|
||||
via: "192.168.0.4"
|
||||
EOF
|
||||
|
||||
cloud-localds --network-config=./network-config-${node} \
|
||||
$vms_path/${node}-seed.iso ./user-data-${node}
|
||||
done
|
||||
|
||||
# Launch VMs
|
||||
for node in "${!nodes[@]}"; do
|
||||
virt-install \
|
||||
--name ${node} \
|
||||
--ram 4096 --vcpus 2 \
|
||||
--os-variant ubuntu24.04 \
|
||||
--disk $vms_path/${node}.qcow2,bus=virtio \
|
||||
--disk $vms_path/${node}-seed.iso,device=cdrom \
|
||||
--network bridge=br0,model=virtio \
|
||||
--graphics none \
|
||||
--console pty,target_type=serial \
|
||||
--noautoconsole \
|
||||
--import
|
||||
done
|
||||
|
||||
```
|
||||
|
||||
|
||||
## RKE2 installation
|
||||
|
||||
```bash
|
||||
# there are no .deb packages - only rpm or tarball
|
||||
# "magic" install script can handle that
|
||||
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.32 sudo sh -
|
||||
|
||||
# Create config directory
|
||||
sudo mkdir -p /etc/rancher/rke2
|
||||
|
||||
# Server configuration
|
||||
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
|
||||
# Bind the API to the node's IP (not 127.0.0.1)
|
||||
tls-san:
|
||||
- rke2-server
|
||||
- 192.168.0.51
|
||||
# Write kubeconfig readable by non-root
|
||||
write-kubeconfig-mode: "0644"
|
||||
# CNI - canal is default and fine for home lab
|
||||
# Alternatives: cilium, calico, multus
|
||||
cni:
|
||||
- canal
|
||||
# Disable servicelb if you plan to use metallb
|
||||
disable:
|
||||
- rke2-service-lb
|
||||
EOF
|
||||
|
||||
# Enable and start
|
||||
sudo systemctl enable rke2-server.service
|
||||
sudo systemctl start rke2-server.service
|
||||
|
||||
# Watch bootstrap (takes 2-3 min on first run)
|
||||
sudo journalctl -u rke2-server -f
|
||||
|
||||
```
|
||||
|
||||
|
||||
## Grab the join token and kubeconfig
|
||||
|
||||
```bash
|
||||
|
||||
# Token for agents to join
|
||||
sudo cat /var/lib/rancher/rke2/server/node-token
|
||||
# Save this somewhere - you'll need it on every agent
|
||||
|
||||
# Set up kubectl on the server node
|
||||
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
|
||||
echo 'export PATH=$PATH:/var/lib/rancher/rke2/bin' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
|
||||
# Verify
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
|
||||
## Install RKE2 Agents (Workers)
|
||||
|
||||
```bash
|
||||
# Install RKE2 agent - same channel as server
|
||||
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.32 INSTALL_RKE2_TYPE=agent sudo sh -
|
||||
|
||||
# Create config
|
||||
sudo mkdir -p /etc/rancher/rke2
|
||||
|
||||
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
|
||||
server: https://192.168.0.51:9345
|
||||
token: K10dba0bfff01d610ffed41c6b82a1b8861ee19e5af34a3bdd21970936823a846da::server:1f02fb36a288dcd06770cab28b015bd7
|
||||
EOF
|
||||
|
||||
# Enable and start
|
||||
sudo systemctl enable rke2-agent.service
|
||||
sudo systemctl start rke2-agent.service
|
||||
|
||||
# Watch join
|
||||
sudo journalctl -u rke2-agent -f
|
||||
|
||||
```
|
||||
|
||||
|
||||
## Copy Kubeconfig to Your Workstation
|
||||
|
||||
```bash
|
||||
# On your workstation (not the VMs)
|
||||
scp sre@192.168.0.51:/etc/rancher/rke2/rke2.yaml ~/.kube/rke2-homelab.yaml
|
||||
|
||||
# Fix the server address (it'll say 127.0.0.1)
|
||||
gsed -i 's/127.0.0.1/192.168.0.51/' ~/.kube/rke2-homelab.yaml
|
||||
|
||||
export KUBECONFIG=~/.kube/rke2-homelab.yaml
|
||||
kubectl get nodes
|
||||
```
|
||||
Reference in New Issue
Block a user