kacerr/home-kubernetes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

misc: zot registry, k8s OIDC, server configs, sandbox experiments, and notes

Browse Source 
- docker-30/zot: add Zot OCI registry with on-demand sync to docker.io,
  registry.k8s.io, ghcr.io, quay.io
- kubernetes-kvm-terraform: wire Kanidm OIDC via structured
  AuthenticationConfiguration; add reference apiserver manifest and
  join-node-02 helper
- servers: reorganize shadow/ under servers/, add saint vhost config and
  utility-101 VM definition, add shadow hrajfrisbee.cz vhost and
  storage-23 notes
- experiments: add notes and configs for e2b dev VM, kata + firecracker
  on kube, microsandbox, orb-stack k3s (terraform + cloud-init), rke2
- vms/docker: document tailscale + node-exporter setup
- blog: stub post on Gateway API
- chore: gitignore tmp/, smtp_password, and the two local-only
  credential caches; add per-project .claude/settings.json

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-05-01 18:12:38 +02:00
parent 5ca27a832b
commit 80d0cc1168
34 changed files with 2814 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.claude/settings.json Normal file
View File

@@ -0,0 +1,10 @@
{
"permissions": {
"allow": [
"Bash(for f:*)",
"Bash(do echo:*)",
"Read(//Users/jan.novak/srv/personal/home-kubernetes/**)",
"Bash(done)"
]
}
}

7
.gitignore vendored
View File

@@ -5,3 +5,10 @@
kubernetes-kvm-terraform/join-command.txt
kubernetes-kvm-terraform/kubeconfig
tmp/
vms/utility-101-shadow/docker/monitoring/smtp_password
docker-30/zot/sync-credentials.json
kubernetes-kvm-terraform/gke_gcloud_auth_plugin_cache

7
blog/content/001-having-fun-with-gatewayApi.md Normal file
View File

@@ -0,0 +1,7 @@
# How i did have some fun with gatewayApi
As nginx-ingress will be no longer maintained it looks like the time to upgrade is coming even for the lazy old schoolers who were very happy with it (even though annotations and custom snippets might not be the cleanest way, people experienced with nginx might have liked it for a reason). Although I tried to pretend that there is no gatewayApi and ingress is the thing for quite some time recently everything aligned for me to give it a go.
## Which implementation to choose?
There is a number of options and I'm only vaguely aware of them as I was not interested in it for a long time. But i have a tendency of trying to do something with Envoy (personal feeling that it is the right next thing here?) and I'm using the Cilium network plugin in my homelab kubernetes cluster. All this considered answer is obvious: use envoy gateway which is part of cilium.

52
docker-30/zot/config.yaml Normal file
View File

@@ -0,0 +1,52 @@
distSpecVersion: "1.1.0"
storage:
rootDirectory: /var/lib/zot
gc: true
gcDelay: "24h"
dedupe: true
http:
address: 0.0.0.0
port: 5000
compat:
- docker2s2
log:
# level: info
level: debug
extensions:
ui:
enable: true
search:
enable: true
sync:
enable: true
credentialsFile: "/etc/zot/sync-credentials.json"
registries:
- urls: ["https://registry-1.docker.io"]
onDemand: true
tlsVerify: true
content:
- prefix: "library/**"
destination: "/docker.io/library"
- prefix: "democraticcsi/**"
destination: "/democraticcsi"
- prefix: "**"
destination: "/docker.io"
- urls: ["https://registry.k8s.io"]
onDemand: true
tlsVerify: true
content:
- prefix: "**"
destination: "/registry.k8s.io"
- urls: ["https://ghcr.io"]
onDemand: true
tlsVerify: true
content:
- prefix: "**"
destination: "/ghcr.io"
- urls: ["https://quay.io"]
onDemand: true
tlsVerify: true
content:
- prefix: "**"
destination: "/quay.io"

12
docker-30/zot/docker-compose.yaml Normal file
View File

@@ -0,0 +1,12 @@
services:
zot:
image: ghcr.io/project-zot/zot-linux-amd64:latest
container_name: zot
restart: unless-stopped
command: serve /etc/zot/config.yaml
ports:
- "5000:5000"
volumes:
- ./config.yaml:/etc/zot/config.yaml:ro
- ./sync-credentials.json:/etc/zot/sync-credentials.json:ro
- /srv/container-registry-data:/var/lib/zot

391
experiments/e2b/dev-vm-deployment-plan.md Normal file
View File

@@ -0,0 +1,391 @@
# E2B Dev VM Setup on KVM Homelab
## Context
The user wants to run the E2B infrastructure dev stack on a KVM virtual machine in their homelab. E2B uses Firecracker microVMs (which need `/dev/kvm`), so the guest VM needs **nested virtualization** (KVM-in-KVM). This guide covers VM creation, OS configuration, toolchain installation, and running the full dev stack.
> **Note from upstream**: [DEV-LOCAL.md](DEV-LOCAL.md) says "Linux is required. This is a work in progress. Not everything will function as expected."
---
## Phase 1: Create the KVM VM on the Hypervisor Host
### 1.1 Enable nested virtualization on the host
**Intel:**
```bash
cat /sys/module/kvm_intel/parameters/nested # check
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel nested=1
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
```
**AMD:**
```bash
cat /sys/module/kvm_amd/parameters/nested
sudo modprobe -r kvm_amd && sudo modprobe kvm_amd nested=1
echo "options kvm_amd nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
# get details about loaded kernel module
systool -v -m kvm_amd
```
### 1.2 Create the VM
```bash
virt-install \
--name e2b-dev \
--ram 16384 \
--vcpus 8 \
--cpu host-passthrough \
--os-variant ubuntu24.04 \
--disk path=/var/lib/libvirt/images/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
--network bridge=virbr0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--cdrom /path/to/ubuntu-24.04-live-server-amd64.iso \
--extra-args 'console=ttyS0,115200n8'
```
**Why these specs:**
| Resource | Value | Rationale |
|----------|-------|-----------|
| RAM | 16 GB (24-32 better) | 4 GB for huge pages (2048 x 2MB), ~4 GB for 10 Docker containers, rest for Go services + Firecracker VMs |
| vCPUs | 8 | Firecracker VMs consume vCPUs; Go services are concurrent |
| Disk | 100 GB | Docker images, FC binaries, kernels, rootfs, Go cache, build artifacts |
| CPU | `host-passthrough` | **Mandatory** -- exposes VMX/SVM to guest so `/dev/kvm` works inside the VM |
### 1.3 Alternative: libvirt XML
If you manage VMs declaratively, the critical part is:
```xml
<cpu mode='host-passthrough' check='none' migratable='off'/>
```
### 1.4 Verify nested KVM works (after OS install)
```bash
ls -la /dev/kvm # must exist
lsmod | grep kvm # kvm + kvm_intel/kvm_amd
grep -cE '(vmx|svm)' /proc/cpuinfo # must be > 0
```
If `/dev/kvm` is missing, go back to 1.1.
---
## Phase 2: OS Configuration (inside the guest VM)
**Recommended OS:** Ubuntu 24.04 LTS Server (matches CI, well-tested with Firecracker)
### 2.1 Base packages
```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
build-essential git curl wget unzip jq make gcc pkg-config \
iptables iproute2 net-tools ca-certificates gnupg \
lsb-release software-properties-common gettext-base
```
### 2.2 Kernel modules
```bash
# Load now
sudo modprobe nbd nbds_max=64
sudo modprobe kvm
sudo modprobe kvm_intel # or kvm_amd
sudo modprobe tun
sudo modprobe veth
sudo modprobe nf_tables
sudo modprobe nft_nat
# Persist across reboots
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
nbd
kvm
kvm_intel
tun
veth
nf_tables
nft_nat
EOF
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
```
### 2.3 Sysctl
```bash
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
vm.nr_hugepages=2048
vm.max_map_count=1048576
vm.swappiness=10
vm.vfs_cache_pressure=50
net.ipv4.ip_forward=1
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF
sudo sysctl --system
```
### 2.4 Udev rules (suppress NBD inotify noise)
```bash
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
KERNEL=="nbd*", OPTIONS+="nowatch"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger
```
### 2.5 File descriptor limits
```bash
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
EOF
```
---
## Phase 3: Install Toolchain
### 3.1 Docker
```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in
docker --version && docker compose version
```
### 3.2 mise (manages all tools from `.tool-versions`)
```bash
curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc
```
### 3.3 Install project tools
The repo's [.tool-versions](.tool-versions) pins:
| Tool | Version |
| --- | --- |
| golang | 1.25.4 |
| buf | 1.28.1 |
| bun | 1.3.2 |
| protoc | 29.3 |
| protoc-gen-go | 1.28.1 |
| protoc-gen-go-grpc | 1.6.1 |
| protoc-gen-connect-go | 1.18.1 |
| golangci-lint | 2.8.0 |
| terraform | 1.5.7 |
| packer | 1.13.1 |
| python | 3.13.11 |
| gcloud | 534.0.0 |
```bash
cd ~/e2b-infra # after cloning
mise install # installs everything from .tool-versions
```
> **Note:** `gcloud` is only needed for `gsutil` to download prebuilt artifacts. If you don't have GCP credentials, you can download them via HTTPS instead (see Phase 5).
---
## Phase 4: Clone Repository
```bash
git clone https://github.com/e2b-dev/infra.git ~/e2b-infra
cd ~/e2b-infra
go work sync
```
---
## Phase 5: Download Prebuilt Artifacts
Firecracker binaries and Linux kernels must be downloaded from the public GCS bucket.
**With gsutil (if gcloud is configured):**
```bash
make download-public-kernels
make download-public-firecrackers
```
**Without GCP credentials (HTTPS alternative):**
The bucket `e2b-prod-public-builds` is publicly accessible. You'll need to browse/list it to find available versions, then download manually:
```bash
# Install gsutil standalone (no GCP project needed for public buckets)
# OR use curl/wget against:
# https://storage.googleapis.com/e2b-prod-public-builds/
# to discover and download kernel and firecracker builds
```
The simplest path is to install just the `gcloud` CLI (via mise) and run the make targets -- no GCP project or auth is needed for public bucket reads.
---
## Phase 6: Prepare Local Environment
### 6.1 Start infrastructure containers
```bash
make local-infra
# Starts: PostgreSQL 17.4, Redis 7.4.2, ClickHouse 25.4.5.24,
# Grafana 12.0.0, Loki 3.4.1, Tempo 2.8.2, Mimir 2.17.1,
# OTEL Collector 0.146.0, Vector, Memcached 1.6.38
```
Wait for all containers to be healthy:
```bash
docker compose -f packages/local-dev/docker-compose.yaml ps
```
### 6.2 Initialize databases
```bash
make -C packages/db migrate-local # PostgreSQL
make -C packages/clickhouse migrate-local # ClickHouse
```
### 6.3 Build envd (in-VM daemon)
```bash
make -C packages/envd build
```
### 6.4 Seed database with dev credentials
```bash
make -C packages/local-dev seed-database
```
Creates test user, team, API key, and access token for local development.
---
## Phase 7: Run the Dev Stack
Each service runs in the foreground. Use **tmux**, **screen**, or separate SSH sessions.
| Terminal | Command | Listens on |
| --- | --- | --- |
| 1 | `make local-infra` | (Docker containers) |
| 2 | `make -C packages/api run-local` | `:3000` |
| 3 | `make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local` | `:5008` |
| 4 | `make -C packages/client-proxy run-local` | `:3002` |
> The orchestrator **requires sudo** -- Firecracker needs root for `/dev/kvm`, network namespaces, veth pairs, nftables rules, and NBD devices.
---
## Phase 8: Verify
### 8.1 Health checks
```bash
curl -s http://localhost:3000/health # API
curl -s -o /dev/null -w "%{http_code}" http://localhost:53000 # Grafana (expect 302)
curl -s 'http://localhost:8123/?query=SELECT%201' # ClickHouse
redis-cli -h localhost -p 6379 ping # Redis
```
### 8.2 Build the base template
```bash
make -C packages/shared/scripts local-build-base-template
```
### 8.3 Test with E2B client SDK
```bash
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
export E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
export E2B_API_URL=http://localhost:3000
export E2B_SANDBOX_URL=http://localhost:3002
# Use E2B SDK/CLI to create a sandbox
```
---
## Phase 9: Access from Host (Optional)
### SSH port forwarding
```bash
ssh -N \
-L 3000:localhost:3000 \
-L 3002:localhost:3002 \
-L 53000:localhost:53000 \
-L 5432:localhost:5432 \
-L 8123:localhost:8123 \
user@<vm-ip>
```
### Or use bridged networking
If the VM has a routable IP on your LAN, services are directly accessible (Docker binds to `0.0.0.0`, Go services listen on all interfaces).
---
## Service Endpoints Reference
| Service | URL |
| --- | --- |
| E2B API | `http://localhost:3000` |
| E2B Client Proxy | `http://localhost:3002` |
| E2B Orchestrator | `http://localhost:5008` |
| Grafana | `http://localhost:53000` |
| PostgreSQL | `postgres://postgres:postgres@localhost:5432` |
| ClickHouse (HTTP) | `http://localhost:8123` |
| ClickHouse (native) | `localhost:9000` |
| Redis | `localhost:6379` |
| OTEL Collector (gRPC) | `localhost:4317` |
| OTEL Collector (HTTP) | `localhost:4318` |
| Loki | `http://localhost:3100` |
| Vector | `localhost:30006` |
---
## Troubleshooting
| Problem | Fix |
| --- | --- |
| `/dev/kvm` missing in guest | Enable nested virt on host (Phase 1.1), use `host-passthrough` CPU |
| `modprobe nbd` fails | Check kernel has NBD support: `modinfo nbd` |
| Orchestrator permission errors | Must run with `sudo` |
| Huge pages < 2048 | Not enough contiguous memory; increase VM RAM or set earlier in boot |
| Docker containers won't start | Check `systemctl status docker`, port conflicts with `ss -tlnp` |
| `gsutil` not found | Install via `mise install gcloud` or download artifacts via HTTPS |
---
## After VM Reboot Checklist
```bash
# 1. Verify kernel modules and sysctl (should be persistent)
lsmod | grep nbd
cat /proc/sys/vm/nr_hugepages # expect 2048
# 2. Start infra
cd ~/e2b-infra && make local-infra
# 3. Start services (separate terminals)
make -C packages/api run-local
sudo make -C packages/orchestrator run-local
make -C packages/client-proxy run-local
```

95
experiments/e2b/installation.md Normal file
View File

@@ -0,0 +1,95 @@
## dev-vm on beelink
```bash
virt-install \
--name e2b-dev \
--ram 16384 \
--vcpus 8 \
--cpu host-passthrough \
--os-variant ubuntu24.04 \
--disk path=/srv/vms/e2b-dev.qcow2,size=100,format=qcow2,bus=virtio \
--network bridge=br0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--location /srv/vms/isos/ubuntu-24.04.3-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
--extra-args 'console=ttyS0,115200n8'
# base packages
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
build-essential git curl wget unzip jq make gcc pkg-config \
iptables iproute2 net-tools ca-certificates gnupg \
lsb-release software-properties-common gettext-base
# kernel modules
# Load now
sudo modprobe nbd nbds_max=64
sudo modprobe kvm
sudo modprobe kvm_amd # or kvm_amd
sudo modprobe tun
sudo modprobe veth
sudo modprobe nf_tables
sudo modprobe nft_nat
# Persist across reboots
cat <<'EOF' | sudo tee /etc/modules-load.d/e2b.conf
nbd
kvm
kvm_amd
tun
veth
nf_tables
nft_nat
EOF
echo "options nbd nbds_max=64" | sudo tee /etc/modprobe.d/nbd.conf
# sysctl
cat <<'EOF' | sudo tee /etc/sysctl.d/99-e2b.conf
vm.nr_hugepages=2048
vm.max_map_count=1048576
vm.swappiness=10
vm.vfs_cache_pressure=50
net.ipv4.ip_forward=1
net.core.somaxconn=65535
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
EOF
sudo sysctl --system
# udev rules
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-e2b-nbd.rules
KERNEL=="nbd*", OPTIONS+="nowatch"
EOF
sudo udevadm control --reload-rules && sudo udevadm trigger
# file descriptor limits
cat <<'EOF' | sudo tee /etc/security/limits.d/e2b.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
EOF
```
## install toolchain
```bash
# docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in
docker --version && docker compose version
# mise
curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc
source ~/.bashrc
```
```

280
experiments/e2b/sanbox-usage-generated-from-infra-repo.md Normal file
View File

@@ -0,0 +1,280 @@
# E2B Sandbox Usage Guide
A practical guide for creating, observing, and using E2B sandboxes via the API.
## Prerequisites
### Local Dev Setup
Seed the database to create a test team and API key:
```bash
make -C packages/local-dev seed-database
```
### Environment Variables
```bash
E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
E2B_API_URL=http://localhost:3000
E2B_SANDBOX_URL=http://localhost:3002
```
All examples below use `$E2B_API_URL` and `$E2B_API_KEY` — export them in your shell for convenience:
```bash
export E2B_API_URL=http://localhost:3000
export E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
```
---
## 1. Create a Sandbox
```bash
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300
}'
```
**Response:**
```json
{
"sandboxID": "i1234567890abcdef",
"templateID": "base",
"envdVersion": "0.5.8",
"domain": "i1234567890abcdef.your-domain"
}
```
Save the `sandboxID` and `domain` from the response for subsequent commands.
### Optional Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `secure` | bool | Returns an `envdAccessToken` for authenticated envd access |
| `envVars` | object | Inject environment variables into the VM |
| `metadata` | object | Tag the sandbox for filtering (e.g. `{"purpose": "test"}`) |
| `autoPause` | bool | Pause instead of kill on timeout |
| `allow_internet_access` | bool | Allow internet egress from the VM |
| `network.allowPublicTraffic` | bool | Allow inbound public traffic |
| `network.allowOut` | array | Allowed egress destinations (IPs, CIDRs, domains) |
| `network.denyOut` | array | Denied egress destinations (IPs/CIDRs only) |
| `volumeMounts` | array | Mount persistent volumes (`{"name": "vol", "path": "/data"}`) |
**Example with options:**
```bash
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300,
"secure": true,
"envVars": {"MY_VAR": "hello"},
"metadata": {"purpose": "demo"}
}'
```
---
## 2. Observe Sandbox State
### Get details of a specific sandbox
```bash
curl $E2B_API_URL/sandboxes/{sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
```
Returns state (`running`/`paused`), CPU/memory/disk config, network settings, and TTL (`endAt`).
### List all sandboxes
```bash
# Running only
curl "$E2B_API_URL/v2/sandboxes?state=running" \
-H "X-API-Key: $E2B_API_KEY"
# Running and paused
curl "$E2B_API_URL/v2/sandboxes?state=running&state=paused" \
-H "X-API-Key: $E2B_API_KEY"
# Filter by metadata
curl "$E2B_API_URL/v2/sandboxes?metadata=purpose%3Ddemo" \
-H "X-API-Key: $E2B_API_KEY"
```
### Get resource metrics (CPU, memory, disk)
```bash
curl "$E2B_API_URL/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
-H "X-API-Key: $E2B_API_KEY"
```
**Response:**
```json
[
{
"timestampUnix": 1234567890,
"cpuCount": 2,
"cpuUsedPct": 25.5,
"memUsed": 268435456,
"memTotal": 536870912,
"diskUsed": 1073741824,
"diskTotal": 5368709120
}
]
```
### Get sandbox logs
```bash
curl "$E2B_API_URL/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
-H "X-API-Key: $E2B_API_KEY"
```
Optional query parameters: `cursor` (ms timestamp), `level` (min log level), `search` (substring match).
---
## 3. Use the Sandbox
The in-VM daemon (**envd**) runs on port 49983 inside each sandbox, exposed via the sandbox's `domain`.
### Upload a file
```bash
curl -X POST https://{domain}/files \
-H "Content-Type: multipart/form-data" \
-F "file=@script.py" \
-F "path=/home/user/script.py"
```
### Using the E2B Python SDK
Point the SDK at your local API:
```python
from e2b import Sandbox
sbx = Sandbox("base", api_url="http://localhost:3000")
# Run a command
result = sbx.commands.run("echo 'Hello from Firecracker VM!'")
print(result.stdout)
# Write and execute a file
sbx.files.write("/home/user/hello.py", "print('Hello world')")
result = sbx.commands.run("python3 /home/user/hello.py")
print(result.stdout)
# List files
files = sbx.files.list("/home/user")
for f in files:
print(f.name)
sbx.kill()
```
### Envd Connect RPC API
The envd daemon exposes Connect RPC services for programmatic access:
**Process Service:**
- `Start(ProcessConfig)` — start a new process
- `List()` — list running processes
- `Connect(ProcessSelector)` — connect to process stdio
- `Signal(ProcessSelector, Signal)` — send signal to process
**Filesystem Service:**
- `ListDir(Path)` — list directory contents
- `Stat(Path)` — get file metadata
- `WatchDir(Path)` — watch for changes
- `Move(Source, Dest)` — move/rename file
- `RemoveDir(Path)` — remove directory
---
## 4. Lifecycle Management
### Extend timeout
```bash
# Add 60 seconds to the TTL
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/refreshes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"duration": 60}'
# Or set an absolute timeout (seconds from now)
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/timeout \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"timeout": 120}'
```
### Pause (snapshot to disk)
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/pause \
-H "X-API-Key: $E2B_API_KEY"
```
### Resume / reconnect
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/connect \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"timeout": 300}'
```
Returns `200` if already running, `201` if resumed from paused state.
### Kill
```bash
curl -X DELETE $E2B_API_URL/sandboxes/{sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
```
### Create a snapshot (template from running sandbox)
```bash
curl -X POST $E2B_API_URL/sandboxes/{sandboxID}/snapshots \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "my-snapshot"}'
```
---
## 5. Network Configuration
Update network rules on a running sandbox:
```bash
curl -X PUT $E2B_API_URL/sandboxes/{sandboxID}/network \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"allowOut": ["8.8.8.8", "example.com"],
"denyOut": ["0.0.0.0/0"]
}'
```
**Rules:**
- If `allowOut` contains domain names, `denyOut` must include `0.0.0.0/0`
- Domain names are not supported in `denyOut` (IPs/CIDRs only)
- `allowOut` entries take precedence over `denyOut`
- Omitting both fields clears all rules

46
experiments/e2b/using-e2b-infra.md Normal file
View File

@@ -0,0 +1,46 @@
## VMs
```bash
E2B_API_URL=http://192.168.0.61:3000
# create vm
curl -X POST $E2B_API_URL/sandboxes \
-H "X-API-Key: $E2B_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"templateID": "base",
"timeout": 300
}'
# response
{"alias":"base","clientID":"6532622b","domain":null,"envdVersion":"0.5.8","sandboxID":"in3h60s6h0ie3kigrcls4","templateID":"pz1l1owhmy0w84e12eqv","trafficAccessToken":null}
sandboxID=in3h60s6h0ie3kigrcls4
# create sandboxVM and capture id
sandboxID=$(curl -s -X POST http://localhost:3000/sandboxes -H "X-API-Key: $E2B_API_KEY" -H "Content-Type: application/json" -d '{
"templateID": "base",
"timeout": 300
}' | jq -r .sandboxID
)
# get state
curl http://localhost:3000/sandboxes/${sandboxID} \
-H "X-API-Key: $E2B_API_KEY"
# list running sandboxes
curl "http://localhost:3000/v2/sandboxes?state=running" \
-H "X-API-Key: $E2B_API_KEY"
# sandbox resource metrics
curl "http://localhost:3000/sandboxes/{sandboxID}/metrics?start=$(date -v-5M +%s)&end=$(date +%s)" \
-H "X-API-Key: $E2B_API_KEY"
# sandbox logs
curl "http://localhost:3000/v2/sandboxes/{sandboxID}/logs?limit=100&direction=backward" \
-H "X-API-Key: $E2B_API_KEY"
```

22
experiments/kata_and_fc_on_kube/manifests/kata-deploy-values.yaml Normal file
View File

@@ -0,0 +1,22 @@
# Disable the default all-in-one installation
shims:
disableAll: true
firecracker:
enabled: true
# In this chart version, 'env' must be a map, not a list.
# The template converts these keys into Environment Variables for you.
env:
kataArtifacts: "firecracker"
multiInstallSuffix: "" # This fixes the specific error you saw
installArtifactsOnly: false
# Automatically create the RuntimeClass
runtimeClass:
create: true
name: kata-fc
handler: kata-fc
# Set Firecracker as the default for amd64 nodes
defaultShim:
amd64: firecracker

14
experiments/kata_and_fc_on_kube/manifests/runtime-class_kata-fc.yaml Normal file
View File

@@ -0,0 +1,14 @@
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata-fc
handler: kata-fc
scheduling:
nodeSelector:
# Ensure these pods only land on your specialized node
runtime: kata-fc
tolerations:
- key: "dedicated"
operator: "Equal"
value: "kata-fc"
effect: "NoSchedule"

198
experiments/kata_and_fc_on_kube/readme.md Normal file
View File

@@ -0,0 +1,198 @@
## Node: kube-node-34
```bash
apt install cpu-checker
# check kvm availability
kvm-ok
cat /sys/module/kvm_amd/parameters/nested
lsmod | grep kvm
# install kata containers
# using kata-deploy ?
export VERSION=$(curl -sSL https://api.github.com/repos/kata-containers/kata-containers/releases/latest | jq .tag_name | tr -d '"')
export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy"
helm upgrade --install kata-deploy "${CHART}" --version "${VERSION}" --values experiments/kata_and_fc_on_kube/manifests/kata-deploy-values.yaml
# label only specific nodes to be used with kata-fc
kubectl label node kube-node-34 runtime=kata-fc
# taint node with NoSchedule
kubectl taint nodes kube-node-34 dedicated=kata-fc:NoSchedule
# configure runtime class
k apply -f experiments/kata_and_fc_on_kube/manifests//runtime-class_kata-fc.yaml
# devmapper configuration for firecracker in containerd
ctr plugins ls | grep devmapper
# create data and metadata files
# might want to replace this with LVM !!!!
# sudo mkdir -p /var/lib/containerd/devmapper
# sudo truncate -s 10G /var/lib/containerd/devmapper/data
# sudo truncate -s 1G /var/lib/containerd/devmapper/init_metadata
# # Associate the files with Loop Devices
# sudo losetup /dev/loop10 /var/lib/containerd/devmapper/data
# sudo losetup /dev/loop11 /var/lib/containerd/devmapper/init_metadata
# # Use dmsetup to create the pool:
# # This command creates the mapping.
# # The numbers '0 20971520' represent the size in 512-byte sectors (for a 10GB file).
# sudo dmsetup create containerd-pool --table "0 20971520 thin-pool /dev/loop11 /dev/loop10 128 32768 1"
lvcreate -L 10G -T kata-vg/kata-pool
# Add the configuration to /etc/containerd/config.toml:
# Find the [plugins."io.containerd.snapshotter.v1.devmapper"] section and update it:
[plugins."io.containerd.snapshotter.v1.devmapper"]
# LVM uses a specific naming convention in /dev/mapper/
# It is VolumeGroupName-LogicalVolumeName
pool_name = "kata--vg-kata--pool"
root_path = "/var/lib/containerd/devmapper"
base_image_size = "10GB"
discard_blocks = true
# restart containerd
systemctl restart containerd
```
### kube node preparation & deployment
```bash
# ---------------------------------------------------------------------------
# Packages
# ---------------------------------------------------------------------------
apt-get update
apt-get install -y \
qemu-guest-agent \
openssh-server \
apt-transport-https \
ca-certificates \
curl \
gnupg \
nvme-cli
systemctl enable --now qemu-guest-agent
systemctl enable --now ssh
# ---------------------------------------------------------------------------
# nvme-tcp
# ---------------------------------------------------------------------------
apt-get install -y linux-modules-extra-$(uname -r)
modprobe nvme-tcp
echo "nvme-tcp" >> /etc/modules-load.d/nvme-tcp.conf
# ---------------------------------------------------------------------------
# Kernel modules for Kubernetes
# ---------------------------------------------------------------------------
cat > /etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
# ---------------------------------------------------------------------------
# containerd
# ---------------------------------------------------------------------------
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update && apt-get install -y containerd.io
cat > /etc/containerd/config.toml <<'EOF'
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
EOF
# Registry mirrors pointing to Zot at 192.168.0.30:5000
mkdir -p \
/etc/containerd/certs.d/docker.io \
/etc/containerd/certs.d/registry.k8s.io \
/etc/containerd/certs.d/ghcr.io \
/etc/containerd/certs.d/quay.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'EOF'
server = "https://registry-1.docker.io"
[host."http://192.168.0.30:5000/v2/docker.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml <<'EOF'
server = "https://registry.k8s.io"
[host."http://192.168.0.30:5000/v2/registry.k8s.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/ghcr.io/hosts.toml <<'EOF'
server = "https://ghcr.io"
[host."http://192.168.0.30:5000/v2/ghcr.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/quay.io/hosts.toml <<'EOF'
server = "https://quay.io"
[host."http://192.168.0.30:5000/v2/quay.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
systemctl restart containerd
# ---------------------------------------------------------------------------
# kubelet systemd drop-in
# ---------------------------------------------------------------------------
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/systemd/system/kubelet.service.d/10-containerd.conf <<'EOF'
[Unit]
After=containerd.service
Requires=containerd.service
[Service]
ExecStartPre=/bin/bash -c 'until [ -S /var/run/containerd/containerd.sock ]; do sleep 1; done'
ExecStartPre=/usr/bin/crictl info
EOF
# ---------------------------------------------------------------------------
# kubeadm / kubelet / kubectl v1.32
# ---------------------------------------------------------------------------
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" \
> /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
# join kube cluster
JOIN_COMMAND="kubeadm join 192.168.0.31:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>"

14
experiments/microsandbox/install.md Normal file
View File

@@ -0,0 +1,14 @@
```bash
virt-install \
--name microsandbox \
--ram 8192\
--vcpus 4 \
--cpu host-passthrough \
--os-variant ubuntu24.04 \
--disk path=/srv/vms/microsandbox.qcow2,size=100,format=qcow2,bus=virtio \
--network bridge=br0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--location /srv/vms/isos/ubuntu-24.04.3-live-server-amd64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
--extra-args 'console=ttyS0,115200n8'
```

27
experiments/orb-stack/modules/base-template/cloud-init-base.yaml Normal file
View File

@@ -0,0 +1,27 @@
#cloud-config
hostname: ${hostname}
manage_etc_hosts: true
# Kernel modules for container networking
write_files:
- path: /etc/modules-load.d/k8s.conf
content: |
overlay
br_netfilter
- path: /etc/sysctl.d/k8s.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
packages:
- curl
- gpg
- apt-transport-https
runcmd:
- modprobe overlay
- modprobe br_netfilter
- sysctl --system
- swapoff -a
- sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

74
experiments/orb-stack/modules/base-template/main.tf Normal file
View File

@@ -0,0 +1,74 @@
terraform {
required_providers {
orbstack = {
source = "robertdebock/orbstack"
version = "~> 3.0"
}
cloudinit = {
source = "hashicorp/cloudinit"
version = "~> 2.0"
}
}
}
variable "node_count" {
description = "Number of nodes to deploy"
type = number
default = 3
}
variable "name" {
description = "Base name for the machines"
type = string
}
variable "distro" {
description = "OS distribution"
type = string
default = "ubuntu"
}
variable "distro_version" {
description = "OS distribution version/codename"
type = string
default = "noble"
}
variable "extra_cloud_init_parts" {
description = "Additional cloud-init parts to layer on top of the base config"
type = list(object({ content = string, content_type = string }))
default = []
}
data "cloudinit_config" "this" {
count = var.node_count
part {
content_type = "text/cloud-config"
content = templatefile("${path.module}/cloud-init-base.yaml", {
hostname = "${var.name}-${count.index + 1}"
})
}
dynamic "part" {
for_each = var.extra_cloud_init_parts
content {
content_type = part.value.content_type
content = part.value.content
}
}
}
resource "orbstack_machine" "this" {
count = var.node_count
name = "${var.name}-${count.index + 1}"
distro = var.distro
region = var.distro_version
user_data = data.cloudinit_config.this[count.index].rendered
}
output "machines" {
value = { for m in orbstack_machine.this : m.name => m.ip_address }
}

11
experiments/orb-stack/stacks/k3s/cloud-init-cp.yaml Normal file
View File

@@ -0,0 +1,11 @@
#cloud-config
merge_how:
- name: list
settings: [append]
- name: dict
settings: [recurse_array]
runcmd:
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" K3S_TOKEN="${k3s_token}" sh -
- until kubectl get nodes; do sleep 2; done
- cp /etc/rancher/k3s/k3s.yaml /root/kubeconfig.yaml

9
experiments/orb-stack/stacks/k3s/cloud-init-worker.yaml Normal file
View File

@@ -0,0 +1,9 @@
#cloud-config
merge_how:
- name: list
settings: [append]
- name: dict
settings: [recurse_array]
runcmd:
- curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" K3S_URL="https://${cp_ip}:6443" K3S_TOKEN="${join_token}" sh -

39
experiments/orb-stack/stacks/k3s/main.tf Normal file
View File

@@ -0,0 +1,39 @@
variable "k3s_token" {
description = "Shared secret for k3s cluster join (set via TF_VAR or tfvars)"
type = string
sensitive = true
}
module "control_plane" {
source = "../../modules/base-template"
name = "k3s-cp"
node_count = 1
extra_cloud_init_parts = [{
content_type = "text/cloud-config"
content = templatefile("${path.module}/cloud-init-cp.yaml", {
k3s_token = var.k3s_token
})
}]
}
module "workers" {
source = "../../modules/base-template"
name = "k3s-worker"
node_count = 2
extra_cloud_init_parts = [{
content_type = "text/cloud-config"
content = templatefile("${path.module}/cloud-init-worker.yaml", {
cp_ip = values(module.control_plane.machines)[0]
join_token = var.k3s_token
})
}]
}
output "cluster" {
value = {
control_plane = module.control_plane.machines
workers = module.workers.machines
}
}

219
experiments/rke2/rke2-basics.md Normal file
View File

@@ -0,0 +1,219 @@
## install
```bash
# master node
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.32.12+rke2r1 sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service
journalctl -u rke2-server -f
# open firewalld
sudo firewall-cmd --permanent --add-port=9345/tcp
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp # Kubelet
sudo firewall-cmd --reload
# install nerdctl
# Set the version
VERSION="2.2.1" # Check GitHub for the latest version
# Download the tarball
wget https://github.com/containerd/nerdctl/releases/download/v${VERSION}/nerdctl-${VERSION}-linux-arm64.tar.gz
# Extract to your path
sudo tar -C /usr/local/bin -xzvf nerdctl-${VERSION}-linux-arm64.tar.gz nerdctl
# configure nerdctl
sudo mkdir -p /etc/nerdctl
sudo tee /etc/nerdctl/nerdctl.toml <<EOF
address = "unix:///run/k3s/containerd/containerd.sock"
namespace = "k8s.io"
EOF
# install buildkit
# Set current stable version
BK_VER="0.28.0"
# Download arm64 binary
wget https://github.com/moby/buildkit/releases/download/v${BK_VER}/buildkit-v${BK_VER}.linux-arm64.tar.gz
# Extract only the binaries to /usr/local/bin
sudo tar -C /usr/local/bin -xzvf buildkit-v${BK_VER}.linux-arm64.tar.gz --strip-components=1 bin/
# Create the service file
sudo tee /etc/systemd/system/buildkit.service <<EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit
[Service]
ExecStart=/usr/local/bin/buildkitd --addr unix:///run/buildkit/buildkitd.sock
[Install]
WantedBy=multi-user.target
EOF
# Reload and Start
sudo systemctl daemon-reload
sudo systemctl enable --now buildkit
# ---------------------------------------------
# agent/worker node
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_VERSION=v1.32.12+rke2r1 sh -
systemctl enable rke2-agent.service
mkdir -p /etc/rancher/rke2/
# token from master node
# cat /var/lib/rancher/rke2/server/node-token
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
server: https://192.168.64.3:9345
token: K107618960f87b9efb3a3255ce00a9743d29f1db9376820c9144cb85fa3c554dc69::server:06b2effdf0c9ce3952efc8a5d80bf084
EOF
systemctl start rke2-agent.service
journalctl -u rke2-agent -f
# Set up kubectl on the server node
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
echo 'export PATH=$PATH:/var/lib/rancher/rke2/bin' >> ~/.bashrc
source ~/.bashrc
```
## build and deploy application
```bash
# build container with nerdctl
nerdctl --namespace k8s.io build --tag hello-world:latest .
# export image as tar on master node
nerdctl save hello-world:latest -o hello-world.tar
# copy it over to worker node
scp hello-world.tar novakj@192.168.64.4:~/
# import image on the agent node
sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock -n k8s.io images import hello-world.tar
kubectl create namespace rke2-apps
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-deployment
namespace: rke2-apps
labels:
type: staticwebapp
spec:
replicas: 1
selector:
matchLabels:
type: staticwebapp
template:
metadata:
labels:
type: staticwebapp
spec:
containers:
- name: staticwebapp
image: hello-world:latest
imagePullPolicy: Never
ports:
- containerPort: 80
resources:
requests:
memory: "32Mi"
cpu: "200m"
limits:
memory: "64Mi"
cpu: "300m"
EOF
kubectl create -f deployment.yaml
# expose deployment
kubectl expose deployment hello-world-deployment --name hello-world-service --port=8080 --target-port=80 -n rke2-apps
# install ingress-nginx (even though i thought that there is ingress controller already deployed)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.2/deploy/static/provider/cloud/deploy.yaml
# create ingress with "localhost" as host
kubectl create ingress hello-world-ingress --class=nginx --rule="test-host/*=hello-world-service:8080" -n rke2-apps
kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 8081:80
# incomplete completion configuration ;-)
dnf install bash-completion -y
alias 'k=kubectl'
# ~/.bashrc
# 1. Load the main bash-completion package first
# On Rocky/RHEL, it's usually at this path:
[[ -r "/usr/share/bash-completion/bash_completion" ]] && . "/usr/share/bash-completion/bash_completion"
# Enable kubectl bash completion
source <(kubectl completion bash)
# Set up the alias
alias k=kubectl
# Link the kubectl completion logic to the 'k' alias
complete -o default -F __start_kubectl k
```
## upgrading RKE2
```bash
# install upgrade controller
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.9.1/system-upgrade-controller.yaml
# server upgrade
cat <<EOF | kubectl apply -f -
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: rke2-server-upgrade
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: In
values: ["true"]
serviceAccountName: system-upgrade
upgrade:
image: rancher/rke2-upgrade
version: v1.33.9+rke2r1
EOF
# agent upgrade
cat <<EOF | kubectl apply -f -
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: rke2-agent-upgrade
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
prepare:
# Logic: "Don't start workers until servers are done"
args: ["wait-for-plan", "rke2-server-upgrade"]
image: rancher/rke2-upgrade
serviceAccountName: system-upgrade
upgrade:
image: rancher/rke2-upgrade
version: v1.33.9+rke2r1
EOF
```

215
experiments/rke2/step-by-step.md Normal file
View File

@@ -0,0 +1,215 @@
## VMS creation
```bash
# hypervisor: beelink (192.168.0.6)
vms_path=/srv/vms/images
isos_path=/srv/vms/isos
cd $isos_path
# Grab Ubuntu 24.04 cloud image
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
# Create a base disk from the cloud image (repeat per node)
for NODE in rke2-server rke2-agent1 rke2-agent2; do
DISK_SIZE="30G"
[[ "$NODE" == rke2-agent* ]] && DISK_SIZE="50G"
qemu-img create -f qcow2 -F qcow2 -b $isos_path/noble-server-cloudimg-amd64.img $vms_path/${NODE}.qcow2
qemu-img resize $vms_path/${NODE}.qcow2 ${DISK_SIZE}
done
```
```bash
# prepare cloud-init/user-data
apt install cloud-image-utils
# prepare cloud-init and launch VMs
declare -A nodes=(
[rke2-server]="192.168.0.51"
[rke2-agent1]="192.168.0.52"
[rke2-agent2]="192.168.0.53"
)
for node in "${!nodes[@]}"; do
ip="${nodes[$node]}"
# user-data
cat <<EOF > user-data-${node}
#cloud-config
hostname: ${node}
manage_etc_hosts: false
users:
- name: sre
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQxxkqmvtVI+8c5BkTaJ5c2HfBFRXJWMmEcevvfP9tV jan.novak@Jans-MacBook-Air.local
write_files:
- path: /etc/hosts
append: true
content: |
192.168.0.51 rke2-server
192.168.0.52 rke2-agent1
192.168.0.53 rke2-agent2
- path: /etc/modules-load.d/rke2.conf
content: |
br_netfilter
overlay
- path: /etc/sysctl.d/99-rke2.conf
content: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
swap:
filename: /swap.img
size: 0
maxsize: 0
runcmd:
- swapoff -a
- sed -i '/swap/d' /etc/fstab
- modprobe br_netfilter
- modprobe overlay
- sysctl --system
package_update: true
packages:
- qemu-guest-agent
- nfs-common
- open-iscsi
power_state:
mode: reboot
EOF
# network config
cat <<EOF > network-config-${node}
network:
version: 2
ethernets:
eth0:
match:
driver: "virtio_net"
addresses:
- "${ip}/24"
nameservers:
addresses:
- 8.8.8.8
routes:
- to: "default"
via: "192.168.0.4"
EOF
cloud-localds --network-config=./network-config-${node} \
$vms_path/${node}-seed.iso ./user-data-${node}
done
# Launch VMs
for node in "${!nodes[@]}"; do
virt-install \
--name ${node} \
--ram 4096 --vcpus 2 \
--os-variant ubuntu24.04 \
--disk $vms_path/${node}.qcow2,bus=virtio \
--disk $vms_path/${node}-seed.iso,device=cdrom \
--network bridge=br0,model=virtio \
--graphics none \
--console pty,target_type=serial \
--noautoconsole \
--import
done
```
## RKE2 installation
```bash
# there are no .deb packages - only rpm or tarball
# "magic" install script can handle that
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.32 sudo sh -
# Create config directory
sudo mkdir -p /etc/rancher/rke2
# Server configuration
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
# Bind the API to the node's IP (not 127.0.0.1)
tls-san:
- rke2-server
- 192.168.0.51
# Write kubeconfig readable by non-root
write-kubeconfig-mode: "0644"
# CNI - canal is default and fine for home lab
# Alternatives: cilium, calico, multus
cni:
- canal
# Disable servicelb if you plan to use metallb
disable:
- rke2-service-lb
EOF
# Enable and start
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
# Watch bootstrap (takes 2-3 min on first run)
sudo journalctl -u rke2-server -f
```
## Grab the join token and kubeconfig
```bash
# Token for agents to join
sudo cat /var/lib/rancher/rke2/server/node-token
# Save this somewhere - you'll need it on every agent
# Set up kubectl on the server node
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
echo 'export PATH=$PATH:/var/lib/rancher/rke2/bin' >> ~/.bashrc
source ~/.bashrc
# Verify
kubectl get nodes
```
## Install RKE2 Agents (Workers)
```bash
# Install RKE2 agent - same channel as server
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=v1.32 INSTALL_RKE2_TYPE=agent sudo sh -
# Create config
sudo mkdir -p /etc/rancher/rke2
cat <<EOF | sudo tee /etc/rancher/rke2/config.yaml
server: https://192.168.0.51:9345
token: K10dba0bfff01d610ffed41c6b82a1b8861ee19e5af34a3bdd21970936823a846da::server:1f02fb36a288dcd06770cab28b015bd7
EOF
# Enable and start
sudo systemctl enable rke2-agent.service
sudo systemctl start rke2-agent.service
# Watch join
sudo journalctl -u rke2-agent -f
```
## Copy Kubeconfig to Your Workstation
```bash
# On your workstation (not the VMs)
scp sre@192.168.0.51:/etc/rancher/rke2/rke2.yaml ~/.kube/rke2-homelab.yaml
# Fix the server address (it'll say 127.0.0.1)
gsed -i 's/127.0.0.1/192.168.0.51/' ~/.kube/rke2-homelab.yaml
export KUBECONFIG=~/.kube/rke2-homelab.yaml
kubectl get nodes
```

14
kubernetes-kvm-terraform/files/auth-config.yaml Normal file
View File

@@ -0,0 +1,14 @@
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://idm.home.hrajfrisbee.cz/oauth2/openid/k8s
audiences:
- k8s
claimMappings:
username:
claim: preferred_username
prefix: ""
groups:
claim: groups
prefix: ""

155
kubernetes-kvm-terraform/files/join-node-02.sh Normal file
View File

@@ -0,0 +1,155 @@
#!/usr/bin/env bash
# Manual equivalent of the cloud-init user_data for kube-node-33 (node_02).
# Run as root on the target VM.
#
# Before running, generate a fresh join command on the control plane:
# kubeadm token create --print-join-command
# Then paste it below.
set -euo pipefail
JOIN_COMMAND="kubeadm join 192.168.0.31:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>"
# ---------------------------------------------------------------------------
# Hostname
# ---------------------------------------------------------------------------
hostnamectl set-hostname kube-node-33
# ---------------------------------------------------------------------------
# Packages
# ---------------------------------------------------------------------------
apt-get update
apt-get install -y \
qemu-guest-agent \
openssh-server \
apt-transport-https \
ca-certificates \
curl \
gnupg \
nvme-cli
systemctl enable --now qemu-guest-agent
systemctl enable --now ssh
# ---------------------------------------------------------------------------
# nvme-tcp
# ---------------------------------------------------------------------------
apt-get install -y linux-modules-extra-$(uname -r)
modprobe nvme-tcp
echo "nvme-tcp" >> /etc/modules-load.d/nvme-tcp.conf
# ---------------------------------------------------------------------------
# Kernel modules for Kubernetes
# ---------------------------------------------------------------------------
cat > /etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
# ---------------------------------------------------------------------------
# containerd
# ---------------------------------------------------------------------------
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update && apt-get install -y containerd.io
cat > /etc/containerd/config.toml <<'EOF'
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
EOF
# Registry mirrors pointing to Zot at 192.168.0.30:5000
mkdir -p \
/etc/containerd/certs.d/docker.io \
/etc/containerd/certs.d/registry.k8s.io \
/etc/containerd/certs.d/ghcr.io \
/etc/containerd/certs.d/quay.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'EOF'
server = "https://registry-1.docker.io"
[host."http://192.168.0.30:5000/v2/docker.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/registry.k8s.io/hosts.toml <<'EOF'
server = "https://registry.k8s.io"
[host."http://192.168.0.30:5000/v2/registry.k8s.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/ghcr.io/hosts.toml <<'EOF'
server = "https://ghcr.io"
[host."http://192.168.0.30:5000/v2/ghcr.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
cat > /etc/containerd/certs.d/quay.io/hosts.toml <<'EOF'
server = "https://quay.io"
[host."http://192.168.0.30:5000/v2/quay.io"]
capabilities = ["pull", "resolve"]
skip_verify = true
override_path = true
EOF
systemctl restart containerd
# ---------------------------------------------------------------------------
# kubelet systemd drop-in
# ---------------------------------------------------------------------------
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/systemd/system/kubelet.service.d/10-containerd.conf <<'EOF'
[Unit]
After=containerd.service
Requires=containerd.service
[Service]
ExecStartPre=/bin/bash -c 'until [ -S /var/run/containerd/containerd.sock ]; do sleep 1; done'
ExecStartPre=/usr/bin/crictl info
EOF
# ---------------------------------------------------------------------------
# kubeadm / kubelet / kubectl v1.32
# ---------------------------------------------------------------------------
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /" \
> /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
# ---------------------------------------------------------------------------
# kubectl shell helpers (available after next login / source)
# ---------------------------------------------------------------------------
cat > /etc/profile.d/kubectl.sh <<'EOF'
alias k='kubectl'
source <(kubectl completion bash)
complete -o default -F __start_kubectl k
EOF
# ---------------------------------------------------------------------------
# Join the cluster
# ---------------------------------------------------------------------------
$JOIN_COMMAND

132
kubernetes-kvm-terraform/files/manifests/kube-apiserver.yaml Normal file
View File

@@ -0,0 +1,132 @@
apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.31:6443
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.0.31
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
#- --oidc-issuer-url=https://idm.home.hrajfrisbee.cz/oauth2/openid/k8s
#- --oidc-client-id=k8s
#- --oidc-signing-algs=ES256
- --authentication-config=/etc/kubernetes/auth-config.yaml
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-issuer=https://kubernetes.default.svc.cluster.local
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: registry.k8s.io/kube-apiserver:v1.32.11
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 192.168.0.31
path: /livez
port: 6443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-apiserver
readinessProbe:
failureThreshold: 3
httpGet:
host: 192.168.0.31
path: /readyz
port: 6443
scheme: HTTPS
periodSeconds: 1
timeoutSeconds: 15
resources:
requests:
cpu: 250m
startupProbe:
failureThreshold: 24
httpGet:
host: 192.168.0.31
path: /livez
port: 6443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes
name: k8s-config
readOnly: true
hostNetwork: true
priority: 2000001000
priorityClassName: system-node-critical
securityContext:
seccompProfile:
type: RuntimeDefault
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/ca-certificates
type: DirectoryOrCreate
name: etc-ca-certificates
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
- hostPath:
path: /usr/local/share/ca-certificates
type: DirectoryOrCreate
name: usr-local-share-ca-certificates
- hostPath:
path: /usr/share/ca-certificates
type: DirectoryOrCreate
name: usr-share-ca-certificates
- hostPath:
path: /etc/kubernetes
type: DirectoryOrCreate
name: k8s-config
status: {}

24
kubernetes-kvm-terraform/master.tf
View File

@@ -115,6 +115,23 @@ locals {
skip_verify = true
override_path = true
- path: /etc/kubernetes/auth-config.yaml
content: |
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://idm.home.hrajfrisbee.cz/oauth2/openid/k8s
audiences:
- k8s
claimMappings:
username:
claim: preferred_username
prefix: ""
groups:
claim: groups
prefix: ""
- path: /root/kubeadm-config.yaml
content: |
apiVersion: kubeadm.k8s.io/v1beta3
@@ -130,6 +147,13 @@ locals {
oidc-signing-algs: "ES256"
networking:
podSubnet: "10.244.0.0/16"
# /etc/kubernetes/auth-config.yaml has to be created on master node somehow
extraVolumes:
- name: auth-config
hostPath: /etc/kubernetes/auth-config.yaml
mountPath: /etc/kubernetes/auth-config.yaml
readOnly: true
pathType: File
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

49
servers/saint/export-vms.md Normal file
View File

@@ -0,0 +1,49 @@
## export VMs
```bash
# vms need to get exported so that i can do something with outdated hypervisor
vm=utility-101
virsh dumpxml $vm > $vm-saint.xml
virsh domblklist $vm
# /srv/vms-normal-hdd/utility-101.qcow2
tar -cvzf ${vm}-saint-backup.tar.gz ${vm}-saint.xml /srv/vms-normal-hdd/utility-101.qcow2
vm=storage
h=saint
virsh shutdown $vm
virsh dumpxml $vm > $vm-$h.xml
virsh domblklist $vm
# /srv/vms-normal-hdd/storage.qcow2
tar -cvzf ${vm}-${h}-backup.tar.gz ${vm}-${h}.xml /srv/vms-normal-hdd/storage.qcow2
# ----
vm=docker
h=saint
virsh shutdown $vm
virsh dumpxml $vm > $vm-$h.xml
virsh domblklist $vm
# /srv/vms-ssd/docker.qcow2
# /srv/vms-normal-hdd/docker-srv.qcow2
# /srv/vms-normal-hdd/docker-vgroot-2.qcow2
tar -cvzf ${vm}-${h}-backup.tar.gz ${vm}-${h}.xml /srv/vms-ssd/docker.qcow2 /srv/vms-normal-hdd/docker-srv.qcow2 /srv/vms-normal-hdd/docker-vgroot-2.qcow2
# ----
vm=demon
h=agent
virsh shutdown $vm
virsh dumpxml $vm > $vm-$h.xml
virsh domblklist $vm
# /srv/nfs/data/vms-moved-from-failing-disk/demon_1.img
# /srv/nfs/data/vms-moved-from-failing-disk/demon_2nd_disk.img
tar -cvzf ${vm}-${h}-backup.tar.gz ${vm}-${h}.xml /srv/nfs/data/vms-moved-from-failing-disk/demon_1.img /srv/nfs/data/vms-moved-from-failing-disk/demon_2nd_disk.img
```

378
servers/saint/files/nginx/104_docker_zlutazimnice.cz.conf Normal file
View File

@@ -0,0 +1,378 @@
server {
server_name
api.psmf.hrajfrisbee.cz
;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.122.25:5001;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/api.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/api.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
# if ($host = api-sync.psmf.hrajfrisbee.cz) {
# return 301 https://$host$request_uri;
# } # managed by Certbot
listen 80;
client_max_body_size 15m; proxy_read_timeout 600;
server_name
api-sync.psmf.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.122.25:8003;
}
}
server {
server_name
api-sync.psmf.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.122.25:8003;
}
client_max_body_size 15m;
proxy_read_timeout 600;
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/api-sync.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
server_name api.evidence.cald.cz
api.evidence.czechultimate.cz
;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://192.168.122.25:8001;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/api.evidence.czechultimate.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/api.evidence.czechultimate.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
server_name evidence.cald.cz
evidence.czechultimate.cz
;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://192.168.122.25:8002;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/evidence.czechultimate.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/evidence.czechultimate.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
server_name
hrajfrisbee.cz
www.hrajfrisbee.cz;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.122.25:5002;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
if ($host = beta.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name beta.hrajfrisbee.cz;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.122.25:5003;
}
}
server {
server_name
hrajfrisbee.cz
www.hrajfrisbee.cz;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.122.25:5002;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/www.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
listen 80;
server_name beta.hrajfrisbee.cz;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.122.25:5003;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
server_name
gitlab.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.122.25:80;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
server_name
registry.gitlab.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.122.25:5005;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
client_max_body_size 3000M;
client_body_buffer_size 200000k;
#auth_basic "Restricted Content";
#auth_basic_user_file /etc/nginx/.htpasswd;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
if ($host = hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = www.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name
hrajfrisbee.cz
www.hrajfrisbee.cz;
return 404; # managed by Certbot
}
server {
if ($host = hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name
hrajfrisbee.cz
www.hrajfrisbee.cz;
return 404; # managed by Certbot
}
server {
if ($host = gitlab.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name
gitlab.hrajfrisbee.cz
;
listen 80;
return 404; # managed by Certbot
}
server {
if ($host = evidence.cald.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name evidence.cald.cz;
return 404; # managed by Certbot
}
server {
if ($host = evidence.czechultimate.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name evidence.czechultimate.cz;
return 404; # managed by Certbot
}
server {
if ($host = api.evidence.cald.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name api.evidence.cald.cz;
return 404; # managed by Certbot
}
server {
if ($host = api.evidence.czechultimate.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name api.evidence.czechultimate.cz;
return 404; # managed by Certbot
}
server {
if ($host = api.psmf.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name api.psmf.hrajfrisbee.cz;
return 404; # managed by Certbot
}
#server {
# if ($host = api-sync.psmf.hrajfrisbee.cz) {
# return 301 https://$host$request_uri;
# } # managed by Certbot
#
#
# listen 80;
# server_name
# api-sync.psmf.hrajfrisbee.cz
# ;
# return 404; # managed by Certbot
#}
server {
# if ($host = api-sync.psmf.hrajfrisbee.cz) {
# return 301 https://$host$request_uri;
# } # managed by Certbot
server_name
api-sync.psmf.hrajfrisbee.cz
;
listen 80;
return 404; # managed by Certbot
}

111
servers/saint/files/utility-101-saint.xml Normal file
View File

@@ -0,0 +1,111 @@
<domain type='kvm'>
<name>utility-101</name>
<uuid>82f853d8-9997-4b6d-96b0-629482f2c61e</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact' check='partial'>
<model fallback='allow'>Haswell-noTSX</model>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/srv/vms-normal-hdd/utility-101.qcow2'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:cd:83:a4'/>
<source network='default'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:cd:66:dc'/>
<source network='network-4'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
<image compression='off'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='1'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>

0
shadow/iptables/rules.v4 → servers/shadow/iptables/rules.v4
View File

0
shadow/iptables/rules.v4.backup → servers/shadow/iptables/rules.v4.backup
View File

136
servers/shadow/nginx/21_docker_hrajfrisbee.cz.conf Normal file
View File

@@ -0,0 +1,136 @@
server {
server_name
api.psmf.hrajfrisbee.cz
;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://192.168.123.21:5001;
}
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 8443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/api.psmf.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/api.psmf.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = api.psmf.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name api.psmf.hrajfrisbee.cz;
return 404; # managed by Certbot
}
# ----------------
# api-sync.psmf.hrajfrisbee.cz
server {
# if ($host = api-sync.psmf.hrajfrisbee.cz) {
# return 301 https://$host$request_uri;
# } # managed by Certbot
listen 80;
client_max_body_size 15m; proxy_read_timeout 600;
server_name
api-sync.psmf.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.123.21:8003;
}
}
server {
if ($host = api.psmf.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name
api.psmf.hrajfrisbee.cz
;
listen 80;
return 404; # managed by Certbot
}
# ----------------
# gitlab & friends
# ----------------
server {
server_name
gitlab.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.123.21:80;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
}
#listen 443 ssl; # managed by Certbot
#ssl_certificate /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
#ssl_certificate_key /etc/letsencrypt/live/gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
#include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
#ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
server {
server_name
registry.gitlab.hrajfrisbee.cz
;
location / {
proxy_pass http://192.168.123.21:5005;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
client_max_body_size 3000M;
client_body_buffer_size 200000k;
#auth_basic "Restricted Content";
#auth_basic_user_file /etc/nginx/.htpasswd;
}
#listen 443 ssl; # managed by Certbot
#ssl_certificate /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
#ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.hrajfrisbee.cz/privkey.pem; # managed by Certbot
#include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
#ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}

28
shadow/nginx-sites-enabled-default → servers/shadow/nginx/nginx-sites-enabled-default
View File

@@ -360,4 +360,32 @@ server {
listen 80 ;
server_name maru-hleda-byt.home.hrajfrisbee.cz;
return 404; # managed by Certbot
}
server {
server_name fuj-management.home.hrajfrisbee.cz; # managed by Certbot
location / {
proxy_pass http://docker-30:8081;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
listen 8443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/fuj-management.home.hrajfrisbee.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/fuj-management.home.hrajfrisbee.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = fuj-management.home.hrajfrisbee.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80 ;
server_name fuj-management.home.hrajfrisbee.cz;
return 404; # managed by Certbot
}

2
shadow/nginx.conf → servers/shadow/nginx/nginx.conf
View File

@@ -65,6 +65,8 @@ stream {
map $ssl_preread_server_name $backend {
# Passthrough to K8s
ghost.lab.home.hrajfrisbee.cz k8s_gatewayapi;
fujarna.lab.home.hrajfrisbee.cz k8s_gatewayapi;
operator-test.lab.home.hrajfrisbee.cz k8s_gatewayapi;
~^.+\.lab\.home\.hrajfrisbee\.cz$ k8s_ingress;
lab\.home\.hrajfrisbee\.cz$ k8s_ingress;

13
servers/shadow/storage-23/readme.md Normal file
View File

@@ -0,0 +1,13 @@
```bash
# install postgres + create users
apt install postgresql postgresql-contrib -y
# su - postgres
# psql
# create database psmfapi owner psmf;
# create users from other database with pg_dumpall --globals-only
# install redis
apt install redis-server -y
```

31
vms/docker/install.md
View File

@@ -1,3 +1,32 @@
## expected services
- tailscaled
- tailscaled
- node-exporter
```bash
# tailscale installation
# add repo and public keys
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/$(lsb_release -cs).tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
# install
sudo apt update
sudo apt install tailscale -y
# enable & start
systemctl enable tailscaled
systemctl start tailscaled
# authenticate
tailscale up
```
### deploy node-exporter
```bash
mkdir -p /srv/docker/node-exporter
cp docker-compose.yaml /srv/docker/node-exporter/
cp node-exporter.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now node-exporter
```