128 lines
6.3 KiB
Markdown
128 lines
6.3 KiB
Markdown
## Task 50
|
|
|
|
The Nautilus DevOps Team has recently been informed by the Development Team that their EC2 instance is running out of storage space. This instance, crucial for development activities, is named xfusion-ec2 and currently has an attached volume of 8 GiB. To accommodate the increasing data requirements, the storage needs to be expanded to 12 GiB. This change should ensure that the expanded space is immediately available for use within the instance without disrupting ongoing activities.
|
|
|
|
Identify Volume: Find the volume attached to the xfusion-ec2 instance.
|
|
|
|
Expand Volume: Increase the volume size from 8 GiB to 12 GiB.
|
|
|
|
Reflect Changes: Ensure the root (/) partition within the instance reflects the expanded size from 8 GiB to 12 GiB.
|
|
|
|
SSH Access: Use the key pair located at /root/xfusion-keypair.pem on the aws-client host to SSH into the EC2 instance.
|
|
|
|
|
|
### Solution
|
|
|
|
# EBS Volume Expansion Task (xfusion-ec2) — 8 GiB → 12 GiB
|
|
|
|
EBS volume expansion — the concept that trips everyone: **growing the EBS volume in AWS is only half the job.** The block device gets bigger, but the OS doesn't know until you grow the *partition* and then the *filesystem* on top of it. Three layers, three separate operations: **EBS volume → partition table → filesystem.** Miss the last two and `df -h` still shows 8 GiB despite the volume being 12.
|
|
|
|
Run on `aws-client`.
|
|
|
|
## Phase 1 — Identify the volume + resize the EBS layer
|
|
|
|
```bash
|
|
REGION=us-east-1
|
|
|
|
# Instance + its root volume
|
|
IID=$(aws ec2 describe-instances \
|
|
--filters "Name=tag:Name,Values=xfusion-ec2" "Name=instance-state-name,Values=running,stopped" \
|
|
--region $REGION --query 'Reservations[0].Instances[0].InstanceId' --output text)
|
|
|
|
VOL_ID=$(aws ec2 describe-instances --instance-ids $IID --region $REGION \
|
|
--query 'Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId' --output text)
|
|
|
|
PUB_IP=$(aws ec2 describe-instances --instance-ids $IID --region $REGION \
|
|
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
|
|
|
|
echo "instance=$IID volume=$VOL_ID ip=$PUB_IP"
|
|
|
|
# Grow the EBS volume 8 -> 12 GiB (online, no detach, no stop)
|
|
aws ec2 modify-volume --volume-id $VOL_ID --size 12 --region $REGION
|
|
|
|
# Wait for the modification to finish optimizing
|
|
aws ec2 describe-volumes-modifications --volume-ids $VOL_ID --region $REGION \
|
|
--query 'VolumesModifications[0].{State:ModificationState,Progress:Progress,Size:TargetSize}'
|
|
```
|
|
|
|
Key points on the EBS layer:
|
|
|
|
- **`modify-volume` is online** — no detach, no stop, no disruption. Satisfies the task's "without disrupting ongoing activities." The volume grows live while the instance runs.
|
|
- **The modification goes through states** `modifying → optimizing → completed`. The **new size is usable as soon as it hits `optimizing`** (usually seconds) — you don't have to wait for `completed` (which can take a while as it re-lays-out storage in the background). Poll `describe-volumes-modifications` until `State` is `optimizing` or `completed`.
|
|
- **6-hour cooldown:** you can't modify the same volume again for 6 hours after a change. Not relevant here, but it exists.
|
|
|
|
## Phase 2 — SSH in and grow the partition + filesystem
|
|
|
|
SSH using the specified key:
|
|
|
|
```bash
|
|
chmod 400 /root/xfusion-keypair.pem
|
|
ssh -i /root/xfusion-keypair.pem -o StrictHostKeyChecking=no ubuntu@$PUB_IP
|
|
```
|
|
|
|
> User is `ubuntu` for an Ubuntu image; `ec2-user` for Amazon Linux. Check what the instance runs and swap if needed.
|
|
|
|
Once on the instance, inspect the mismatch:
|
|
|
|
```bash
|
|
lsblk # disk shows 12G, but its partition still 8G
|
|
df -h / # / still 8G
|
|
```
|
|
|
|
`lsblk` shows the disk (e.g. `nvme0n1` or `xvda`) at **12G** but its partition (`nvme0n1p1` / `xvda1`) still at **8G**. Two grows needed:
|
|
|
|
```bash
|
|
# --- Grow the PARTITION to fill the disk ---
|
|
# Nitro/newer instances: device is /dev/nvme0n1, partition 1
|
|
sudo growpart /dev/nvme0n1 1
|
|
# (older Xen instances: sudo growpart /dev/xvda 1)
|
|
|
|
# --- Grow the FILESYSTEM to fill the partition ---
|
|
# ext4 (most common):
|
|
sudo resize2fs /dev/nvme0n1p1
|
|
# XFS (some AMIs, incl. many Amazon Linux): sudo xfs_growfs /
|
|
```
|
|
|
|
Confirm:
|
|
|
|
```bash
|
|
df -h / # / should now show ~12G
|
|
```
|
|
|
|
The three layers explained (this is the whole task):
|
|
|
|
- **`growpart`** rewrites the partition table so partition 1 extends to the end of the now-larger disk. Note the **space between device and partition number** — it's `growpart /dev/nvme0n1 1`, not `growpart /dev/nvme0n1p1`. Common syntax error.
|
|
- **`resize2fs` (ext4) / `xfs_growfs` (xfs)** grows the filesystem to fill the enlarged partition. This is the step that actually makes `df` report the new size. **Pick the right tool for your filesystem** — run `df -T /` or `lsblk -f` to see the fstype. `resize2fs` on an XFS volume (or vice versa) just errors.
|
|
- Both are **online operations** — no unmount, no reboot. The `/` filesystem grows live.
|
|
|
|
## Verify
|
|
|
|
From the instance:
|
|
|
|
```bash
|
|
lsblk # disk, partition, and fs all ~12G
|
|
df -h / # / shows ~12G
|
|
```
|
|
|
|
From `aws-client`, confirm the EBS side settled:
|
|
|
|
```bash
|
|
aws ec2 describe-volumes --volume-ids $VOL_ID --region $REGION \
|
|
--query 'Volumes[0].{Size:Size,State:State}'
|
|
aws ec2 describe-volumes-modifications --volume-ids $VOL_ID --region $REGION \
|
|
--query 'VolumesModifications[0].ModificationState'
|
|
```
|
|
|
|
Want: EBS `Size: 12`, modification `optimizing`/`completed`, `lsblk` partition at 12G, and `df -h /` showing `/` at ~12G. That last one — `/` reflecting 12G — is the task's actual success condition (step 3).
|
|
|
|
## Debug order
|
|
|
|
1. **`df` still shows 8G after `modify-volume`** → you only did the EBS layer. Do `growpart` + `resize2fs`/`xfs_growfs` on the instance.
|
|
2. **`growpart: NOCHANGE` / "already the maximum"** → partition already grown (ran twice), or the EBS modify hasn't reached `optimizing` yet — check `describe-volumes-modifications`, wait, retry.
|
|
3. **`resize2fs: Bad magic number`** → wrong filesystem tool; it's XFS, use `xfs_growfs /`.
|
|
4. **Wrong device name** → `nvme0n1` on Nitro, `xvda` on older Xen. `lsblk` shows which; use the actual name.
|
|
5. **SSH refused** → wrong user (`ubuntu` vs `ec2-user`), key perms (`chmod 400`), or SG doesn't allow 22 from aws-client.
|
|
|
|
## KodeKloud note
|
|
|
|
Nothing here trips a KodeKloud limit — no new resource, no IAM, no instance-type change; you're just growing an existing volume. The `kodekloud-aws-limits` constraints don't apply to a pure resize op (region is already us-east-1). |