kodekloud-engineer/aws-49.md

Task 49

The Nautilus DevOps team needs to build a secure and scalable log aggregation setup within their AWS environment. The goal is to gather log files from an internal EC2 instance running in a private VPC, transfer them securely to another EC2 instance in a public VPC, and then push those logs to a secure S3 bucket.

1. A VPC named nautilus-priv-vpc already exists with a private subnet named nautilus-priv-subnet, a route table named nautilus-priv-rt, and an EC2 instance named nautilus-priv-ec2 (using ubuntu image). This instance uses the SSH key pair nautilus-key.pem already available on the AWS client host at /root/.ssh/.

2. Your task is to:

Create a new VPC named nautilus-pub-vpc. Create a subnet named nautilus-pub-subnet and a route table named nautilus-pub-rt under this public VPC. Attach an internet gateway to nautilus-pub-vpc and configure the public route table to enable internet access. Launch an EC2 instance named nautilus-pub-ec2 into the public subnet using the same key pair as the private instance. Create an IAM role named nautilus-s3-role with PutObject permission to an S3 bucket and attach it to the public EC2 instance. Create a new private S3 bucket named nautilus-s3-logs-29442. Configure a VPC Peering named nautilus-vpc-peering between the private and public VPCs. Modify both nautilus-priv-rt and nautilus-pub-rt to route each other's CIDR blocks through the peering connection. On the private instance, configure a cron job to push the /var/log/boots.log file to the public instance (using scp or rsync). On the public instance, configure a cron job to push that same file to the created S3 bucket. The uploaded file must be stored in the S3 bucket under the path nautilus-priv-vpc/boot/boots.log.

Solution

Log Aggregation Pipeline Task (devops) — KodeKloud-Constrained, jump-host + aws-cli-validated

Full cross-VPC log pipeline: devops-priv-ec2 (private VPC, no public IP / no internet) → scp over VPC peering → devops-pub-ec2 (public VPC) → aws s3 cp via IAM role → private S3 bucket.

Because the private instance has no public IP, all setup access to it goes through devops-pub-ec2 as a jump host over peering:

aws-client → (public IP) → devops-pub-ec2 → (private IP via peering) → devops-priv-ec2

The actual pipeline hops are independent of the jump: private→public scp runs over peering; public→S3 runs via the instance role.

KodeKloud limits applied

EC2 t2.micro + Standard credits · IAM managed policy (AmazonS3FullAccess, no custom) · check-for-pre-created-role · us-east-1 · S3 private by default.

---

Phase 1 — Discover the private side

REGION=us-east-1

PRIV_VPC=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=devops-priv-vpc \
  --region $REGION --query 'Vpcs[0].VpcId' --output text)
PRIV_CIDR=$(aws ec2 describe-vpcs --vpc-ids $PRIV_VPC --region $REGION \
  --query 'Vpcs[0].CidrBlock' --output text)
PRIV_RT=$(aws ec2 describe-route-tables --filters "Name=tag:Name,Values=devops-priv-rt" \
  --region $REGION --query 'RouteTables[0].RouteTableId' --output text)
PRIV_IID=$(aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=devops-priv-ec2" "Name=instance-state-name,Values=running,stopped" \
  --region $REGION --query 'Reservations[0].Instances[0].InstanceId' --output text)
PRIV_EC2_IP=$(aws ec2 describe-instances --instance-ids $PRIV_IID --region $REGION \
  --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
PRIV_SG=$(aws ec2 describe-instances --instance-ids $PRIV_IID --region $REGION \
  --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupId' --output text)

echo "PRIV_VPC=$PRIV_VPC CIDR=$PRIV_CIDR RT=$PRIV_RT instance=$PRIV_IID privIP=$PRIV_EC2_IP SG=$PRIV_SG"

Phase 2 — Build the public VPC (non-overlapping CIDR)

PUB_CIDR=10.1.0.0/16
[ "$PRIV_CIDR" = "10.1.0.0/16" ] && PUB_CIDR=10.2.0.0/16

PUB_VPC=$(aws ec2 create-vpc --cidr-block $PUB_CIDR --region $REGION \
  --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=devops-pub-vpc}]' \
  --query 'Vpc.VpcId' --output text)

PUB_SUBNET=$(aws ec2 create-subnet --vpc-id $PUB_VPC --cidr-block 10.1.1.0/24 --region $REGION \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=devops-pub-subnet}]' \
  --query 'Subnet.SubnetId' --output text)
aws ec2 modify-subnet-attribute --subnet-id $PUB_SUBNET --map-public-ip-on-launch --region $REGION

IGW=$(aws ec2 create-internet-gateway --region $REGION \
  --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=devops-pub-igw}]' \
  --query 'InternetGateway.InternetGatewayId' --output text)
aws ec2 attach-internet-gateway --internet-gateway-id $IGW --vpc-id $PUB_VPC --region $REGION

PUB_RT=$(aws ec2 create-route-table --vpc-id $PUB_VPC --region $REGION \
  --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=devops-pub-rt}]' \
  --query 'RouteTable.RouteTableId' --output text)
aws ec2 create-route --route-table-id $PUB_RT --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW --region $REGION
aws ec2 associate-route-table --route-table-id $PUB_RT --subnet-id $PUB_SUBNET --region $REGION

Adjust the subnet block to fit your chosen $PUB_CIDR.

Phase 3 — Private S3 bucket

BUCKET=devops-s3-logs-3826
aws s3api create-bucket --bucket $BUCKET --region $REGION
# Private by default (BPA on).

Phase 4 — IAM role devops-s3-role (managed policy, KK-safe) + instance profile

EXISTING=$(aws iam get-role --role-name devops-s3-role --query 'Role.Arn' --output text 2>/dev/null || echo MISSING)
if [ "$EXISTING" = "MISSING" ]; then
  cat > /tmp/ec2-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF
  aws iam create-role --role-name devops-s3-role --assume-role-policy-document file:///tmp/ec2-trust.json
fi
aws iam attach-role-policy --role-name devops-s3-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam create-instance-profile --instance-profile-name devops-s3-role 2>/dev/null || true
aws iam add-role-to-instance-profile --instance-profile-name devops-s3-role --role-name devops-s3-role 2>/dev/null || true

KK: task wants "PutObject" — AmazonS3FullAccess (AWS-managed, allowed) includes it. Custom scoped s3:PutObject policy is usually blocked by CreatePolicy; use it only if your lab permits and the grader demands scoped least-privilege.

Phase 5 — Launch the public EC2 (t2.micro, same key, instance profile, jump-capable SG)

PUB_SG=$(aws ec2 create-security-group --group-name devops-pub-sg \
  --description "public jump + log-agg" --vpc-id $PUB_VPC --region $REGION --query 'GroupId' --output text)
# SSH from anywhere so this box can serve as your jump host
aws ec2 authorize-security-group-ingress --group-id $PUB_SG --protocol tcp --port 22 \
  --cidr 0.0.0.0/0 --region $REGION

sleep 8
PUB_IID=$(aws ec2 run-instances \
  --image-id resolve:ssm:/aws/service/canonical/ubuntu/server/noble/stable/current/amd64/hvm/ebs-gp3/ami-id \
  --instance-type t2.micro --key-name devops-key --subnet-id $PUB_SUBNET \
  --security-group-ids $PUB_SG --iam-instance-profile Name=devops-s3-role --region $REGION \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=devops-pub-ec2}]' \
  --query 'Instances[0].InstanceId' --output text)

aws ec2 wait instance-running --instance-ids $PUB_IID --region $REGION
PUB_EC2_PUBIP=$(aws ec2 describe-instances --instance-ids $PUB_IID --region $REGION \
  --query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
PUB_EC2_PRIVIP=$(aws ec2 describe-instances --instance-ids $PUB_IID --region $REGION \
  --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
echo "public: pub=$PUB_EC2_PUBIP priv=$PUB_EC2_PRIVIP"

• --key-name devops-key — same pair as the private instance (.pem is devops-key.pem).
• --iam-instance-profile Name=devops-s3-role — role at launch for the S3 push.
• t2.micro, Standard credits — KK compliant.

Phase 6 — Peering + routes both directions

PCX=$(aws ec2 create-vpc-peering-connection --vpc-id $PUB_VPC --peer-vpc-id $PRIV_VPC --region $REGION \
  --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=devops-vpc-peering}]' \
  --query 'VpcPeeringConnection.VpcPeeringConnectionId' --output text)
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id $PCX --region $REGION

aws ec2 create-route --route-table-id $PUB_RT --destination-cidr-block $PRIV_CIDR \
  --vpc-peering-connection-id $PCX --region $REGION
aws ec2 create-route --route-table-id $PRIV_RT --destination-cidr-block $PUB_CIDR \
  --vpc-peering-connection-id $PCX --region $REGION

Routes on BOTH tables — non-negotiable.

Phase 6.5 — Open private SG for the jump + define the ProxyCommand

# Private instance must accept SSH from the public VPC CIDR (jump traffic over peering)
aws ec2 authorize-security-group-ingress --group-id $PRIV_SG \
  --protocol tcp --port 22 --cidr $PUB_CIDR --region $REGION 2>/dev/null || echo "already allowed"

KEY=/root/.ssh/devops-key.pem
JUMP="ssh -i $KEY -o StrictHostKeyChecking=no -W %h:%p ubuntu@$PUB_EC2_PUBIP"

Phase 7 — Private instance (via jump): place key + install scp cron

# 7a: copy the key onto the private instance THROUGH the jump
scp -i $KEY -o StrictHostKeyChecking=no -o ProxyCommand="$JUMP" \
  $KEY ubuntu@$PRIV_EC2_IP:/home/ubuntu/.ssh/devops-key.pem

# 7b: install the scp cron (targets the public instance's PRIVATE IP over peering)
ssh -i $KEY -o StrictHostKeyChecking=no -o ProxyCommand="$JUMP" ubuntu@$PRIV_EC2_IP bash -s << EOF
chmod 600 /home/ubuntu/.ssh/devops-key.pem
echo "* * * * * scp -i /home/ubuntu/.ssh/devops-key.pem -o StrictHostKeyChecking=no /var/log/boots.log ubuntu@$PUB_EC2_PRIVIP:/home/ubuntu/boots.log" | crontab -
echo "--- private crontab ---"; crontab -l
EOF

If /var/log/boots.log doesn't exist on the private instance, the pipeline moves nothing. Confirm it's there (the standard Ubuntu file is boot.log singular — the task uses boots.log, so it may need creating): ls -l /var/log/boots.log.

Phase 8 — Public instance: VALIDATE + install aws CLI, then the S3 cron

This is the step that bit us before — the cron uses aws s3 cp, so aws CLI must be present, and cron's minimal PATH must find it.

ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@$PUB_EC2_PUBIP bash -s << 'EOF'
# --- VALIDATE aws CLI; install only if missing ---
if command -v aws >/dev/null 2>&1; then
  echo "aws CLI already present: $(aws --version 2>&1)"
else
  echo "aws CLI missing — installing..."
  sudo apt-get update -y
  sudo apt-get install -y awscli
  # fallback if the apt package is unavailable on this Ubuntu release:
  if ! command -v aws >/dev/null 2>&1; then
    sudo apt-get install -y unzip curl
    curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip
    cd /tmp && unzip -q awscliv2.zip && sudo ./aws/install
  fi
fi

AWS_BIN=$(command -v aws)
echo "aws resolved at: $AWS_BIN"
EOF

Then install the cron using the full binary path + an explicit PATH line (cron's PATH is minimal — /usr/bin:/bin — and won't find a /usr/local/bin/aws from the v2 installer otherwise):

ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@$PUB_EC2_PUBIP bash -s << EOF
AWS_BIN=\$(command -v aws)
( echo "PATH=/usr/local/bin:/usr/bin:/bin"; \
  echo "* * * * * \$AWS_BIN s3 cp /home/ubuntu/boots.log s3://$BUCKET/devops-priv-vpc/boot/boots.log --region $REGION" ) | crontab -
echo "--- public crontab ---"; crontab -l
EOF

• No aws configure — role creds come from IMDS.
• S3 key path devops-priv-vpc/boot/boots.log set literally (S3 "folders" are prefixes).

Phase 8.5 — Prove both hops manually before waiting on cron

# Hop 1 delivered? (file present on public instance)
ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@$PUB_EC2_PUBIP \
  "ls -l /home/ubuntu/boots.log; echo exit=\$?"

# Hop 2 works? (manual upload, confirms aws CLI + role + path)
ssh -i $KEY -o StrictHostKeyChecking=no ubuntu@$PUB_EC2_PUBIP \
  "aws s3 cp /home/ubuntu/boots.log s3://$BUCKET/devops-priv-vpc/boot/boots.log --region $REGION; echo exit=\$?"

Both exit=0 means the crons will succeed too. If hop 1's ls shows no file, the private→public scp isn't working (check peering route on the private RT + the key on the private box).

Verify

aws ec2 describe-vpc-peering-connections --vpc-peering-connection-ids $PCX --region $REGION \
  --query 'VpcPeeringConnections[0].Status.Code'
aws ec2 describe-route-tables --route-table-ids $PUB_RT $PRIV_RT --region $REGION \
  --query 'RouteTables[].Routes[?VpcPeeringConnectionId!=null].DestinationCidrBlock'
aws ec2 describe-instances --instance-ids $PUB_IID --region $REGION \
  --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn'

for n in $(seq 1 6); do
  aws s3 ls s3://$BUCKET/devops-priv-vpc/boot/boots.log --region $REGION 2>/dev/null && { echo "SUCCESS"; break; }
  echo "waiting for cron pipeline... ($n)"; sleep 30
done

Want peering active, both peering routes present, the public instance showing devops-s3-role, and the object at s3://devops-s3-logs-3826/devops-priv-vpc/boot/boots.log.

Debug order

1. aws: command not found in public cron → aws CLI missing or not in cron's PATH. Phase 8's validate/install + the PATH= line + full $AWS_BIN path fix both. Verify with grep CRON /var/log/syslog | tail on the public box.
2. Hop 1 file absent on public instance → private RT missing $PUB_CIDR → pcx, public SG not allowing 22 from $PRIV_CIDR (it allows 0.0.0.0/0 here, so usually the route), or devops-key.pem not on the private box / wrong perms.
3. S3 cp AccessDenied → role not attached (IamInstanceProfile.Arn null) or managed policy not attached to the role.
4. /var/log/boots.log missing on private instance → nothing to move; create it.
5. Jump SSH fails → test hops separately: ssh -i $KEY ubuntu@$PUB_EC2_PUBIP must work first; then the ProxyCommand hop needs the private SG open to $PUB_CIDR + the peering routes.
6. Wrong user → Ubuntu image = ubuntu@ everywhere.

Skills used

kodekloud-aws-limits — drove t2.micro/Standard-credits, managed-policy IAM (AmazonS3FullAccess over a scoped custom policy since KK blocks CreatePolicy), check-for-pre-created-role, us-east-1, S3-private-by-default. The jump-host ProxyCommand and the aws-CLI validate/install are standard ops patterns, not skill-driven.