Task 46

The DevOps team is working on automating file management between two S3 buckets. The task is to create a public S3 bucket for file uploads and a private S3 bucket for securely storing the files. A Lambda function will be triggered automatically whenever a file is uploaded to the public S3 bucket, which will copy the file to the private bucket. Additionally, logs of the operation will be stored in a DynamoDB table. The logs should include details such as the source bucket, destination bucket, and the object key of the file that was copied. This will help the team maintain better security and visibility for file transfers.

Create a public S3 bucket named datacenter-public-8776. Ensure that the bucket allows public access to its objects. Create a private S3 bucket named datacenter-private-12032. Ensure that the bucket does not allow public access. Create a Lambda function named datacenter-copyfunction. This function should be triggered by uploads to the public S3 bucket and should copy the uploaded file to the private bucket. Create the necessary policies and a role named lambda_execution_role. Attach these policies to the role, and then link this role to the Lambda function. lambda-function.py is already present under the /root/ directory on AWS client host, replace REPLACE-WITH-YOUR-DYNAMODB-TABLE and REPLACE-WITH-YOUR-PRIVATE-BUCKET values. Create a DynamoDB table named datacenter-S3CopyLogs with a partition key LogID (string). This table will store logs generated by the Lambda function, including details such as source bucket name, destination bucket name, and object key. For testing upload the file sample.zip located in the /root directory on the client host to the public S3 bucket. The Lambda function should trigger and copy the file to the private bucket. Verify that the file has been successfully copied to the private bucket by checking the private bucket in the S3 console. Verify that a log entry has been created in the DynamoDB table containing the file copy details.

Solution

S3 → Lambda → S3 + DynamoDB Event-Driven Copy Task (datacenter)

An event-driven serverless pipeline — the most concept-dense task in the set. The crux is the S3 → Lambda trigger, which is a two-part wiring most people get half-right: (1) a resource-based permission on the Lambda (add-permission) letting the S3 service invoke it, AND (2) the event notification on the bucket pointing at the function. Miss either and uploads silently don't trigger anything. Plus the Lambda's role needs four distinct permission sets: read public S3, write private S3, write DynamoDB, write logs.

⚠️ Filename gotcha — read this first

The task says the code file is lambda-function.py (hyphen). Python module names cannot contain hyphens, so a handler of lambda-function.lambda_handler will fail at import with Runtime.ImportModuleError. The fix: rename to lambda_function.py (underscore) when zipping, and set the handler to lambda_function.lambda_handler. Handled below.

Run on aws-client.

Phase 1 — The two S3 buckets

REGION=us-east-1
PUB_BUCKET=datacenter-public-8776
PRIV_BUCKET=datacenter-private-12032

# --- Public bucket: allow public access ---
aws s3api create-bucket --bucket $PUB_BUCKET --region $REGION

aws s3api put-public-access-block --bucket $PUB_BUCKET \
  --public-access-block-configuration \
    "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false" \
  --region $REGION

cat > /tmp/pub-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "PublicRead",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::$PUB_BUCKET/*"
  }]
}
EOF
aws s3api put-bucket-policy --bucket $PUB_BUCKET \
  --policy file:///tmp/pub-policy.json --region $REGION

# --- Private bucket: keep the secure defaults (public access blocked) ---
aws s3api create-bucket --bucket $PRIV_BUCKET --region $REGION
# New buckets are private by default (BPA fully on) — nothing to do.

The public bucket needs BPA disabled + a public-read policy (same pattern as the static-website task). The private bucket just uses the default locked-down state — no action needed, which is the "does not allow public access" requirement.

Phase 2 — DynamoDB table

aws dynamodb create-table \
  --table-name datacenter-S3CopyLogs \
  --attribute-definitions AttributeName=LogID,AttributeType=S \
  --key-schema AttributeName=LogID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region $REGION

aws dynamodb wait table-exists --table-name datacenter-S3CopyLogs --region $REGION

Only LogID (string partition key) is declared — the source/dest/objectKey attributes are written per-item by the Lambda, schemaless.

Phase 3 — IAM role + policy (four permission sets)

REGION=us-east-1
PUB_BUCKET=datacenter-public-8776
PRIV_BUCKET=datacenter-private-12032

# --- STEP 1: Does the role already exist (lab-provided)? ---
EXISTING=$(aws iam get-role --role-name lambda_execution_role \
  --query 'Role.Arn' --output text 2>/dev/null || echo "MISSING")
echo "Role status: [$EXISTING]"

# --- STEP 2: If MISSING, create it (trust policy only — that's allowed) ---
if [ "$EXISTING" = "MISSING" ]; then
  cat > /tmp/lambda-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF
  aws iam create-role --role-name lambda_execution_role \
    --assume-role-policy-document file:///tmp/lambda-trust.json
fi

# --- STEP 3: See what's already attached (lab may have pre-attached perms) ---
echo "=== attached managed policies ==="
aws iam list-attached-role-policies --role-name lambda_execution_role
echo "=== inline policies ==="
aws iam list-role-policies --role-name lambda_execution_role

# --- STEP 4: Attach AWS-managed policies (permitted where custom creation isn't) ---
# S3 (read public + write private), DynamoDB (write logs), Logs (basic execution)
for ARN in \
  arn:aws:iam::aws:policy/AmazonS3FullAccess \
  arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess \
  arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole ; do
  aws iam attach-role-policy --role-name lambda_execution_role --policy-arn "$ARN" \
    2>/dev/null && echo "attached: $ARN" || echo "skip/denied: $ARN"
done

ROLE_ARN=$(aws iam get-role --role-name lambda_execution_role --query 'Role.Arn' --output text)
echo "ROLE_ARN=[$ROLE_ARN]"

Four permission sets, each essential: GetObject on the public bucket (read the upload), PutObject on the private bucket (write the copy), PutItem on the table (write the log), and AWSLambdaBasicExecutionRole for CloudWatch Logs (or the function can't even log). Note the S3 ARNs use /* (object-level) since Get/Put act on objects.

Phase 4 — Prep the code, zip (with the rename), deploy

# Replace placeholders; write out with UNDERSCORE filename (hyphen breaks Python import)
mkdir -p /tmp/fn
sed \
  -e "s/REPLACE-WITH-YOUR-DYNAMODB-TABLE/datacenter-S3CopyLogs/g" \
  -e "s/REPLACE-WITH-YOUR-PRIVATE-BUCKET/$PRIV_BUCKET/g" \
  /root/lambda-function.py > /tmp/fn/lambda_function.py

(cd /tmp/fn && zip -q function.zip lambda_function.py)

# Deploy with KodeKloud-compliant timeout (10s) and memory (256 MB)
sleep 10   # let role propagate
aws lambda create-function \
  --function-name datacenter-copyfunction \
  --runtime python3.13 \
  --role "$ROLE_ARN" \
  --handler lambda_function.lambda_handler \
  --zip-file fileb:///tmp/fn/function.zip \
  --timeout 10 \
  --memory-size 256 \
  --region $REGION

FUNC_ARN=$(aws lambda get-function --function-name datacenter-copyfunction \
  --region $REGION --query 'Configuration.FunctionArn' --output text)
echo "FUNC_ARN=[$FUNC_ARN]"

The sed handles step 4 — swaps REPLACE-WITH-YOUR-DYNAMODB-TABLE → datacenter-S3CopyLogs and REPLACE-WITH-YOUR-PRIVATE-BUCKET → the private bucket name, writing the result out as lambda_function.py (underscore) to dodge the hyphen import problem.
--handler lambda_function.lambda_handler matches the underscore filename + the function inside.
--timeout 30 — the default 3s can be too short for an S3 copy + DynamoDB write; bump it.
python3.13 — current supported runtime (3.14 also valid; the code was written for whatever the lab targets, 3.13 is safe).

Phase 5 — Wire the S3 trigger (the two-part gotcha)

# PART 1: Let the S3 service invoke this Lambda (resource-based permission)
aws lambda add-permission \
  --function-name datacenter-copyfunction \
  --statement-id s3-invoke-permission \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::$PUB_BUCKET \
  --region $REGION

# PART 2: Configure the public bucket to notify Lambda on object creation
cat > /tmp/notif.json << EOF
{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "$FUNC_ARN",
      "Events": ["s3:ObjectCreated:*"]
    }
  ]
}
EOF

aws s3api put-bucket-notification-configuration \
  --bucket $PUB_BUCKET \
  --notification-configuration file:///tmp/notif.json \
  --region $REGION

This ordering is mandatory. put-bucket-notification-configuration validates at config time that S3 can actually invoke the target Lambda — so the add-permission (Part 1) must exist first, or Part 2 fails with Unable to validate the following destination configurations. The permission is a resource-based policy on the Lambda; the notification is config on the bucket. Both halves = a working trigger; either alone = silent no-op.

Phase 6 — Test: upload sample.zip

aws s3 cp /root/sample.zip s3://$PUB_BUCKET/sample.zip --region $REGION
# Give the async trigger + copy + log-write a few seconds
sleep 10

Phase 7 & 8 — Verify copy + log entry

# File copied to private bucket?
echo "=== private bucket ==="
aws s3 ls s3://$PRIV_BUCKET/ --region $REGION

# Log entry in DynamoDB?
echo "=== DynamoDB logs ==="
aws dynamodb scan --table-name datacenter-S3CopyLogs --region $REGION \
  --query 'Items[].{LogID:LogID.S,Source:sourceBucket.S,Dest:destinationBucket.S,Key:objectKey.S}' \
  --output table

Want sample.zip present in the private bucket, and at least one log item in the table showing source (datacenter-public-8776), destination (datacenter-private-12032), and object key (sample.zip). Both appearing is the end-to-end proof: upload → S3 event → Lambda invoked → copied + logged.

Note: the exact DynamoDB attribute names (sourceBucket, destinationBucket, objectKey) depend on what the provided lambda-function.py writes — adjust the --query to match its actual keys if they differ. A plain aws dynamodb scan --table-name datacenter-S3CopyLogs with no query shows the raw item so you can see the real attribute names.

Debug order if nothing copies

Nothing in private bucket, nothing in DynamoDB → the trigger didn't fire. Check the two-part wiring: aws lambda get-policy --function-name datacenter-copyfunction should show the s3.amazonaws.com permission, and aws s3api get-bucket-notification-configuration --bucket $PUB_BUCKET should show the Lambda config. If either's missing, redo Phase 5 (in order).
Trigger fired but errored → read the Lambda's logs: aws logs tail /aws/lambda/datacenter-copyfunction --region $REGION. Common causes: AccessDenied (role missing a permission — check Phase 3), ImportModuleError (the hyphen filename problem — confirm the zip has lambda_function.py with underscore), or ResourceNotFoundException (placeholders not replaced — the private bucket / table name still says REPLACE-WITH-...).
Copy works but no log → DynamoDB PutItem permission missing from the role, or the table name placeholder wasn't substituted.
ImportModuleError: No module named 'lambda-function' → the classic — you zipped/handler'd with the hyphen. Re-zip as lambda_function.py, update handler, update-function-code.

The core concept

The whole task hinges on event-driven decoupling: S3 doesn't "call" Lambda directly — it emits an event that the Lambda service consumes, gated by a resource-based permission. This is the fundamental AWS serverless pattern (S3/SNS/SQS/EventBridge → Lambda all work this way). The two-part permission+notification wiring is the thing to internalize; it shows up on the DevOps Pro exam constantly.

12 KiB Raw Blame History