222 lines
12 KiB
Markdown
222 lines
12 KiB
Markdown
## Task 46
|
|
|
|
The DevOps team is working on automating file management between two S3 buckets. The task is to create a public S3 bucket for file uploads and a private S3 bucket for securely storing the files. A Lambda function will be triggered automatically whenever a file is uploaded to the public S3 bucket, which will copy the file to the private bucket. Additionally, logs of the operation will be stored in a DynamoDB table. The logs should include details such as the source bucket, destination bucket, and the object key of the file that was copied. This will help the team maintain better security and visibility for file transfers.
|
|
|
|
Create a public S3 bucket named datacenter-public-8776. Ensure that the bucket allows public access to its objects.
|
|
Create a private S3 bucket named datacenter-private-12032. Ensure that the bucket does not allow public access.
|
|
Create a Lambda function named datacenter-copyfunction. This function should be triggered by uploads to the public S3 bucket and should copy the uploaded file to the private bucket. Create the necessary policies and a role named lambda_execution_role. Attach these policies to the role, and then link this role to the Lambda function.
|
|
lambda-function.py is already present under the /root/ directory on AWS client host, replace REPLACE-WITH-YOUR-DYNAMODB-TABLE and REPLACE-WITH-YOUR-PRIVATE-BUCKET values.
|
|
Create a DynamoDB table named datacenter-S3CopyLogs with a partition key LogID (string). This table will store logs generated by the Lambda function, including details such as source bucket name, destination bucket name, and object key.
|
|
For testing upload the file sample.zip located in the /root directory on the client host to the public S3 bucket. The Lambda function should trigger and copy the file to the private bucket.
|
|
Verify that the file has been successfully copied to the private bucket by checking the private bucket in the S3 console.
|
|
Verify that a log entry has been created in the DynamoDB table containing the file copy details.
|
|
|
|
### Solution
|
|
|
|
# S3 → Lambda → S3 + DynamoDB Event-Driven Copy Task (datacenter)
|
|
|
|
An event-driven serverless pipeline — the most concept-dense task in the set. The crux is the **S3 → Lambda trigger**, which is a *two-part* wiring most people get half-right: (1) a **resource-based permission** on the Lambda (`add-permission`) letting the S3 service invoke it, AND (2) the **event notification** on the bucket pointing at the function. Miss either and uploads silently don't trigger anything. Plus the Lambda's role needs four distinct permission sets: read public S3, write private S3, write DynamoDB, write logs.
|
|
|
|
## ⚠️ Filename gotcha — read this first
|
|
|
|
The task says the code file is `lambda-function.py` (**hyphen**). Python module names **cannot contain hyphens**, so a handler of `lambda-function.lambda_handler` will fail at import with `Runtime.ImportModuleError`. The fix: **rename to `lambda_function.py` (underscore) when zipping**, and set the handler to `lambda_function.lambda_handler`. Handled below.
|
|
|
|
Run on `aws-client`.
|
|
|
|
## Phase 1 — The two S3 buckets
|
|
|
|
```bash
|
|
REGION=us-east-1
|
|
PUB_BUCKET=datacenter-public-8776
|
|
PRIV_BUCKET=datacenter-private-12032
|
|
|
|
# --- Public bucket: allow public access ---
|
|
aws s3api create-bucket --bucket $PUB_BUCKET --region $REGION
|
|
|
|
aws s3api put-public-access-block --bucket $PUB_BUCKET \
|
|
--public-access-block-configuration \
|
|
"BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false" \
|
|
--region $REGION
|
|
|
|
cat > /tmp/pub-policy.json << EOF
|
|
{
|
|
"Version": "2012-10-17",
|
|
"Statement": [{
|
|
"Sid": "PublicRead",
|
|
"Effect": "Allow",
|
|
"Principal": "*",
|
|
"Action": "s3:GetObject",
|
|
"Resource": "arn:aws:s3:::$PUB_BUCKET/*"
|
|
}]
|
|
}
|
|
EOF
|
|
aws s3api put-bucket-policy --bucket $PUB_BUCKET \
|
|
--policy file:///tmp/pub-policy.json --region $REGION
|
|
|
|
# --- Private bucket: keep the secure defaults (public access blocked) ---
|
|
aws s3api create-bucket --bucket $PRIV_BUCKET --region $REGION
|
|
# New buckets are private by default (BPA fully on) — nothing to do.
|
|
```
|
|
|
|
The public bucket needs BPA disabled + a public-read policy (same pattern as the static-website task). The private bucket just uses the default locked-down state — no action needed, which *is* the "does not allow public access" requirement.
|
|
|
|
## Phase 2 — DynamoDB table
|
|
|
|
```bash
|
|
aws dynamodb create-table \
|
|
--table-name datacenter-S3CopyLogs \
|
|
--attribute-definitions AttributeName=LogID,AttributeType=S \
|
|
--key-schema AttributeName=LogID,KeyType=HASH \
|
|
--billing-mode PAY_PER_REQUEST \
|
|
--region $REGION
|
|
|
|
aws dynamodb wait table-exists --table-name datacenter-S3CopyLogs --region $REGION
|
|
```
|
|
|
|
Only `LogID` (string partition key) is declared — the source/dest/objectKey attributes are written per-item by the Lambda, schemaless.
|
|
|
|
## Phase 3 — IAM role + policy (four permission sets)
|
|
|
|
```bash
|
|
REGION=us-east-1
|
|
PUB_BUCKET=datacenter-public-8776
|
|
PRIV_BUCKET=datacenter-private-12032
|
|
|
|
# --- STEP 1: Does the role already exist (lab-provided)? ---
|
|
EXISTING=$(aws iam get-role --role-name lambda_execution_role \
|
|
--query 'Role.Arn' --output text 2>/dev/null || echo "MISSING")
|
|
echo "Role status: [$EXISTING]"
|
|
|
|
# --- STEP 2: If MISSING, create it (trust policy only — that's allowed) ---
|
|
if [ "$EXISTING" = "MISSING" ]; then
|
|
cat > /tmp/lambda-trust.json << 'EOF'
|
|
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}
|
|
EOF
|
|
aws iam create-role --role-name lambda_execution_role \
|
|
--assume-role-policy-document file:///tmp/lambda-trust.json
|
|
fi
|
|
|
|
# --- STEP 3: See what's already attached (lab may have pre-attached perms) ---
|
|
echo "=== attached managed policies ==="
|
|
aws iam list-attached-role-policies --role-name lambda_execution_role
|
|
echo "=== inline policies ==="
|
|
aws iam list-role-policies --role-name lambda_execution_role
|
|
|
|
# --- STEP 4: Attach AWS-managed policies (permitted where custom creation isn't) ---
|
|
# S3 (read public + write private), DynamoDB (write logs), Logs (basic execution)
|
|
for ARN in \
|
|
arn:aws:iam::aws:policy/AmazonS3FullAccess \
|
|
arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess \
|
|
arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole ; do
|
|
aws iam attach-role-policy --role-name lambda_execution_role --policy-arn "$ARN" \
|
|
2>/dev/null && echo "attached: $ARN" || echo "skip/denied: $ARN"
|
|
done
|
|
|
|
ROLE_ARN=$(aws iam get-role --role-name lambda_execution_role --query 'Role.Arn' --output text)
|
|
echo "ROLE_ARN=[$ROLE_ARN]"
|
|
```
|
|
|
|
Four permission sets, each essential: **GetObject** on the public bucket (read the upload), **PutObject** on the private bucket (write the copy), **PutItem** on the table (write the log), and **AWSLambdaBasicExecutionRole** for CloudWatch Logs (or the function can't even log). Note the S3 ARNs use `/*` (object-level) since Get/Put act on objects.
|
|
|
|
## Phase 4 — Prep the code, zip (with the rename), deploy
|
|
|
|
```bash
|
|
# Replace placeholders; write out with UNDERSCORE filename (hyphen breaks Python import)
|
|
mkdir -p /tmp/fn
|
|
sed \
|
|
-e "s/REPLACE-WITH-YOUR-DYNAMODB-TABLE/datacenter-S3CopyLogs/g" \
|
|
-e "s/REPLACE-WITH-YOUR-PRIVATE-BUCKET/$PRIV_BUCKET/g" \
|
|
/root/lambda-function.py > /tmp/fn/lambda_function.py
|
|
|
|
(cd /tmp/fn && zip -q function.zip lambda_function.py)
|
|
|
|
# Deploy with KodeKloud-compliant timeout (10s) and memory (256 MB)
|
|
sleep 10 # let role propagate
|
|
aws lambda create-function \
|
|
--function-name datacenter-copyfunction \
|
|
--runtime python3.13 \
|
|
--role "$ROLE_ARN" \
|
|
--handler lambda_function.lambda_handler \
|
|
--zip-file fileb:///tmp/fn/function.zip \
|
|
--timeout 10 \
|
|
--memory-size 256 \
|
|
--region $REGION
|
|
|
|
FUNC_ARN=$(aws lambda get-function --function-name datacenter-copyfunction \
|
|
--region $REGION --query 'Configuration.FunctionArn' --output text)
|
|
echo "FUNC_ARN=[$FUNC_ARN]"
|
|
```
|
|
|
|
- **The `sed` handles step 4** — swaps `REPLACE-WITH-YOUR-DYNAMODB-TABLE` → `datacenter-S3CopyLogs` and `REPLACE-WITH-YOUR-PRIVATE-BUCKET` → the private bucket name, writing the result out as `lambda_function.py` (underscore) to dodge the hyphen import problem.
|
|
- **`--handler lambda_function.lambda_handler`** matches the underscore filename + the function inside.
|
|
- **`--timeout 30`** — the default 3s can be too short for an S3 copy + DynamoDB write; bump it.
|
|
- **`python3.13`** — current supported runtime (3.14 also valid; the code was written for whatever the lab targets, 3.13 is safe).
|
|
|
|
## Phase 5 — Wire the S3 trigger (the two-part gotcha)
|
|
|
|
```bash
|
|
# PART 1: Let the S3 service invoke this Lambda (resource-based permission)
|
|
aws lambda add-permission \
|
|
--function-name datacenter-copyfunction \
|
|
--statement-id s3-invoke-permission \
|
|
--action lambda:InvokeFunction \
|
|
--principal s3.amazonaws.com \
|
|
--source-arn arn:aws:s3:::$PUB_BUCKET \
|
|
--region $REGION
|
|
|
|
# PART 2: Configure the public bucket to notify Lambda on object creation
|
|
cat > /tmp/notif.json << EOF
|
|
{
|
|
"LambdaFunctionConfigurations": [
|
|
{
|
|
"LambdaFunctionArn": "$FUNC_ARN",
|
|
"Events": ["s3:ObjectCreated:*"]
|
|
}
|
|
]
|
|
}
|
|
EOF
|
|
|
|
aws s3api put-bucket-notification-configuration \
|
|
--bucket $PUB_BUCKET \
|
|
--notification-configuration file:///tmp/notif.json \
|
|
--region $REGION
|
|
```
|
|
|
|
**This ordering is mandatory.** `put-bucket-notification-configuration` *validates at config time* that S3 can actually invoke the target Lambda — so the `add-permission` (Part 1) must exist first, or Part 2 fails with `Unable to validate the following destination configurations`. The permission is a resource-based policy on the Lambda; the notification is config on the bucket. Both halves = a working trigger; either alone = silent no-op.
|
|
|
|
## Phase 6 — Test: upload sample.zip
|
|
|
|
```bash
|
|
aws s3 cp /root/sample.zip s3://$PUB_BUCKET/sample.zip --region $REGION
|
|
# Give the async trigger + copy + log-write a few seconds
|
|
sleep 10
|
|
```
|
|
|
|
## Phase 7 & 8 — Verify copy + log entry
|
|
|
|
```bash
|
|
# File copied to private bucket?
|
|
echo "=== private bucket ==="
|
|
aws s3 ls s3://$PRIV_BUCKET/ --region $REGION
|
|
|
|
# Log entry in DynamoDB?
|
|
echo "=== DynamoDB logs ==="
|
|
aws dynamodb scan --table-name datacenter-S3CopyLogs --region $REGION \
|
|
--query 'Items[].{LogID:LogID.S,Source:sourceBucket.S,Dest:destinationBucket.S,Key:objectKey.S}' \
|
|
--output table
|
|
```
|
|
|
|
Want `sample.zip` present in the private bucket, and at least one log item in the table showing source (`datacenter-public-8776`), destination (`datacenter-private-12032`), and object key (`sample.zip`). Both appearing is the end-to-end proof: upload → S3 event → Lambda invoked → copied + logged.
|
|
|
|
> Note: the exact DynamoDB attribute names (`sourceBucket`, `destinationBucket`, `objectKey`) depend on what the provided `lambda-function.py` writes — adjust the `--query` to match its actual keys if they differ. A plain `aws dynamodb scan --table-name datacenter-S3CopyLogs` with no query shows the raw item so you can see the real attribute names.
|
|
|
|
## Debug order if nothing copies
|
|
|
|
1. **Nothing in private bucket, nothing in DynamoDB** → the trigger didn't fire. Check the two-part wiring: `aws lambda get-policy --function-name datacenter-copyfunction` should show the s3.amazonaws.com permission, and `aws s3api get-bucket-notification-configuration --bucket $PUB_BUCKET` should show the Lambda config. If either's missing, redo Phase 5 (in order).
|
|
2. **Trigger fired but errored** → read the Lambda's logs: `aws logs tail /aws/lambda/datacenter-copyfunction --region $REGION`. Common causes: `AccessDenied` (role missing a permission — check Phase 3), `ImportModuleError` (the hyphen filename problem — confirm the zip has `lambda_function.py` with underscore), or `ResourceNotFoundException` (placeholders not replaced — the private bucket / table name still says `REPLACE-WITH-...`).
|
|
3. **Copy works but no log** → DynamoDB `PutItem` permission missing from the role, or the table name placeholder wasn't substituted.
|
|
4. **`ImportModuleError: No module named 'lambda-function'`** → the classic — you zipped/handler'd with the hyphen. Re-zip as `lambda_function.py`, update handler, `update-function-code`.
|
|
|
|
## The core concept
|
|
|
|
The whole task hinges on **event-driven decoupling**: S3 doesn't "call" Lambda directly — it emits an event that the Lambda service consumes, gated by a resource-based permission. This is the fundamental AWS serverless pattern (S3/SNS/SQS/EventBridge → Lambda all work this way). The two-part permission+notification wiring is the thing to internalize; it shows up on the DevOps Pro exam constantly. |