AWS Builder Center

Builder Center

I automated my lab data summaries with S3 and Lambda

I automated my lab data summaries with S3 and Lambda

In my mechatronics lab courses I end up with lots of CSV files: motor tests, sensor readings, temperature curves. I do the same thing with almost every file: open it, pull out minimums, maximums and averages, and copy them into the report.

Rodrigo Ruiz Orozco

Mecatrónica IPN | Python + IA | Aprendiendo AWS para proyectos de automatización
I wanted to see if AWS could handle that step for me. The idea is simple: I upload a CSV to an S3 bucket and a Lambda function automatically generates a JSON summary. No servers, no opening Excel.

Architecture

1
CSV → s3://my-bucket/raw/ → Lambda (Python) → s3://my-bucket/summary/ → JSON
  • S3 stores the files.
  • An S3 event triggers the Lambda every time a new file lands in raw/.
  • The Lambda reads the CSV, computes per-column statistics and saves the result to summary/.
Using two different prefixes matters: if the Lambda wrote to the same folder that triggers it, it would invoke itself in a loop.

Step 1: Create the bucket

In the S3 console I created a bucket (names are globally unique) with two folders: raw/ and summary/. I left Block Public Access enabled; nobody else needs to see this data.

Step 2: Create the Lambda function

  • Runtime: Python 3.12
  • Architecture: arm64 (slightly cheaper than x86)
  • Role: I let Lambda create a basic one and then added bucket permissions (Step 4)
Under Configuration → General, I raised the timeout from 3 to 30 seconds. For larger files, 3 seconds isn’t enough.

Step 3: The code

I used only the standard library (csv) to avoid setting up layers with pandas:
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
import csv
import io
import json
import urllib.parse

import boto3

s3 = boto3.client("s3")

def summarize_csv(text):
reader = csv.DictReader(io.StringIO(text))
values = {}
rows = 0

for row in reader:
rows += 1
for column, value in row.items():
try:
values.setdefault(column, []).append(float(value))
except (TypeError, ValueError):
pass # skip empty or text cells

columns = {}
for column, data in values.items():
if data:
columns[column] = {
"n": len(data),
"min": min(data),
"max": max(data),
"mean": round(sum(data) / len(data), 4),
}
return rows, columns

def lambda_handler(event, context):
outputs = []

for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])

obj = s3.get_object(Bucket=bucket, Key=key)
text = obj["Body"].read().decode("utf-8-sig")

rows, columns = summarize_csv(text)
summary = {"file": key, "rows": rows, "columns": columns}

name = key.split("/")[-1].rsplit(".", 1)[0]
out_key = f"summary/{name}.json"

s3.put_object(
Bucket=bucket,
Key=out_key,
Body=json.dumps(summary, indent=2),
ContentType="application/json",
)
outputs.append(out_key)

return {"statusCode": 200, "summaries": outputs}
Two details that took me a while:
  • unquote_plus: S3 sends the object key URL-encoded. If your file is named motor test.csv, it arrives as motor+test.csv and get_object fails.
  • utf-8-sig: CSVs exported from Excel include an invisible character (BOM) at the start. Without this, the first column shows up as \ufefftime_s.

Step 4: Permissions

I added a policy to the Lambda’s role scoped to this bucket only, instead of granting full S3 access:
json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-bucket/raw/*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-bucket/summary/*"
}
]
}
This way the function can only read from raw/ and write to summary/.

Step 5: Connect the event

In the bucket: Properties → Event notifications → Create event notification.
  • Event type: s3:ObjectCreated:*
  • Prefix: raw/
  • Suffix: .csv
  • Destination: the Lambda function

Test

I uploaded a CSV from a motor test:
1
2
3
4
5
time_s,current_A,temperature_C
0.0,0.12,24.8
0.5,1.85,25.1
1.0,2.03,25.9
...
A few seconds later, summary/motor_test.json appeared:
json
1
2
3
4
5
6
7
8
{
"file": "raw/motor_test.csv",
"rows": 240,
"columns": {
"current_A": { "n": 240, "min": 0.12, "max": 2.41, "mean": 1.9273 },
"temperature_C": { "n": 240, "min": 24.8, "max": 38.6, "mean": 31.402 }
}
}
[Add a screenshot of the bucket with the generated JSON and of the CloudWatch logs]

Errors I ran into

ErrorCauseFix
AccessDenied on GetObjectRole lacked bucket permissionsCheck the ARN in the policy
NoSuchKeyFile name with spaces not decodedurllib.parse.unquote_plus
Task timed out after 3.00 secondsDefault timeoutRaise it to 30 s
Lambda never runsEvent missing the right prefixCheck raw/ prefix and .csv suffix
For debugging, anything you print() shows up in CloudWatch Logs.

Cost

Lambda includes 1 million free requests per month, so for a few CSVs a week the cost is practically zero. Still, check your account’s terms and set up a budget alert before experimenting.

What’s next

I want to add an automatic PNG chart and then send the summary to Amazon Bedrock to draft a short interpretation of the results, which is the part of the report that takes me the longest.

# aws-lambda# amazon-s3# serverless# python# students
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article