Everyone Wanted Spark. We Kept Apache Pig and Won Anyway.
How we kept 15-year-old Apache Pig scripts alive on AWS with EMR, cut costs by a third, and automated deploys with GitHub Actions and OIDC.
Every few months, someone asks why I'd reach for Apache Pig instead of Spark or dbt when a client needs batch ETL on AWS. The honest answer is that this particular client had eleven years of Pig Latin scripts, a data engineering team that could write and debug them blindfolded, and zero appetite for a rewrite. My job wasn't to modernize their stack for its own sake. My job was to get those scripts running reliably, cheaply, and observably on AWS, and to make deploying a change to one of them as boring as merging a pull request. This is the story of how that pipeline got built, the decisions that mattered, and the parts that didn't go the way the documentation implied they would.
The starting point
The client's existing setup was an on-prem Hadoop cluster running nightly Pig jobs against clickstream logs, producing aggregate tables that fed a handful of internal dashboards. The cluster was aging hardware, the ops burden had fallen on two people who had better things to do, and the business wanted the same jobs running with less manual babysitting. Nobody wanted to touch the fifteen
.pig scripts that already worked. That constraint shaped almost every decision that followed.Amazon EMR was the obvious landing spot because it ships Pig as a first-class application alongside Hadoop, Hive, and Spark, so the scripts didn't need to be touched to run on YARN instead of the old on-prem cluster. The rest of the architecture grew around that one fixed point.
The shape of the pipeline
At a high level: raw clickstream logs land in an S3 raw zone from an existing ingestion process I didn't need to touch, an EMR cluster running Pig reads and transforms that data, the output lands back in S3 as Parquet in a curated zone, and a Glue Data Catalog table sitting on top of that curated zone makes it queryable from Athena without anyone needing cluster access.
The diagram below is close to what actually shipped, minus a few subnet and security group details that don't add much to the story.
A few choices worth explaining. The EMR cluster runs inside a private subnet with no public IP addresses on any node; the only way in is via SSH through a bastion, and in practice, nobody needs SSH access at all once the steps are automated. The core nodes hold HDFS and handle the steady-state workload, while task nodes run as a Spot fleet because Pig's MapReduce jobs tolerate node loss reasonably well as long as
mapreduce.map.maxattempts and mapreduce.reduce.maxattempts are left at sane values, losing a task node mid-job just means the affected tasks get rescheduled, not a failed run. That single decision cut the compute bill by roughly a third compared to running everything on-demand.The IAM side is worth a sentence: the cluster runs under
EMR_EC2_DefaultRole scoped down to read the raw bucket and write the curated bucket, nothing broader. Pig scripts that write output don't need catalog permissions themselves; a separate Glue crawler, running on its own schedule, is what keeps the catalog in sync with whatever partitions Pig produces.Standing up the cluster
For a workload this predictable, I didn't want a cluster living 24/7 racking up cost during idle hours, but I also didn't want the fifteen-minute bootstrap penalty on every single run. The compromise was a cluster that stays up for the nightly batch window and terminates itself afterward, provisioned through a CLI call that's version-controlled alongside everything else:
Bash
aws emr create-cluster \--name "pig-clickstream-nightly" \--release-label emr-6.15.0 \--applications Name=Pig Name=Hadoop \--instance-groups \InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \InstanceGroupType=CORE,InstanceCount=3,InstanceType=m5.2xlarge \InstanceGroupType=TASK,InstanceCount=4,InstanceType=m5.2xlarge,BidPrice=OnDemandPrice \--ec2-attributes SubnetId=subnet-0a1b2c3d,InstanceProfile=EMR_EC2_DefaultRole \--service-role EMR_DefaultRole \--log-uri s3://our-emr-logs/clickstream/ \--auto-terminate \--tags team=data-eng env=prod
The
--auto-terminate flag matters more than it looks: without it, an idle cluster from a failed run can sit there burning money until someone notices, which is exactly the kind of manual babysitting this project was supposed to eliminate.The Pig script itself
I'm not going to pretend the fifteen production scripts are exciting reading, but the shape of the core aggregation job is representative:
pig
raw = LOAD '$INPUT' USING PigStorage('\t')AS (session_id:chararray, user_id:chararray, event_type:chararray,page:chararray, ts:long);filtered = FILTER raw BY event_type == 'pageview';sessions = GROUP filtered BY session_id;session_stats = FOREACH sessions GENERATEgroup AS session_id,COUNT(filtered) AS page_count,MIN(filtered.ts) AS session_start,MAX(filtered.ts) AS session_end;STORE session_stats INTO '$OUTPUT' USING ParquetStorer();
$INPUT and $OUTPUT are parameters passed at submission time rather than hardcoded, which is what lets the same script run against dev, staging, and prod paths without a single line changing. The one adaptation the existing scripts needed wasn't logic; it was swapping their output storage function from a plain PigStorage write to ParquetStorer , since Athena and Glue work far better against columnar Parquet than the tab-separated files the on-prem cluster used to produce.Wiring up deployment
The part that actually changed day-to-day life for the data engineering team was automating step submission so that merging a change to a
.pig script is what triggers a run, not someone SSHing into a master node.
The GitHub Actions workflow assumes an IAM role via OIDC, no long-lived access keys stored as repo secrets, syncs the changed script to the S3 scripts bucket, and adds it as a step on the existing cluster:
YAML
name: deploy-pigon:push:branches: [main]paths: ['pig/**.pig']jobs:submit-step:runs-on: ubuntu-latestpermissions:id-token: writecontents: readsteps:- uses: actions/checkout@v4- uses: aws-actions/configure-aws-credentials@v4with:role-to-assume: arn:aws:iam::123456789012:role/gha-pig-deployaws-region: us-east-1- name: Sync script to S3run: aws s3 sync pig/ s3://our-pig-scripts/pig/ --delete- name: Submit Pig steprun: |aws emr add-steps \--cluster-id ${{ vars.EMR_CLUSTER_ID }} \--steps Type=PIG,Name="clickstream-nightly",ActionOnFailure=CONTINUE,\Args=[-f,s3://our-pig-scripts/pig/session_stats.pig,-p,INPUT=s3://our-raw-zone/events/,-p,OUTPUT=s3://our-curated-zone/session_stats/]
ActionOnFailure=CONTINUE was a deliberate choice rather than the default. With fifteen scripts occasionally sharing a cluster, one bad step shouldn't take the whole cluster down for the others queued behind it. The workflow then polls describe-step until the step reports COMPLETED or FAILED , and posts the result to Slack either way, which is the single change that got the team to stop manually checking the EMR console every morning.What didn't go smoothly?
Two things bit me during the build that I'd tell anyone else attempting this.
First,
ILLUSTRATE Pig's built-in tool for sampling data through a script without running the full job is genuinely useful for debugging locally, but it does not behave identically to a real run against skewed production data. I caught a join that looked fine under ILLUSTRATE and then ran for forty extra minutes in production because one session_id had an outlier number of events. The fix was USING SKEWED on the join, which tells Pig to sample the join keys first and redistribute the heaviest ones across more reducers instead of dumping them all on one. That single keyword change cut the slowest job's runtime by more than half.Second, Spot task node interruptions were noisier than I expected the first week, not because Spot capacity was scarce but because I'd initially sized the task fleet too aggressively relative to the core nodes, so losing even one or two task nodes meant a meaningful chunk of in-flight reduce tasks had to be rescheduled. Rebalancing the ratio of core to task nodes, and giving reducers a bit more headroom via
SET default_parallel made the interruptions a non-event instead of an occasional half-hour delay.Where it landed
The nightly batch window that used to require someone awake to babysit an on-prem cluster now runs unattended, costs roughly a third less than the always-on alternative would have, and ships a Slack message when it's done. Nobody had to relearn Pig Latin, and nobody had to defend keeping Pig around to a stakeholder who'd rather have heard "Spark." Sometimes the right AWS architecture is the one that lets fifteen years of working code keep working, just somewhere cheaper and less fragile than where it used to live.
Conclusion
The lesson here isn't really about Pig, or even about EMR; it's about resisting the urge to modernize for its own sake. This client had working code, a team that understood it deeply, and a business problem that had nothing to do with the processing engine underneath it. The actual problems were operational: an aging cluster nobody wanted to maintain, and deploys that depended on someone remembering to SSH in and run a command by hand. Solving those didn't require a rewrite; it required moving the same logic onto infrastructure that manages itself, and wiring deploys into something that can't be forgotten.
If you're staring down a similar decision: legacy scripts, a framework that's fallen out of fashion, pressure to rewrite in whatever's trendy this year, it's worth asking what's actually broken. Sometimes it's the code. More often, in my experience, it's everything around the code: how it gets deployed, how it's monitored, and who has to wake up at 2 a.m. when it fails. Fix that, and "boring" technology can keep earning its keep for a lot longer than anyone expects.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article