Building a Governed Procurement Data Lake on AWS
Learn how to build a governed procurement data lake with Amazon S3, AWS Glue, Lake Formation, Athena, and Amazon Quick Sight to consolidate purchasing data, improve visibility, and create trusted procurement dashboards.
Introduction
Procurement data is usually distributed across multiple systems.
Purchase orders may be stored in an enterprise resource planning platform, contracts in a document repository, supplier information in a master-data application, invoices in an accounts-payable system, and performance indicators in spreadsheets.
Each system may use different identifiers, formats, currencies, category structures, and definitions.
As a result, even basic questions can become difficult to answer:
- How much did the organization spend?
- Which suppliers received the most spend?
- Which purchases were made without a contract?
- Which categories are concentrated in one supplier?
- How long does the approval process take?
- Which business units have the highest number of exceptions?
In this article, I present a reference architecture for building a governed Procurement data lake on AWS.
The objective is to centralize authorized data, preserve its history, improve its quality, control access, and create reusable analytical datasets.
Disclaimer: This is a generic reference architecture. All suppliers, amounts, identifiers, datasets, and queries are fictional. The architecture must be adapted to the organization’s security, privacy, retention, accounting, and data-governance requirements.
Business Objectives
A Procurement data lake should support defined business outcomes.
Possible objectives include:
- Spend visibility
- Supplier performance monitoring
- Contract utilization analysis
- Purchase-order compliance
- Maverick-spend detection
- Supplier concentration analysis
- Procurement cycle-time analysis
- Risk and mitigation reporting
- Savings tracking
- Budget-versus-actual analysis
The project should not begin by collecting every available field.
It should begin with clear business questions, responsible data owners, agreed definitions, and approved access rules.
Reference Architecture
The proposed architecture uses:
- Amazon S3 as the data-lake storage layer.
- AWS Glue to discover, prepare, validate, and transform data.
- AWS Glue Data Catalog as the metadata repository.
- AWS Lake Formation to manage governed access to data-lake resources.
- Amazon Athena to query data in Amazon S3 using SQL.
- Amazon Quick Sight to create Procurement dashboards.
- AWS Lambda to perform lightweight event-driven processing.
- Amazon EventBridge to initiate processing when new data becomes available.
- Amazon CloudWatch to monitor jobs, failures, and operational metrics.
- AWS CloudTrail to support auditing of AWS API activity.
The AWS Glue Data Catalog stores dataset metadata and integrates with Athena and Lake Formation. Lake Formation can apply fine-grained permissions to Data Catalog resources and Amazon S3 locations, while Athena analyzes Amazon S3 data with SQL. Quick Sight can use Athena as a data source for analytical datasets and dashboards.
Data-Lake Layers
A practical design separates data according to its processing stage.
AWS Prescriptive Guidance describes raw, stage, and analytics layers, with an additional landing layer when sensitive data requires processing before entering the main lake.
Landing Layer
The landing layer temporarily receives incoming files.
Possible sources include:
- ERP exports
- Purchase-to-pay platforms
- Contract systems
- Supplier-management applications
- Risk systems
- Accounts-payable systems
- Authorized external data providers
Files should be validated before moving to the raw layer.
Validation may include:
- File type
- Expected filename
- Source identity
- File size
- Schema version
- Encryption
- Malware controls
- Required columns
- Duplicate-file detection
Sensitive fields can also be removed, tokenized, or masked when they are not required for analytics.
Raw Layer
The raw layer preserves the data in its original approved form.
This layer supports:
- Traceability to the source
- Reprocessing
- Investigation of transformation issues
- Historical reconstruction
- Comparison between source and analytical records
Raw data should generally not be exposed directly to dashboard users.
Stage Layer
The stage layer contains validated and standardized data.
Typical transformations include:
- Converting dates to a standard format
- Converting text to consistent case
- Standardizing currency codes
- Mapping supplier identifiers
- Mapping category structures
- Removing duplicate records
- Validating numeric values
- Converting files to analytical formats
- Adding ingestion timestamps
- Adding source-system identifiers
Analytics Layer
The analytics layer contains datasets organized for specific business use cases.
Examples include:
- Spend analysis
- Supplier performance
- Contract utilization
- Purchase-order compliance
- Risk monitoring
- Procurement cycle time
- Savings reporting
This layer should contain agreed business definitions rather than unverified source-system terminology.
Example Amazon S3 Structure
A simplified structure could look like this:
s3://example-procurement-landing/ source=erp/ source=contracts/ source=supplier-management/
s3://example-procurement-raw/ source=erp/entity=purchase_orders/year=2026/month=08/ source=erp/entity=invoices/year=2026/month=08/ source=contracts/entity=contract_metadata/year=2026/month=08/
s3://example-procurement-stage/ domain=purchasing/table=purchase_orders/year=2026/month=08/ domain=suppliers/table=supplier_master/year=2026/month=08/ domain=contracts/table=contracts/year=2026/month=08/
s3://example-procurement-analytics/ use_case=spend-analysis/ use_case=contract-compliance/ use_case=supplier-concentration/The actual bucket and prefix strategy should follow organizational standards for environments, AWS accounts, Regions, data classification, and ownership.
Core Procurement Data Domains
A Procurement data lake may contain several data domains.
Supplier Master
Possible fields include:
- Supplier ID
- Legal name
- Country
- Status
- Procurement category
- Approval status
- Risk level
- Parent supplier
- Onboarding date
Purchase Requisitions
Possible fields include:
- Requisition ID
- Requester
- Cost center
- Category
- Requested amount
- Approval status
- Submission date
- Approval date
Purchase Orders
Possible fields include:
- Purchase-order ID
- Supplier ID
- Contract ID
- Order date
- Category
- Quantity
- Currency
- Net amount
- Cost center
- Business unit
Invoices
Possible fields include:
- Invoice ID
- Purchase-order ID
- Supplier ID
- Invoice date
- Payment date
- Currency
- Invoice amount
- Status
Contracts
Possible fields include:
- Contract ID
- Supplier ID
- Start date
- End date
- Contract value
- Currency
- Category
- Owner
- Renewal status
Supplier Performance
Possible fields include:
- Supplier ID
- Measurement period
- On-time delivery
- Defect rate
- Service-level result
- Incident count
- Corrective actions
Canonical Purchase-Order Record
A standardized purchase-order record could look like this:
{ "purchaseOrderId": "PO-2026-004281", "supplierId": "SUP-1048", "contractId": "CTR-2026-0042", "orderDate": "2026-08-20", "categoryId": "CAT-PROF-SERV", "costCenter": "CC-2200", "businessUnit": "Technology", "currency": "USD", "netAmount": 4500.00, "approvedSupplier": true, "sourceSystem": "erp", "sourceRecordId": "4281", "ingestionTimestamp": "2026-08-21T01:15:00Z", "schemaVersion": "1.0"}A source record should retain enough lineage information to identify where it originated.
Data Ingestion Process
A batch ingestion process may work as follows:
- The source system exports an approved file.
- The file is delivered to the landing bucket.
- An Amazon S3 event is sent to EventBridge.
- EventBridge invokes a Lambda validation function or starts an AWS Glue workflow.
- The file is validated.
- Invalid files are moved to a quarantine location.
- Valid files are copied to the raw layer.
- An AWS Glue job standardizes and transforms the data.
- The processed data is written to the stage layer.
- Business transformations create analytics-layer datasets.
- The Data Catalog is updated.
- CloudWatch records the processing result.
The process should be idempotent so that receiving the same file twice does not create duplicate business records.
Supplier Identity Resolution
Supplier identification is one of the most important data-quality challenges.
The same supplier may appear as:
Example Technology Services Ltd.Example Tech ServicesExample TechnologyEX TECH SERVICESThe organization should define a trusted supplier identifier and maintain cross-reference tables that map source-system identifiers to the canonical supplier.
Supplier matching should not rely only on company names.
Possible matching attributes include:
- Source supplier ID
- Legal name
- Registration identifier
- Country
- Parent organization
- Approved master-data identifier
Ambiguous matches should be routed for human review.
Data Quality
A dashboard is only as reliable as its underlying data.
AWS Glue Data Quality provides a managed way to evaluate data-quality rules using Data Quality Definition Language, or DQDL.
An illustrative ruleset for purchase-order data could look like this:
Rules = [ IsComplete "purchase_order_id", IsComplete "supplier_id", IsComplete "order_date", Uniqueness "purchase_order_id" = 1.0, ColumnValues "net_amount" >= 0, ColumnValues "currency" in ["USD", "EUR", "BRL"], Completeness "contract_id" > 0.70]Possible Procurement data-quality checks include:
- Purchase-order ID is present.
- Purchase-order ID is unique.
- Supplier ID exists in the supplier master.
- Amount is not negative.
- Currency uses an approved code.
- Order date is valid.
- Contract ID is present when required.
- Cost center is valid.
- Category mapping is complete.
- Ingestion date is within the expected period.
Failed records can be quarantined for investigation instead of being silently included in analytical datasets.
Querying Procurement Data
Athena can query the curated Amazon S3 datasets using SQL.
Spend by Category
SELECT category_id, currency, COUNT(DISTINCT purchase_order_id) AS purchase_orders, COUNT(DISTINCT supplier_id) AS suppliers, SUM(net_amount) AS total_spendFROM procurement_purchase_ordersWHERE order_date BETWEEN DATE '2026-01-01' AND DATE '2026-12-31'GROUP BY category_id, currencyORDER BY total_spend DESC;Amounts in different currencies should not be added together without an approved currency-conversion method.
Purchases Without a Contract
SELECT purchase_order_id, supplier_id, category_id, cost_center, order_date, net_amount, currencyFROM procurement_purchase_ordersWHERE contract_id IS NULL AND order_date BETWEEN DATE '2026-01-01' AND DATE '2026-12-31'ORDER BY net_amount DESC;A missing contract does not automatically mean that a purchase violated policy. Some categories or value ranges may not require a contract.
Supplier Concentration
WITH supplier_spend AS ( SELECT category_id, supplier_id, SUM(net_amount) AS supplier_spend FROM procurement_purchase_orders WHERE order_date BETWEEN DATE '2026-01-01' AND DATE '2026-12-31' GROUP BY category_id, supplier_id),category_spend AS ( SELECT category_id, SUM(supplier_spend) AS total_category_spend FROM supplier_spend GROUP BY category_id)SELECT s.category_id, s.supplier_id, s.supplier_spend, c.total_category_spend, ROUND( 100.0 * s.supplier_spend / NULLIF(c.total_category_spend, 0), 2 ) AS category_spend_percentageFROM supplier_spend sJOIN category_spend c ON s.category_id = c.category_idORDER BY category_spend_percentage DESC;High spend concentration is a risk signal, not an automatic conclusion that the supplier relationship is inappropriate.
Procurement Dashboards
Quick Sight can use the Athena datasets to create dashboards for different audiences.
Executive Dashboard
Possible indicators include:
- Total spend
- Addressable spend
- Spend under contract
- Top categories
- Top suppliers
- Savings and avoidance
- Open high supplier risks
Category-Management Dashboard
Possible indicators include:
- Spend by supplier
- Spend by business unit
- Contract utilization
- Supplier concentration
- Purchase-price trends
- Supplier performance
Compliance Dashboard
Possible indicators include:
- Purchases without a purchase order
- Purchases without a contract
- Unapproved suppliers
- Approval exceptions
- Missing category classification
- Transactions above defined thresholds
Each metric should have:
- A documented definition
- A data owner
- A calculation method
- A refresh frequency
- An approved audience
Access Control and Governance
Procurement datasets may contain sensitive commercial and organizational information.
Lake Formation can help implement fine-grained permissions for cataloged data and Amazon S3 locations.
The organization may define roles such as:
- Data-lake administrator
- Procurement data owner
- Data engineer
- Procurement analyst
- Finance analyst
- Risk analyst
- Dashboard reader
- Auditor
A category manager may need access to category-level spend but not employee-level details.
An executive user may need aggregated indicators but not individual transactions.
The architecture should also:
- Encrypt data in transit and at rest.
- Block public access to Procurement buckets.
- Apply least-privilege IAM permissions.
- Separate production and nonproduction datasets.
- Mask unnecessary personal information.
- Define retention and deletion policies.
- Record schema changes.
- Document data lineage.
- Review user access periodically.
- Prevent dashboard publication to unauthorized audiences.
Performance and Cost Optimization
Athena charges are influenced by the amount of data scanned.
Partitioning data can restrict the data scanned by a query, improving performance and reducing query cost.
Practical optimizations include:
- Store curated data in a columnar format such as Apache Parquet.
- Partition data according to common query patterns.
- Avoid selecting unnecessary columns.
- Compress analytical files.
- Avoid creating excessive small files.
- Use Athena workgroups.
- Set query limits where appropriate.
- Separate query-result locations.
- Apply Amazon S3 lifecycle policies.
- Monitor Quick Sight dataset refreshes.
- Review CloudWatch log-retention periods.
Partitioning should reflect actual access patterns. Creating too many partitions without a clear query benefit can increase operational complexity.
Operating Model
Technology alone does not create trusted Procurement data.
The organization should define:
- Business data owners
- Technical data owners
- Data stewards
- Metric owners
- Source-system owners
- Data-quality responsibilities
- Access-approval processes
- Incident-management procedures
- Schema-change processes
A data catalog is useful only when datasets have understandable names, descriptions, owners, classifications, and definitions.
Business Benefits
A governed Procurement data lake can help an organization:
- Consolidate fragmented purchasing data.
- Improve spend visibility.
- Standardize supplier identifiers.
- Reuse analytical datasets.
- Detect data-quality issues.
- Reduce manual spreadsheet consolidation.
- Strengthen access governance.
- Create consistent Procurement indicators.
- Analyze supplier concentration.
- Identify potential compliance exceptions.
- Support category strategies.
- Preserve analytical history.
Important Limitations
A data lake does not automatically correct poor source data.
It also does not define Procurement policy or determine whether a transaction is compliant.
The quality of the result depends on:
- Source-system quality
- Supplier-master governance
- Category mapping
- Currency-conversion rules
- Metric definitions
- Business context
- Access controls
- Human review
Analytical indicators should support qualified decision-makers rather than replace them.
Conclusion
Procurement analytics requires more than copying files into a storage location.
The organization must define business questions, establish trusted identifiers, validate data quality, control access, and create documented metrics.
Amazon S3 can provide scalable storage, AWS Glue can catalog and transform the information, Lake Formation can support governed access, and Athena can query the analytical datasets using SQL.
Amazon Quick Sight can then present those datasets through Procurement dashboards.
By starting with a focused use case, such as spend visibility or contract utilization, organizations can demonstrate value, improve data quality, and progressively expand the data lake across the Procurement lifecycle.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article