AWS Builder Center

Building a Governed Procurement Data Lake on AWS

Learn how to build a governed procurement data lake with Amazon S3, AWS Glue, Lake Formation, Athena, and Amazon Quick Sight to consolidate purchasing data, improve visibility, and create trusted procurement dashboards.

Introduction

Procurement data is usually distributed across multiple systems.
Purchase orders may be stored in an enterprise resource planning platform, contracts in a document repository, supplier information in a master-data application, invoices in an accounts-payable system, and performance indicators in spreadsheets.
Each system may use different identifiers, formats, currencies, category structures, and definitions.
As a result, even basic questions can become difficult to answer:
  • How much did the organization spend?
  • Which suppliers received the most spend?
  • Which purchases were made without a contract?
  • Which categories are concentrated in one supplier?
  • How long does the approval process take?
  • Which business units have the highest number of exceptions?
In this article, I present a reference architecture for building a governed Procurement data lake on AWS.
The objective is to centralize authorized data, preserve its history, improve its quality, control access, and create reusable analytical datasets.
Disclaimer: This is a generic reference architecture. All suppliers, amounts, identifiers, datasets, and queries are fictional. The architecture must be adapted to the organization’s security, privacy, retention, accounting, and data-governance requirements.

Business Objectives

A Procurement data lake should support defined business outcomes.
Possible objectives include:
  • Spend visibility
  • Supplier performance monitoring
  • Contract utilization analysis
  • Purchase-order compliance
  • Maverick-spend detection
  • Supplier concentration analysis
  • Procurement cycle-time analysis
  • Risk and mitigation reporting
  • Savings tracking
  • Budget-versus-actual analysis
The project should not begin by collecting every available field.
It should begin with clear business questions, responsible data owners, agreed definitions, and approved access rules.

Reference Architecture

The proposed architecture uses:
  1. Amazon S3 as the data-lake storage layer.
  1. AWS Glue to discover, prepare, validate, and transform data.
  1. AWS Glue Data Catalog as the metadata repository.
  1. AWS Lake Formation to manage governed access to data-lake resources.
  1. Amazon Athena to query data in Amazon S3 using SQL.
  1. Amazon Quick Sight to create Procurement dashboards.
  1. AWS Lambda to perform lightweight event-driven processing.
  1. Amazon EventBridge to initiate processing when new data becomes available.
  1. Amazon CloudWatch to monitor jobs, failures, and operational metrics.
  1. AWS CloudTrail to support auditing of AWS API activity.
The AWS Glue Data Catalog stores dataset metadata and integrates with Athena and Lake Formation. Lake Formation can apply fine-grained permissions to Data Catalog resources and Amazon S3 locations, while Athena analyzes Amazon S3 data with SQL. Quick Sight can use Athena as a data source for analytical datasets and dashboards.

Data-Lake Layers

A practical design separates data according to its processing stage.
AWS Prescriptive Guidance describes raw, stage, and analytics layers, with an additional landing layer when sensitive data requires processing before entering the main lake.

Landing Layer

The landing layer temporarily receives incoming files.
Possible sources include:
  • ERP exports
  • Purchase-to-pay platforms
  • Contract systems
  • Supplier-management applications
  • Risk systems
  • Accounts-payable systems
  • Authorized external data providers
Files should be validated before moving to the raw layer.
Validation may include:
  • File type
  • Expected filename
  • Source identity
  • File size
  • Schema version
  • Encryption
  • Malware controls
  • Required columns
  • Duplicate-file detection
Sensitive fields can also be removed, tokenized, or masked when they are not required for analytics.

Raw Layer

The raw layer preserves the data in its original approved form.
This layer supports:
  • Traceability to the source
  • Reprocessing
  • Investigation of transformation issues
  • Historical reconstruction
  • Comparison between source and analytical records
Raw data should generally not be exposed directly to dashboard users.

Stage Layer

The stage layer contains validated and standardized data.
Typical transformations include:
  • Converting dates to a standard format
  • Converting text to consistent case
  • Standardizing currency codes
  • Mapping supplier identifiers
  • Mapping category structures
  • Removing duplicate records
  • Validating numeric values
  • Converting files to analytical formats
  • Adding ingestion timestamps
  • Adding source-system identifiers

Analytics Layer

The analytics layer contains datasets organized for specific business use cases.
Examples include:
  • Spend analysis
  • Supplier performance
  • Contract utilization
  • Purchase-order compliance
  • Risk monitoring
  • Procurement cycle time
  • Savings reporting
This layer should contain agreed business definitions rather than unverified source-system terminology.

Example Amazon S3 Structure

A simplified structure could look like this:
s3://example-procurement-landing/
source=erp/
source=contracts/
source=supplier-management/

s3://example-procurement-raw/
source=erp/entity=purchase_orders/year=2026/month=08/
source=erp/entity=invoices/year=2026/month=08/
source=contracts/entity=contract_metadata/year=2026/month=08/

s3://example-procurement-stage/
domain=purchasing/table=purchase_orders/year=2026/month=08/
domain=suppliers/table=supplier_master/year=2026/month=08/
domain=contracts/table=contracts/year=2026/month=08/

s3://example-procurement-analytics/
use_case=spend-analysis/
use_case=contract-compliance/
use_case=supplier-concentration/
The actual bucket and prefix strategy should follow organizational standards for environments, AWS accounts, Regions, data classification, and ownership.

Core Procurement Data Domains

A Procurement data lake may contain several data domains.

Supplier Master

Possible fields include:
  • Supplier ID
  • Legal name
  • Country
  • Status
  • Procurement category
  • Approval status
  • Risk level
  • Parent supplier
  • Onboarding date

Purchase Requisitions

Possible fields include:
  • Requisition ID
  • Requester
  • Cost center
  • Category
  • Requested amount
  • Approval status
  • Submission date
  • Approval date

Purchase Orders

Possible fields include:
  • Purchase-order ID
  • Supplier ID
  • Contract ID
  • Order date
  • Category
  • Quantity
  • Currency
  • Net amount
  • Cost center
  • Business unit

Invoices

Possible fields include:
  • Invoice ID
  • Purchase-order ID
  • Supplier ID
  • Invoice date
  • Payment date
  • Currency
  • Invoice amount
  • Status

Contracts

Possible fields include:
  • Contract ID
  • Supplier ID
  • Start date
  • End date
  • Contract value
  • Currency
  • Category
  • Owner
  • Renewal status

Supplier Performance

Possible fields include:
  • Supplier ID
  • Measurement period
  • On-time delivery
  • Defect rate
  • Service-level result
  • Incident count
  • Corrective actions

Canonical Purchase-Order Record

A standardized purchase-order record could look like this:
{
"purchaseOrderId": "PO-2026-004281",
"supplierId": "SUP-1048",
"contractId": "CTR-2026-0042",
"orderDate": "2026-08-20",
"categoryId": "CAT-PROF-SERV",
"costCenter": "CC-2200",
"businessUnit": "Technology",
"currency": "USD",
"netAmount": 4500.00,
"approvedSupplier": true,
"sourceSystem": "erp",
"sourceRecordId": "4281",
"ingestionTimestamp": "2026-08-21T01:15:00Z",
"schemaVersion": "1.0"
}
A source record should retain enough lineage information to identify where it originated.

Data Ingestion Process

A batch ingestion process may work as follows:
  1. The source system exports an approved file.
  1. The file is delivered to the landing bucket.
  1. An Amazon S3 event is sent to EventBridge.
  1. EventBridge invokes a Lambda validation function or starts an AWS Glue workflow.
  1. The file is validated.
  1. Invalid files are moved to a quarantine location.
  1. Valid files are copied to the raw layer.
  1. An AWS Glue job standardizes and transforms the data.
  1. The processed data is written to the stage layer.
  1. Business transformations create analytics-layer datasets.
  1. The Data Catalog is updated.
  1. CloudWatch records the processing result.
The process should be idempotent so that receiving the same file twice does not create duplicate business records.

Supplier Identity Resolution

Supplier identification is one of the most important data-quality challenges.
The same supplier may appear as:
Example Technology Services Ltd.
Example Tech Services
Example Technology
EX TECH SERVICES
The organization should define a trusted supplier identifier and maintain cross-reference tables that map source-system identifiers to the canonical supplier.
Supplier matching should not rely only on company names.
Possible matching attributes include:
  • Source supplier ID
  • Legal name
  • Registration identifier
  • Country
  • Parent organization
  • Approved master-data identifier
Ambiguous matches should be routed for human review.

Data Quality

A dashboard is only as reliable as its underlying data.
AWS Glue Data Quality provides a managed way to evaluate data-quality rules using Data Quality Definition Language, or DQDL.
An illustrative ruleset for purchase-order data could look like this:
Rules = [
IsComplete "purchase_order_id",
IsComplete "supplier_id",
IsComplete "order_date",
Uniqueness "purchase_order_id" = 1.0,
ColumnValues "net_amount" >= 0,
ColumnValues "currency" in ["USD", "EUR", "BRL"],
Completeness "contract_id" > 0.70
]
Possible Procurement data-quality checks include:
  • Purchase-order ID is present.
  • Purchase-order ID is unique.
  • Supplier ID exists in the supplier master.
  • Amount is not negative.
  • Currency uses an approved code.
  • Order date is valid.
  • Contract ID is present when required.
  • Cost center is valid.
  • Category mapping is complete.
  • Ingestion date is within the expected period.
Failed records can be quarantined for investigation instead of being silently included in analytical datasets.

Querying Procurement Data

Athena can query the curated Amazon S3 datasets using SQL.

Spend by Category

SELECT
category_id,
currency,
COUNT(DISTINCT purchase_order_id) AS purchase_orders,
COUNT(DISTINCT supplier_id) AS suppliers,
SUM(net_amount) AS total_spend
FROM procurement_purchase_orders
WHERE order_date BETWEEN DATE '2026-01-01' AND DATE '2026-12-31'
GROUP BY category_id, currency
ORDER BY total_spend DESC;
Amounts in different currencies should not be added together without an approved currency-conversion method.

Purchases Without a Contract

SELECT
purchase_order_id,
supplier_id,
category_id,
cost_center,
order_date,
net_amount,
currency
FROM procurement_purchase_orders
WHERE contract_id IS NULL
AND order_date BETWEEN DATE '2026-01-01' AND DATE '2026-12-31'
ORDER BY net_amount DESC;
A missing contract does not automatically mean that a purchase violated policy. Some categories or value ranges may not require a contract.

Supplier Concentration

WITH supplier_spend AS (
SELECT
category_id,
supplier_id,
SUM(net_amount) AS supplier_spend
FROM procurement_purchase_orders
WHERE order_date BETWEEN
DATE '2026-01-01' AND DATE '2026-12-31'
GROUP BY category_id, supplier_id
),
category_spend AS (
SELECT
category_id,
SUM(supplier_spend) AS total_category_spend
FROM supplier_spend
GROUP BY category_id
)
SELECT
s.category_id,
s.supplier_id,
s.supplier_spend,
c.total_category_spend,
ROUND(
100.0 * s.supplier_spend / NULLIF(c.total_category_spend, 0),
2
) AS category_spend_percentage
FROM supplier_spend s
JOIN category_spend c
ON s.category_id = c.category_id
ORDER BY category_spend_percentage DESC;
High spend concentration is a risk signal, not an automatic conclusion that the supplier relationship is inappropriate.

Procurement Dashboards

Quick Sight can use the Athena datasets to create dashboards for different audiences.

Executive Dashboard

Possible indicators include:
  • Total spend
  • Addressable spend
  • Spend under contract
  • Top categories
  • Top suppliers
  • Savings and avoidance
  • Open high supplier risks

Category-Management Dashboard

Possible indicators include:
  • Spend by supplier
  • Spend by business unit
  • Contract utilization
  • Supplier concentration
  • Purchase-price trends
  • Supplier performance

Compliance Dashboard

Possible indicators include:
  • Purchases without a purchase order
  • Purchases without a contract
  • Unapproved suppliers
  • Approval exceptions
  • Missing category classification
  • Transactions above defined thresholds
Each metric should have:
  • A documented definition
  • A data owner
  • A calculation method
  • A refresh frequency
  • An approved audience

Access Control and Governance

Procurement datasets may contain sensitive commercial and organizational information.
Lake Formation can help implement fine-grained permissions for cataloged data and Amazon S3 locations.
The organization may define roles such as:
  • Data-lake administrator
  • Procurement data owner
  • Data engineer
  • Procurement analyst
  • Finance analyst
  • Risk analyst
  • Dashboard reader
  • Auditor
A category manager may need access to category-level spend but not employee-level details.
An executive user may need aggregated indicators but not individual transactions.
The architecture should also:
  • Encrypt data in transit and at rest.
  • Block public access to Procurement buckets.
  • Apply least-privilege IAM permissions.
  • Separate production and nonproduction datasets.
  • Mask unnecessary personal information.
  • Define retention and deletion policies.
  • Record schema changes.
  • Document data lineage.
  • Review user access periodically.
  • Prevent dashboard publication to unauthorized audiences.

Performance and Cost Optimization

Athena charges are influenced by the amount of data scanned.
Partitioning data can restrict the data scanned by a query, improving performance and reducing query cost.
Practical optimizations include:
  • Store curated data in a columnar format such as Apache Parquet.
  • Partition data according to common query patterns.
  • Avoid selecting unnecessary columns.
  • Compress analytical files.
  • Avoid creating excessive small files.
  • Use Athena workgroups.
  • Set query limits where appropriate.
  • Separate query-result locations.
  • Apply Amazon S3 lifecycle policies.
  • Monitor Quick Sight dataset refreshes.
  • Review CloudWatch log-retention periods.
Partitioning should reflect actual access patterns. Creating too many partitions without a clear query benefit can increase operational complexity.

Operating Model

Technology alone does not create trusted Procurement data.
The organization should define:
  • Business data owners
  • Technical data owners
  • Data stewards
  • Metric owners
  • Source-system owners
  • Data-quality responsibilities
  • Access-approval processes
  • Incident-management procedures
  • Schema-change processes
A data catalog is useful only when datasets have understandable names, descriptions, owners, classifications, and definitions.

Business Benefits

A governed Procurement data lake can help an organization:
  • Consolidate fragmented purchasing data.
  • Improve spend visibility.
  • Standardize supplier identifiers.
  • Reuse analytical datasets.
  • Detect data-quality issues.
  • Reduce manual spreadsheet consolidation.
  • Strengthen access governance.
  • Create consistent Procurement indicators.
  • Analyze supplier concentration.
  • Identify potential compliance exceptions.
  • Support category strategies.
  • Preserve analytical history.

Important Limitations

A data lake does not automatically correct poor source data.
It also does not define Procurement policy or determine whether a transaction is compliant.
The quality of the result depends on:
  • Source-system quality
  • Supplier-master governance
  • Category mapping
  • Currency-conversion rules
  • Metric definitions
  • Business context
  • Access controls
  • Human review
Analytical indicators should support qualified decision-makers rather than replace them.

Conclusion

Procurement analytics requires more than copying files into a storage location.
The organization must define business questions, establish trusted identifiers, validate data quality, control access, and create documented metrics.
Amazon S3 can provide scalable storage, AWS Glue can catalog and transform the information, Lake Formation can support governed access, and Athena can query the analytical datasets using SQL.
Amazon Quick Sight can then present those datasets through Procurement dashboards.
By starting with a focused use case, such as spend visibility or contract utilization, organizations can demonstrate value, improve data quality, and progressively expand the data lake across the Procurement lifecycle.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article