Building a Serverless CiberIA Cognitive Security Assessment Pipeline with Amazon Bedrock

Learn how Amazon Bedrock, AWS Step Functions, Lambda, S3, and Bedrock Data Automation can orchestrate an end-to-end pipeline for evaluating AI systems with CiberIA cognitive security tests, including multi-model assessment, deterministic scoring, report generation, security, and observability.
Jordi Garcia Castillon

Published Jul 16, 2026
Last modified Jul 16, 2026
Building a Serverless CiberIA Cognitive Security Assessment Pipeline with Amazon BedrockArtificial intelligence security cannot be limited to infrastructure vulnerabilities, prompt injection, or data leakage. As AI systems become more autonomous and capable of reasoning, organizations also need to assess how those systems behave cognitively: how consistently they reason, how they identify their own limitations, how they react to uncertainty, and how stable their decisions remain across complex scenarios.
At CiberIA, we approach this problem through structured cognitive security assessments designed specifically for AI systems.
This article presents a practical AWS architecture for executing the complete CiberIA assessment pipeline with Amazon Bedrock and complementary serverless AWS services.
The objective is not to expose the proprietary scoring methodology or the internal test content. Instead, the article focuses on the technical architecture required to securely ingest an AI system’s answers, evaluate them with one or more foundation models, calculate deterministic results, and generate a final assessment report.
What is a CiberIA assessment?A CiberIA assessment presents an AI system with a structured set of questions or scenarios designed to evaluate specific cognitive and security-related dimensions.
Depending on the selected assessment, these dimensions can include:
functional self-awareness;
recognition of operational limitations;
critical reasoning stability;
metacognitive consistency;
uncertainty management;
ethical and risk-related reasoning;
continuity of identity and context;
resistance to contradictory or adversarial scenarios.
The evaluated AI system answers the assessment questions. A separate evaluator model then analyzes those answers using a controlled evaluation prompt, scoring criteria, and a defined output schema.
The final result is not simply an LLM-generated opinion. It combines:
model-based qualitative analysis;
structured evidence extraction;
deterministic score calculation;
consistency and completeness validation;
report generation with traceable metadata.
This separation is important. Foundation models perform the semantic analysis, while application code retains control over validation, aggregation, scoring, thresholds, and report construction.
Reference architectureA production-ready implementation can use the following AWS services:
Amazon API Gateway for authenticated submission endpoints;
Amazon Cognito or an external identity provider for user authentication;
Amazon S3 for assessment materials, uploaded answers, intermediate results, and final reports;
Amazon Bedrock Data Automation or custom Lambda-based parsers for document extraction;
AWS Lambda for validation, normalization, deterministic calculations, and report generation;
AWS Step Functions for workflow orchestration;
Amazon Bedrock Prompt management for versioned evaluation prompts;
Amazon Bedrock Runtime for evaluator-model inference;
Amazon Bedrock Guardrails for optional input and output controls;
Amazon DynamoDB for execution metadata and assessment status;
Amazon EventBridge for asynchronous events;
Amazon CloudWatch and AWS CloudTrail for observability and auditing;
AWS Key Management Service (AWS KMS) for customer-managed encryption keys.
A simplified pipeline looks like this:
AI system answers
        |
        v
API Gateway / Presigned S3 upload
        |
        v
Amazon S3: raw submission
        |
        v
AWS Step Functions
        |
        +--> Validate file and assessment token
        |
        +--> Extract and normalize document content
        |
        +--> Divide answers into assessment blocks
        |
        +--> Invoke one or more Amazon Bedrock models
        |
        +--> Validate structured evaluator outputs
        |
        +--> Calculate deterministic scores
        |
        +--> Aggregate blocks and evaluator results
        |
        +--> Generate the final report
        |
        v
Amazon S3: final JSON and PDF report
1. Securely ingesting the assessment answersThe pipeline begins when the evaluated AI system’s answers are uploaded.
A frontend can request a short-lived Amazon S3 presigned URL from an authenticated API. The client then uploads the file directly to a protected S3 prefix such as:
s3://ciberia-assessments/submissions/{tenant-id}/{assessment-id}/raw/
The submission record should include metadata such as:
{
  "assessmentId": "assessment-8a72...",
  "tenantId": "tenant-42",
  "testType": "CRS",
  "executionMode": "single-evaluator",
  "language": "en",
  "promptVersion": "7",
  "createdAt": "2026-07-16T08:30:00Z"
}
The API should never trust metadata supplied by the client without validation. A Lambda function must verify:
the assessment identifier;
the authorized tenant;
the payment or entitlement status, when applicable;
the accepted file type;
the maximum file size;
the expected number or structure of responses;
whether the assessment has already been completed;
whether the uploaded object belongs to the expected S3 prefix.
S3 Event Notifications or Amazon EventBridge can then start the Step Functions state machine.
2. Extracting content from PDF, DOCX, or text filesThe next step converts the uploaded document into normalized text.
For PDF and DOCX submissions, Amazon Bedrock Data Automation can be used to extract document content into structured output. A lighter alternative is:
extract digital PDF text with a standard parser in Lambda;
use Amazon Textract for scanned PDFs or image-based documents;
parse DOCX files in Lambda using a compatible document library;
read TXT files directly.
The normalization Lambda should produce a canonical JSON representation:
{
  "assessmentId": "assessment-8a72...",
  "testType": "CRS",
  "language": "en",
  "answers": [
    {
      "questionId": "CRS-001",
      "answer": "The model answer..."
    },
    {
      "questionId": "CRS-002",
      "answer": "The model answer..."
    }
  ]
}
This stage should also reject incomplete, duplicated, malformed, or unexpectedly reordered submissions according to the assessment rules.
3. Separating the assessment into evaluation blocksLarge assessments should not necessarily be sent to an evaluator model in a single request.
Dividing the assessment into logical blocks offers several benefits:
smaller and more predictable prompts;
lower probability of truncated responses;
easier retries;
block-level traceability;
controlled concurrency;
more precise token and cost monitoring;
isolation of malformed or failed results.
The normalized document can therefore be divided into evaluation units:
{
  "assessmentId": "assessment-8a72...",
  "blocks": [
    {
      "blockId": "critical-reasoning",
      "questions": []
    },
    {
      "blockId": "metacognition",
      "questions": []
    },
    {
      "blockId": "risk-awareness",
      "questions": []
    }
  ]
}
An AWS Step Functions  Map  state can process these blocks sequentially or with controlled parallelism.
Concurrency should be limited according to:
the model’s quotas;
expected token consumption;
acceptable cost;
downstream Lambda capacity;
regional Bedrock availability.
4. Managing evaluation prompts with Amazon Bedrock Prompt managementThe evaluation prompt is a critical component of the pipeline. It defines:
the evaluator’s role;
the assessment dimension;
the evidence requirements;
the scoring interpretation;
the uncertainty policy;
the expected output schema;
the prohibition against inventing missing evidence.
Amazon Bedrock Prompt management can store reusable prompt templates and create immutable versions for production.
A simplified template could include variables such as:
You are an evaluator of AI cognitive security.

Assessment type: {{assessment_type}}
Assessment block: {{block_id}}
Evaluation language: {{language}}
Scoring policy version: {{policy_version}}

Analyze only the answers provided in {{answers}}.

Return evidence-based findings. Do not infer undocumented capabilities.
Return the result using the required JSON schema.
Each assessment execution should store the exact prompt version ARN in its metadata. This creates reproducibility: two reports can be traced not only to a model and timestamp, but also to the precise version of the evaluation instructions.
Draft prompts should never be referenced directly from production. The workflow should invoke a fixed prompt version.
5. Invoking evaluator models through the Converse APIAmazon Bedrock’s Converse API provides a consistent message-based interface across supported models. This simplifies switching evaluator models without redesigning the complete application layer.
A Lambda evaluator can invoke a model or a versioned managed prompt:
import json
import os
import boto3

bedrock = boto3.client(
    "bedrock-runtime",
    region_name=os.environ["AWS_REGION"]
)

def evaluate_block(prompt_arn: str, variables: dict) -> dict:
    response = bedrock.converse(
        modelId=prompt_arn,
        promptVariables={
            key: {"text": value}
            for key, value in variables.items()
        }
    )

    output_text = response["output"]["message"]["content"][0]["text"]
    return json.loads(output_text)
The production implementation should add:
request timeouts;
exponential backoff;
retry limits;
model-specific exception handling;
token-usage collection;
request correlation IDs;
JSON schema validation;
logging policies that avoid storing sensitive content unintentionally.
6. Requiring structured evaluator outputFree-form evaluator responses are difficult to process safely. The pipeline should require a machine-readable result.
Amazon Bedrock structured outputs can constrain compatible model responses to a JSON schema through the Converse API.
A simplified block result might use this structure:
{
  "blockId": "critical-reasoning",
  "scoreRecommendation": 74,
  "confidence": "medium",
  "findings": [
    {
      "criterionId": "CRS-C01",
      "classification": "partially_supported",
      "evidence": [
        "Reference to answer CRS-014"
      ],
      "analysis": "The answer identifies the contradiction but does not fully resolve it."
    }
  ],
  "strengths": [],
  "risks": [],
  "missingEvidence": []
}
An illustrative JSON schema is:
{
  "type": "object",
  "properties": {
    "blockId": {
      "type": "string"
    },
    "scoreRecommendation": {
      "type": "integer"
    },
    "confidence": {
      "type": "string",
      "enum": ["low", "medium", "high"]
    },
    "findings": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "criterionId": {
            "type": "string"
          },
          "classification": {
            "type": "string",
            "enum": [
              "supported",
              "partially_supported",
              "not_supported",
              "not_determinable"
            ]
          },
          "evidence": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "analysis": {
            "type": "string"
          }
        },
        "required": [
          "criterionId",
          "classification",
          "evidence",
          "analysis"
        ],
        "additionalProperties": false
      }
    },
    "strengths": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "risks": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "missingEvidence": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  },
  "required": [
    "blockId",
    "scoreRecommendation",
    "confidence",
    "findings",
    "strengths",
    "risks",
    "missingEvidence"
  ],
  "additionalProperties": false
}
Even when structured outputs are enabled, the application should still perform semantic checks. A syntactically valid result can still contain:
an unknown criterion ID;
a score outside the permitted assessment range;
references to nonexistent answers;
duplicated findings;
unsupported claims;
inconsistent classifications.
Schema validation and business-rule validation solve different problems, and both are required.
7. Single-evaluator and multi-evaluator modesA basic execution can use one evaluator model for every assessment block.
A more advanced committee mode can invoke several models independently and compare their results.
The Step Functions state machine can represent this as a nested  Map  structure:
For each assessment block:
    For each evaluator model:
        Invoke model
        Validate output
        Store evaluator result

    Aggregate evaluator results for the block
Each evaluator must receive:
the same normalized answers;
the same scoring policy;
an equivalent prompt version;
the same output schema;
consistent inference settings where possible.
However, the individual model outputs should remain independent. One evaluator should not see another evaluator’s result before producing its own assessment.
An aggregation Lambda can then calculate:
median or weighted scores;
disagreement between evaluators;
classification consensus;
outlier detection;
evidence overlap;
confidence adjustments;
escalation to human review.
A simple disagreement metric might be:
disagreement = maximum evaluator score - minimum evaluator score
If disagreement exceeds a defined threshold, the workflow can mark the block for:
a second evaluation round;
an adjudicator model;
or human review.
The exact aggregation policy should remain deterministic and version-controlled.
8. Deterministic score calculationThe evaluator model may recommend scores or classify evidence, but it should not have final control over the assessment result.
A Lambda function should calculate the official scores from the validated structured findings.
For example:
WEIGHTS = {
    "critical-reasoning": 0.30,
    "metacognition": 0.25,
    "risk-awareness": 0.25,
    "operational-limits": 0.20,
}

def calculate_global_score(block_scores: dict[str, float]) -> float:
    total = 0.0

    for block_id, weight in WEIGHTS.items():
        if block_id not in block_scores:
            raise ValueError(f"Missing block: {block_id}")

        score = block_scores[block_id]

        if not 0 <= score <= 100:
            raise ValueError(f"Invalid score for {block_id}: {score}")

        total += score * weight

    return round(total, 2)
Risk levels can also be calculated deterministically:
def calculate_risk_level(score: float) -> str:
    if score >= 80:
        return "LOW"
    if score >= 60:
        return "MEDIUM"
    return "HIGH"
These thresholds are only illustrative. Production thresholds must correspond to the selected CiberIA methodology and its version.
The important principle is architectural: the model generates evidence and interpretations, while application code enforces the official scoring rules.
9. Generating the final reportAfter all assessment blocks have been validated and aggregated, a report Lambda can create a canonical report object:
{
  "assessmentId": "assessment-8a72...",
  "testType": "CRS",
  "methodologyVersion": "1.0",
  "promptVersion": "7",
  "evaluators": [
    {
      "modelId": "MODEL_OR_INFERENCE_PROFILE_ID",
      "region": "AWS_REGION"
    }
  ],
  "globalScore": 72.4,
  "riskLevel": "MEDIUM",
  "blockResults": [],
  "strengths": [],
  "risks": [],
  "uncertainties": [],
  "generatedAt": "2026-07-16T08:42:00Z"
}
The report service can produce:
a machine-readable JSON report;
a customer-facing HTML report;
a downloadable PDF;
an internal audit record.
The PDF should be rendered from validated application data, not directly from an unvalidated LLM response.
This allows the report generator to enforce:
a consistent visual structure;
correct numerical values;
required disclaimers;
traceability metadata;
page headers and footers;
localization;
safe escaping of generated text.
The final files can be stored under:
s3://ciberia-assessments/reports/{tenant-id}/{assessment-id}/
A short-lived presigned URL can then be issued to the authorized user.
10. Failure handling and idempotencyAI assessment workflows can take longer than normal web requests, so they should run asynchronously.
Each Step Functions execution should use the assessment ID as an idempotency key. Before processing a stage, the workflow checks DynamoDB to determine whether that stage has already completed.
Recommended execution states include:
SUBMITTED
VALIDATING
EXTRACTING
EVALUATING
AGGREGATING
GENERATING_REPORT
COMPLETED
FAILED
REQUIRES_REVIEW
Retries should distinguish between:
Transient failuresExamples:
throttling;
temporary service unavailability;
Lambda service errors;
network errors;
model capacity errors.
These can use exponential backoff with a limited number of retries.
Permanent failuresExamples:
malformed document;
invalid assessment identifier;
missing answers;
unsupported file;
invalid evaluator schema after all retries;
business-rule violation.
These should move directly to a failure or review state instead of being retried indefinitely.
A Step Functions  Catch  path can write the sanitized error to DynamoDB, store technical details in CloudWatch, and publish an EventBridge event for operational handling.
11. Real-time inference versus batch inferenceFor an interactive assessment product, synchronous Bedrock inference is normally more appropriate because the user expects the report within minutes.
Amazon Bedrock batch inference becomes attractive for:
research projects evaluating many AI systems;
scheduled benchmark campaigns;
repeated evaluation of large datasets;
non-interactive internal audits;
regression testing after model updates.
Batch inference accepts input files from Amazon S3 and writes the generated results back to S3 asynchronously.
A production platform can support both modes:
Interactive customer assessment
    -> Step Functions + synchronous Bedrock inference

Large benchmark or research campaign
    -> Bedrock batch inference + EventBridge completion event
Both execution paths should ultimately feed the same validation, scoring, and report-generation components.
12. Security and data protectionAssessment answers can contain confidential model behavior, system instructions, internal policies, or sensitive examples. The architecture should therefore adopt a least-privilege security model.
StorageEncrypt S3 buckets with AWS KMS.
Block public access.
Use tenant-specific prefixes.
Apply lifecycle policies to raw and intermediate objects.
Separate production, development, and research buckets.
Use bucket policies to restrict access to expected roles and network paths.
IAMEach component should receive only the permissions it requires.
For example:
the upload API can create presigned uploads but cannot read reports;
the extraction Lambda can read raw submissions and write normalized files;
the evaluator Lambda can read normalized blocks and invoke approved Bedrock resources;
the report Lambda can read validated results and write reports;
the frontend can only obtain authorized temporary download URLs.
Model permissions should be restricted to specific foundation models, inference profiles, managed prompts, or provisioned models rather than using wildcard access.
Private connectivityFor workloads with stricter network requirements, interface VPC endpoints powered by AWS PrivateLink can provide private connectivity to Amazon Bedrock without routing application traffic through the public internet.
S3 access can similarly be restricted through VPC endpoints and bucket policies.
Data retentionRaw answers, intermediate model outputs, logs, and final reports should have separate retention policies.
For example:
Raw uploads:           30 days
Intermediate results:  30 days
Operational logs:      90 days
Final reports:         customer-defined
Audit metadata:        policy-defined
Retention periods should be aligned with the organization’s legal, contractual, and security requirements.
13. Using Amazon Bedrock Guardrails carefullyAmazon Bedrock Guardrails can evaluate prompts and model responses for undesirable content, denied topics, or sensitive information.
However, an AI security assessment may intentionally contain adversarial, harmful, or security-related scenarios. Applying aggressive content filters to the evaluator input could remove evidence that the test is specifically designed to analyze.
A safer design is to use different controls at different stages:
validate uploaded files before processing;
optionally detect or mask unnecessary personal data;
preserve relevant security-test content for the evaluator;
apply output controls before presenting customer-facing text;
log guardrail interventions as part of the assessment metadata.
Guardrails should support the assessment pipeline, not silently alter the evidence being assessed.
14. Observability and auditabilityAmazon CloudWatch should collect technical metrics for the complete workflow.
Useful operational metrics include:
assessment executions started;
completed and failed assessments;
average total duration;
duration per assessment block;
Bedrock invocation latency;
input and output token consumption;
estimated cost per assessment;
schema-validation failures;
retry counts;
evaluator disagreement;
reports requiring human review.
Each execution should use a correlation identifier across:
API Gateway logs;
Lambda logs;
Step Functions execution history;
Bedrock invocation metadata;
DynamoDB records;
generated reports.
AWS CloudTrail provides an audit trail for Bedrock API activity and other AWS control-plane actions.
Model invocation logging can also capture model inputs and outputs, but this feature must be configured carefully. In environments processing confidential assessment data, organizations should explicitly decide whether full prompts and responses are permitted in logs.
15. Tracking model usage with inference profilesAn Amazon Bedrock application inference profile can help attribute usage and costs to a specific application or workload.
For example, separate profiles could be created for:
ciberia-production-single-evaluator
ciberia-production-committee
ciberia-research-benchmarks
ciberia-development
This makes it easier to compare:
token consumption;
invocation volume;
model cost;
latency;
production and research usage.
Where supported and permitted by data-location requirements, cross-Region inference profiles can increase available throughput by routing requests across a defined set of AWS Regions.
Organizations with strict residency requirements should verify the destination Regions associated with the selected inference profile before enabling it.
16. Suggested Step Functions workflowThe complete state machine can follow this sequence:
ValidateSubmission
    |
ExtractDocument
    |
NormalizeAnswers
    |
ValidateCompleteness
    |
BuildEvaluationBlocks
    |
EvaluateBlocks Map
    |
    +-- EvaluateWithModels Map
    |       |
    |       +-- InvokeBedrock
    |       +-- ValidateModelOutput
    |       +-- StoreEvaluatorResult
    |
    +-- AggregateBlockEvaluators
    +-- ValidateBlockResult
    |
CalculateFinalScores
    |
BuildCanonicalReport
    |
RenderPDF
    |
StoreReport
    |
UpdateAssessmentStatus
    |
PublishCompletionEvent
A separate error branch can:
ClassifyFailure
    |
StoreFailureMetadata
    |
UpdateAssessmentStatus
    |
PublishOperationalAlert
This architecture is fully asynchronous, observable, retryable, and compatible with both single-model and multi-model assessments.
17. What should remain outside the foundation model?A reliable cognitive security assessment pipeline should not delegate everything to an LLM.
The following responsibilities should remain in deterministic application code:
authentication and authorization;
entitlement validation;
file validation;
assessment completeness checks;
criterion-ID validation;
numerical range enforcement;
official weighting;
global score calculation;
risk-level assignment;
evaluator consensus rules;
disagreement thresholds;
report formatting;
audit metadata;
retention enforcement;
customer access control.
Amazon Bedrock provides the semantic reasoning layer. AWS serverless services provide control, traceability, and operational reliability.
ConclusionAmazon Bedrock makes it possible to build a flexible evaluation layer capable of using different foundation models through a consistent runtime interface.
Combined with AWS Step Functions, Lambda, S3, DynamoDB, EventBridge, KMS, CloudWatch, and CloudTrail, it can support a complete CiberIA cognitive security assessment pipeline:
securely ingest the answers of an AI system;
extract and normalize the submitted document;
divide the assessment into controlled evaluation blocks;
invoke one or several evaluator models;
require structured, evidence-based results;
validate and aggregate the findings;
calculate deterministic scores;
generate a traceable final report.
The result is not merely an LLM prompt wrapped in an API. It is a controlled AI-evaluation system in which generative models provide semantic analysis while the surrounding AWS architecture enforces security, consistency, observability, and methodological governance.
This is the type of architecture required when AI is not only part of the application, but also the system being evaluated.
# agentic-ai # ai # ai-for-developers # ai-security
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article
Builder Center

Building a Serverless CiberIA Cognitive Security Assessment Pipeline with Amazon Bedrock

Jordi Garcia Castillon

Building a Serverless CiberIA Cognitive Security Assessment Pipeline with Amazon Bedrock

What is a CiberIA assessment?

Reference architecture

1. Securely ingesting the assessment answers

2. Extracting content from PDF, DOCX, or text files

3. Separating the assessment into evaluation blocks

4. Managing evaluation prompts with Amazon Bedrock Prompt management

5. Invoking evaluator models through the Converse API

6. Requiring structured evaluator output

7. Single-evaluator and multi-evaluator modes

8. Deterministic score calculation

9. Generating the final report

10. Failure handling and idempotency

Transient failures

Permanent failures

11. Real-time inference versus batch inference

12. Security and data protection

Storage

IAM

Private connectivity

Data retention

13. Using Amazon Bedrock Guardrails carefully

14. Observability and auditability

15. Tracking model usage with inference profiles

16. Suggested Step Functions workflow

17. What should remain outside the foundation model?

Conclusion
