AWS Builder Center
Evaluate a SOC Knowledge Base on Amazon S3 Vectors

Evaluate a SOC Knowledge Base on Amazon S3 Vectors

Load 7115 SOC chunks from Wazuh documentation, MITRE ATT&CK, playbooks and MISP threat intelligence into Amazon Bedrock Knowledge Bases on Amazon S3 Vectors, verify every stored text, and test runbook answers, exact indicators, stale and missing sources, latency and cost against a Wazuh Indexer baseline. Length is checked with the counter in the review; the limit is 512 characters.

I do magic

Evaluate a SOC Knowledge Base on Amazon S3 Vectors

A runbook answer needs its containment section

The question was the one a tier 1 analyst asks in the middle of an incident: what does our runbook prescribe for an SSH brute force against  web-server-01 ? The answer lives in one section of one document, the Containment section of a playbook, and a retrieval layer that returns five similar paragraphs without that section produces a fluent answer that is not the team's procedure.
I loaded the same 7115 text chunks into two stores: an Amazon Bedrock Knowledge Base whose vectors live in Amazon S3 Vectors, and a Wazuh Indexer, the OpenSearch-based store of the Wazuh SIEM, running k-NN search over Titan embeddings of the same text. For that question the Knowledge Base ranked the playbook's Scope and Containment sections first and third in the top five, and Claude answered with the 12-hour firewall block the playbook prescribes. The Indexer's semantic search returned four sections of a public Wazuh documentation page and one MITRE ATT&CK technique, and the model said it could not find a runbook for that host.
The corpus has four sources: MITRE ATT&CK as a Wazuh manager serves it, the Wazuh 4.14 user manual at a pinned commit, four playbooks I wrote for the lab, and 200 pinned MISP threat intelligence events. This article loads that corpus into a Knowledge Base, checks that what it stored is what was sent, and runs the controls that decide whether its answers can be trusted: an exact indicator, a stale document, a missing source, invalid metadata and a refused reader. Wazuh appears only as the origin of the corpus and as the baseline; the AWS path needs no Wazuh installation.

What runs in S3, Bedrock, and the client

Documents and vectors sit in two different buckets. A general purpose S3 bucket holds one  .txt  object and one  .metadata.json  object per chunk under the  chunks/  prefix, plus the four playbooks as whole files under  sources/ . An S3 vector bucket holds one vector index,  soc-knowledge . Between them, the Knowledge Base owns the ingestion: its S3 data source reads  chunks/ , it calls Titan Text Embeddings V2 for a 1024-dimensional float32 vector per document, and it writes the vector, the text and the metadata into the index ( S3 Vectors with Bedrock Knowledge Bases ).

The client calls  Retrieve  and sends the returned passages to Claude Sonnet 4.5 in a separate  Converse  call. I did not use RetrieveAndGenerate: with the two steps apart, the context the model saw is stored before the answer exists, and every claim in the answer can be checked against it. The two steps also need different permissions, because Converse requires  bedrock:InvokeModel  and the reader role holds nothing on any model, so retrieval and generation run under separate identities. Three jobs stay with the client. It reads a whole playbook from  sources/  when a section is not enough, it chooses a version filter when a document exists in two versions, and it checks an exact indicator, because a Knowledge Base on an S3 vector store offers semantic search only, with no hybrid text search and no  startsWith  or  stringContains  filters ( query configuration ).
The baseline is a Wazuh Indexer 4.14.7 with a  lucene  HNSW index over the same text. Its ingest pipeline calls the same Titan model, and it adds one thing the Knowledge Base does not have: a hybrid query that combines a keyword  term  clause on an  indicators  field with the semantic clause.

Prerequisites and a bounded lab

The lab ran in  us-east-1  from one client machine with Python, boto3 1.43.51 and botocore 1.43.52. The account needs model access to  amazon.titan-embed-text-v2:0  and to Claude Sonnet 4.5 through the  us.anthropic.claude-sonnet-4-5-20250929-v1:0  inference profile. Per the  model access documentation , foundation models are enabled by default for an identity that carries the AWS Marketplace permissions, and Anthropic models additionally require the first-time use form, once per account or organization. That profile id is a cross-Region profile: per the  inference profile documentation , a request is routed to one of the profile's destination Regions, and the inference actions have to be allowed in all of them. Every resource below that accepts tags carries two, a lab name and a run id, and the cleanup deletes only what carries this run's id.
Resource
Purpose
Notes
General purpose S3 bucket
Chunk objects, metadata files, whole playbooks
Public Access Block set and read back; default SSE-S3 encryption and tags verified
S3 vector bucket and index soc-knowledge
Vectors, stored text, metadata
float32, dimension 1024, cosine; two keys declared non-filterable
Knowledge Base service role
Assumed by Bedrock during sync and retrieval
Titan invoke, S3 read on chunks/, five S3 Vectors actions on the one index
Knowledge Base and S3 data source
Ingestion and retrieval
chunkingStrategy NONE,dataDeletionPolicy DELETE
Reader role
Retrieval and whole-document reads
bedrock:Retrieve on one Knowledge Base, s3:GetObject on sources/*, and no permission on any model
Generation identity
The separate Converse call
bedrock:InvokeModel on the inference profile and on the models it routes to; in this lab the operator identity made those calls
Three identities keep the permissions apart. The operator who creates resources needs IAM, S3, S3 Vectors and Bedrock administration; the service role needs only what the  Knowledge Base permissions page  lists for an S3 data source and an S3 vector index; the reader needs two actions. On this setup the reader role called Retrieve successfully without any permission on the embedding model.
The whole experiment, including a second Knowledge Base for the controls, a small probe base described below, the Indexer baseline and 33 generation calls, came to between 0.75 at list prices. That figure is a calculation from measured volumes and the price list, not an invoice.

Freeze the corpus before choosing chunk boundaries

Both stores received text from one frozen snapshot. The capture step fetched every source body once and stored the raw bytes with their digests; the corpus replay and chunk generation then ran offline against the stored bodies, through the loaders of the original lab, and two replays produced byte-identical chunk files. The Wazuh documentation comes from one pinned commit, the 200 MISP events matched the digests recorded eight days earlier, and MITRE ATT&CK is the manager's dataset at  mitre_version  2.0.
Source
Documents
Chunks
MITRE ATT&CK (techniques, mitigations, tactics)
1031
1031
Wazuh 4.14 user manual, commit e3296dd
316 pages
3,851
SOC playbooks
4
24
MISP OSINT events
200
2209
Total

7115
The chunk boundaries were decided before upload: 3000 characters with a 200-character overlap, cut at section titles first. The Knowledge Base does no chunking of its own. Per the  chunking documentation , with strategy  NONE  each file is treated as one chunk, so one chunk of the snapshot becomes one object and one vector. Here is the Containment section of the SSH playbook as uploaded; the metadata file is pretty-printed here and holds 573 bytes on disk.
Containment

Block the source address on the host firewall of web-server-01 with the active response firewall-drop for 12 hours. Confirm the action in /var/ossec/logs/active-responses.log and verify the firewall state on the endpoint. A zero count of later alerts is supporting evidence only, because the source may have stopped sending traffic. Do not disable the sshd service.

{
"metadataAttributes": {
"chunk": {"value": {"type": "NUMBER", "numberValue": 3}, "includeForEmbedding": false},
"chunk_key": {"value": {"type": "STRING", "stringValue": "28166435c35efdcedeacb3a9de15a52667d53263"}, "includeForEmbedding": false},
"doc_id": {"value": {"type": "STRING", "stringValue": "ssh-brute-force-web-server-01.md"}, "includeForEmbedding": false},
"source": {"value": {"type": "STRING", "stringValue": "playbook"}, "includeForEmbedding": false},
"version": {"value": {"type": "STRING", "stringValue": "playbook-2026-09"}, "includeForEmbedding": false}
}
}

Keeping the metadata to five keys is deliberate. The  vector store prerequisites  allow 1 KB of custom metadata and 35 keys per vector through this integration, far below the native limits of S3 Vectors, so long URLs and indicator lists stay in the local manifest. Before the full load I sent a pilot of 12 chunks, covering every source, the longest chunks and the SSH playbook; the largest custom metadata among them was 345 bytes. In the stored vectors the service added  AMAZON_BEDROCK_TEXT ,  AMAZON_BEDROCK_METADATA ,  x-amz-bedrock-kb-data-source-id  and  x-amz-bedrock-kb-source-file-modality , nine keys in all. With  includeForEmbedding  set to false, only the chunk text is embedded.
The snapshot changed twice after I saw what the Knowledge Base stored. The pilot showed that the service removes leading and trailing whitespace from a document and, on those 12 chunks, changed nothing else; 231 of the chunks had edge whitespace, so the snapshot now strips it for both stores. The second change came from the full load and is the subject of the next section.

Create the vector store and ingest the documents

The index configuration cannot be changed after creation, so it is the first thing to get right. The two service keys hold the chunk text and the service's own metadata. The vector store prerequisites require both to be non-filterable, and the text alone would not fit otherwise: a chunk of 3000 characters can exceed the 2 KB that  S3 Vectors allows for filterable metadata .
s3v.create_index(
vectorBucketName=vector_bucket, indexName='soc-knowledge',
dataType='float32', dimension=1024, distanceMetric='cosine',
metadataConfiguration={'nonFilterableMetadataKeys': ['AMAZON_BEDROCK_METADATA', 'AMAZON_BEDROCK_TEXT']},
tags=tags)

The service role starts with a trust policy for any Knowledge Base in the account, because the Knowledge Base id does not exist yet, and is narrowed to the one id immediately after creation. Before every sync the client compares the trust, the single inline policy and the absence of attached policies with the expected documents and refuses to start the job on any difference.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "bedrock.amazonaws.com"},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {"aws:SourceAccount": "<account-id>"},
"ArnLike": {"AWS:SourceArn": "arn:aws:bedrock:us-east-1:<account-id>:knowledge-base/<kb-id>"}
}
}
]
}

The first  CreateKnowledgeBase  call, about one second after  CreateRole , failed with  ValidationException  and the message that the Knowledge Base "was unable to assume the given role". IAM had not propagated the new role yet. The client now retries that one error every 10 seconds, for up to 12 attempts, with the same client token; a later creation needed one such wait.
kb = agent.create_knowledge_base(
clientToken=token, name='soc-s3v-main', roleArn=service_role_arn,
knowledgeBaseConfiguration={'type': 'VECTOR', 'vectorKnowledgeBaseConfiguration': {
'embeddingModelArn': 'arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0',
'embeddingModelConfiguration': {'bedrockEmbeddingModelConfiguration': {'dimensions': 1024, 'embeddingDataType': 'FLOAT32'}}}},
storageConfiguration={'type': 'S3_VECTORS', 's3VectorsConfiguration': {'indexArn': index_arn}})['knowledgeBase']

agent.create_data_source(
knowledgeBaseId=kb['knowledgeBaseId'], clientToken=ds_token, name='soc-s3v-chunks',
dataSourceConfiguration={'type': 'S3', 's3Configuration': {
'bucketArn': 'arn:aws:s3:::' + docs_bucket, 'inclusionPrefixes': ['chunks/']}},
dataDeletionPolicy='DELETE',
vectorIngestionConfiguration={'chunkingConfiguration': {'chunkingStrategy': 'NONE'}})

The upload put 14,212 objects in 388 seconds with 16 threads; the 24 objects of the pilot were already present with the same SHA-256 and were skipped. One  StartIngestionJob  then ran for 1929 seconds and finished  COMPLETE : 7118 documents and 7118 metadata files scanned, 7106 new (the other 12 were the pilot), zero failed.
A complete job with zero failures still does not show what the index holds. I read every vector back with  ListVectors , took  AMAZON_BEDROCK_TEXT  and compared its SHA-256 with the snapshot text of the same chunk:
def stored_texts(s3v, vector_bucket, index):
token = None
while True:
page = s3v.list_vectors(vectorBucketName=vector_bucket, indexName=index, returnMetadata=True,
maxResults=1000, **({'nextToken': token} if token else {}))
for v in page['vectors']:
yield v['metadata']['chunk_key'], v['metadata']['AMAZON_BEDROCK_TEXT']
token = page.get('nextToken')
if not token:
return

Three chunks came back different. Their markup tags were gone, paragraph breaks had become spaces, a curly quote had become a straight one and backslashes had appeared. The shortest of them, a documentation example made of two XML configuration blocks, went in as 455 bytes and was stored as 194: the values survived as one line of words separated by spaces, and every element name that said what those values configure was gone.
The three chunks are exactly the chunks of the snapshot that contain the string  <title ; of the 1519 chunks with tag-like tokens, only these three changed. The service stored these  .txt  objects, uploaded with  Content-Type: text/plain , as if it had parsed them as HTML. The documentation says the default parser handles text in  .txt ,  .md ,  .html  and other text files but does not describe how the type is detected, so the marker is an observed correlation, not a documented rule. To rule out the S3 connector, I ingested the same three chunks plus the 12 pilot chunks into a separate Knowledge Base with a custom data source, sending the text inline through  IngestKnowledgeBaseDocuments . The same three texts were stored with byte-identical conversions, and all 15 vectors matched the main index with a cosine similarity of 1.0. These runs do not establish whether the embedding is computed before or after the conversion.
I excluded chunks containing  <title  from both stores, which took the corpus from 7118 to 7115 chunks and removed no text the question set depends on. After deleting the six objects, a second sync reported 7115 scanned, 3 deleted, no new, modified or failed documents, and the readback found no differences; the whole step took 275 seconds. For this corpus, the success criterion for an ingestion is the job statistics together with a text readback. The status alone reported success for a load that had changed three documents.

Inspect retrieval before generating an answer

The client assumes the reader role for retrieval. A version filter is part of the same call:
results = runtime.retrieve(
knowledgeBaseId=kb_id,
retrievalQuery={'text': question},
retrievalConfiguration={'vectorSearchConfiguration': {
'numberOfResults': 5,
'filter': {'equals': {'key': 'version', 'value': 'docs-4.14'}}}})['retrievalResults']

Each result carries the text, the S3 location of its object, a score and the metadata, including the five custom keys and an  x-amz-bedrock-kb-chunk-id  added by the service. On the pilot, filters on  source  and  version  and an  andAll  on  source  and  doc_id  returned only the chunks they named.
For the runbook question, the five results were the playbook's Scope section first, a section of the level-12 escalation playbook second, the Containment section third and two sections of the Wazuh page on blocking SSH brute force with active response. The Indexer's  neural  query over the same text returned four sections of that documentation page and MITRE technique T1184. Both stores embedded the same text with the same model at the same dimension; I did not establish why their rankings differ.
The generation step runs under the identity that holds  bedrock:InvokeModel ; in this lab the operator identity made those calls, never the reader role. It received the five passages, numbered, with a system prompt that asks for citations and a refusal when the passages do not contain the answer; temperature 0, at most 800 output tokens. From the Knowledge Base context, Claude answered:
According to the runbook for SSH brute force against web-server-01, the prescribed containment is:

**Block the source address on the host firewall of web-server-01 with the active response firewall-drop for 12 hours.** You should then:

1. Confirm the action in `/var/ossec/logs/active-responses.log`
2. Verify the firewall state on the endpoint
3. Note that a zero count of later alerts is only supporting evidence, as the source may have stopped sending traffic independently
4. **Do not disable the sshd service**

[3]

Every step in it is in passage 3. From the Indexer context the model reported that it could not find a runbook for  web-server-01  and described the generic active response from the documentation. The third context was the whole playbook, 1906 characters read by the reader role from  sources/  and accepted only after its SHA-256 matched the captured file; the answer was the same containment step. Section retrieval found the containment step for this question. The whole-document read contains it by construction, which starts to matter once a runbook has more sections than the top five can hold.
Each context was answered three times. At temperature 0 the three answers were identical in only one of eleven contexts; in the others the wording and the length changed, while the substance quoted above did not.

Test an exact indicator, a stale document, and a missing source

Exact indicator. The question asked whether a SHA-256 hash was known malware, once with a hash carried by one MISP event, once with the same hash with its last digit changed, and once with a hash that no chunk carries. The Indexer's hybrid query returned the carrying event first for the exact value and did not return it for the other two, because its  term  clause matches the  indicators  field exactly. The Knowledge Base returned the same event at position 4 for all three values: without an indicator field in its metadata and without the hash in the event's text, the event matched the topic of the question, not the value. The model declined in all six contexts, including the four that contained the carrying event, because the hash is recorded only in that field and never reached the prompt. An exact lookup has to run against the indicator data itself, and if the answer should cite the indicator, the value has to be in the text the model receives.
Stale document. A separate Knowledge Base held three documentation pages about vulnerability detection from both Wazuh 4.7 and 4.14, 23 chunks, with  version  in the metadata. Asked what the XML block for vulnerability detection is called, the unfiltered search returned three 4.14 chunks and two 4.7 chunks, with the current name at position 2 and the retired name at position 3; the five scores lay between 0.7591 and 0.7946. A filter on either version returned only that version. The model answered  vulnerability-detection  from the unfiltered context and cited the 4.14 passage in all three runs. Nothing in the retrieval result marked the 4.7 chunks as outdated; only the filter kept them out.
Missing source. The control base was then synchronized to 1098 chunks from the vulnerability detection and  ossec.conf  reference directories of the 4.14 manual, without the four pages that mention either tag name. The five results still arrived, with scores from 0.7546 to 0.7945, and none contained either name. The model said the context did not contain the tag name. That refusal is this run's output; the retrieval result itself gave no signal that the answer was missing.
Invalid metadata. Two test documents exceeded the metadata budget, one with 1161 bytes of custom metadata and one with 36 keys. The local validator refused both before upload; uploaded anyway for this control, they produced a  COMPLETE  job with zero failed and zero skipped documents, and the reason appeared only in  failureReasons : "Ignored 2 files as the associated metadata was larger than service limit of MaximumFileSizeSupported: 1024 bytes". A client that reads only the status and the failure count records a clean load with two documents missing. The 36-key file was also larger than 1,024 bytes, so this run does not separate the key limit from the size limit.
Refused reader. The reader role has Retrieve on the main Knowledge Base only. Its call against the control base returned  AccessDeniedException , and the client stopped without calling the model; in this client an access error cannot turn into an empty result that the model would answer from memory.

Compare retrieval quality, latency, and cost

Both stores held the same 7115 texts, each checked against the snapshot by reading every stored text back. Latency was measured from a laptop over the public internet to  us-east-1 . The Indexer ran in Docker on the same laptop and called Titan over the same link to embed each question. Every call was a single attempt with SDK retries disabled; the order of branches and questions was shuffled with a recorded seed, after three warm-up calls per branch; percentiles are nearest-rank.
Questions
Branch
Measured calls
Failed
p50, ms
p95, ms
24 EN, k=5, 10 repetitions
Knowledge Base
240
0
481.4
658.8
24 EN, k=5, 10 repetitions
Indexer neural
240
0
279.7
696.4
8 RU, k=5, 10 repetitions
Knowledge Base
80
0
477.8
517.1
8 RU, k=5, 10 repetitions
Indexer neural
80
0
276.0
304.9
The questions came with answers fixed before any retrieval: each has a verbatim excerpt in a primary chunk, and every chunk containing that excerpt counts as supporting. Coverage is the share of questions with a supporting chunk in the top k.
Questions
Knowledge Base
Indexer neural
24 EN, top 5
21 (MRR 0.778)
18 (MRR 0.681)
24 EN, top 10
23 (MRR 0.791)
21 (MRR 0.707)
8 RU, top 5 and top 10
7 (MRR 0.667)
5 (MRR 0.417)
These are 32 questions on one corpus from one client on one day, a diagnostic set and not a benchmark. The Russian group is kept apart: the  Titan Text Embeddings documentation  describes V2 as optimized for English and cross-language queries as sub-optimal, and these are Russian questions against an English corpus.
The following estimate uses measured volumes and the us-east-1 list prices in  S3 pricing  and  Bedrock pricing , with the S3 and Titan entries effective 2026-08-01 and the Claude entries effective 2026-07-01:
Component
Basis
Estimate
Titan V2 embeddings
2.11-4.44 million tokens across both stores, the control base and the probe, $0.00002 per 1000
$0.04-0.09
S3 requests
about 16,500 PUT and 43000 GET-class requests
$0.10
S3 Vectors writes
8255 vectors, about 50 MB logical; up to 1.08 GB if the 128 KB PUT minimum applied to each vector
$0.01-0.22
Claude Sonnet 4.5
33 calls, 71,595 input and 6505 output tokens
$0.31-0.34
Total

$0.46-0.75
For a running system, the split matters more than the total. At 10000 questions a month on this index of about 43 MB, the S3 Vectors query charges (0.004 per TB of index processed per query), vector storage, document storage and question embeddings at about 30 tokens each add up to about 95-104. The vector store is not where this design spends money.

Refresh the knowledge base and remove the lab

Updates go through the same bucket. In the control base I appended a sentence to one chunk and deleted another, then synchronized. The job reported 1 modified and 1 deleted document out of 1097 scanned, with no failures, and the text readback showed the new text for the edited chunk and no vector for the deleted one. A second sync with no changes reported nothing new, modified or deleted. The earlier removal of the three HTML-parsed chunks from the main base took the same path.
Removing the lab comes after the measurements above and is not one of them; what follows is the order the client uses and the check it makes after each call. The cleanup reverses the creation order, and every step deletes only a resource whose run tag matches. Each data source goes first; with  dataDeletionPolicy   DELETE , deleting it removes its vectors from the index. Then the Knowledge Bases, the service roles and the reader role with their inline policies, the vector indexes, the vector bucket, and finally the general purpose bucket, emptied in batches of 1000 keys before it is deleted. After each deletion the client repeats the read call until the service answers that the resource does not exist, and that answer is the confirmation the API can give; the invoice arrives later and is checked separately.
Removing this lab took three active deletion attempts, followed by a clean confirmation run, and the reason is worth knowing before you build the same thing. A data source created with  dataDeletionPolicy   DELETE  removes its vectors while it reports  DELETING , and on the main index that ran at roughly 46 vectors a minute, so deleting 7115 vectors outlived two bounded waits. Deleting the vector index instead removed the remaining 4283 vectors at once, but it also pulled the store out from under the running deletion: the data source, and then the knowledge base, ended in  DELETE_UNSUCCESSFUL , and the service named the remedy in  failureReasons : "Unable to delete data from vector store for data source with ID [...]. Check your vector store configurations and permissions and retry your request. If the issue persists, consider updating the dataDeletionPolicy of the data source to RETAIN and retry your request." With  RETAIN  set on the data source and both deletions repeated, the data source went in about five minutes and the knowledge base five minutes after it. The three indexes, the vector bucket, the general purpose bucket with its 16428 objects and the four roles were deleted and confirmed gone, and a final listing of the account found no bucket, vector bucket, knowledge base or role belonging to this lab. Delete the knowledge base and its data source first if you want the service to finish its own bookkeeping, or accept the  RETAIN  step as part of the procedure.

Where this design fits

For this corpus and these questions, the Knowledge Base on S3 Vectors covered more questions than the Indexer's semantic search, 21 of 24 in English and 7 of 8 in Russian at k=5, and it found the containment section that made the runbook answer correct. It cost cents to load, and storing its vectors costs less per month than one generated answer. Its median retrieval latency from a laptop, about 480 ms, was about 200 ms above the Indexer's; the Indexer ran on that laptop and crossed the network only to embed the question.
It is the better choice when the corpus is prose, when managed ingestion and IAM-scoped access matter more than query flexibility, and when the application already separates retrieval from generation. It is the weaker choice when an answer depends on exact values such as hashes, addresses or rule ids: the Knowledge Base returned the same event for a hash, a near miss and an absent value, and only the Indexer's keyword clause told them apart. Keep an exact lookup next to it, or put the value into the text if the model has to see it.
A trustworthy load requires checks beyond the ingestion status. Separate  COMPLETE  jobs with zero failures stored three documents altered in the main load and ignored two documents entirely in the control base, so read the stored text back and read  failureReasons  before trusting a sync. A filter on version is what separates a current document from a stale one, and in these controls the scores did not separate supporting passages from neighbours that do not answer the question.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article