Domain adaptation for neural sparse search in Amazon OpenSearch Service

Fine-tune a neural sparse retriever for financial document search with knowledge distillation on Amazon SageMaker AI and Amazon OpenSearch Service

Alex Tarasov

Published Jun 20, 2026

Last modified Jun 20, 2026

Companies that search regulatory filings — SEC 10-K and 10-Q reports, earnings-call transcripts — are increasingly moving away from relying solely on keyword (BM25) search or on heavyweight dense-vector models. Neural sparse retrieval, shipped in Amazon OpenSearch Service as the neural_sparse query type, is an attractive middle ground: a SPLADE-style encoder expands each passage into a sparse bag of weighted vocabulary terms, so you keep BM25's cheap, interpretable, inverted-index serving while gaining the semantic recall that a learned model provides.

The constraint is domain fit. Off-the-shelf sparse encoders are trained on general web text, and financial filings are not general web text. They are dense with figures, fiscal periods, segment names, and accounting jargon, and a web-trained encoder systematically under-weights exactly the terms that distinguish one filing from another — so a search for "operating margin expansion" may rank a passage that merely repeats those words above the one that actually reports the number. Fine-tuning is intended to close that gap: teach the encoder what relevant means for this corpus without giving up the sparse model's speed.

A short glossary, since we use these terms throughout. Sparse vector / active tokens — the encoder turns a passage into a list of weighted vocabulary terms; "active tokens" is how many terms have a non-zero weight (fewer = sparser = cheaper to serve). Regularizer — a penalty added to the training loss; here the FLOPS regularizer pushes vectors to be sparse. Converged — training loss stopped meaningfully decreasing. Recall@100 — did the right filing land anywhere in the top 100 results (coverage for a downstream reranker). NDCG@10 / MRR — how high the right filing ranks in the top 10 (ranking quality). The practical conclusions do not depend on the underlying mathematics; these terms simply make the trade-offs precise.

In this post, we walk through how we did that on AWS. The result is worth stating up front: our first three fine-tuning runs each made the model worse, because they varied the wrong parameter. The run that succeeded changed the training objective — knowledge distillation from a cross-encoder teacher — and we provide the complete, reproducible recipe for it, including the exact hyperparameters and the training and inference hardware. We close with results on a human-annotated dataset: the improvement is meaningful for recall , and more modest for top-of-ranking quality (NDCG@10) than a synthetic test set we evaluated first had indicated.

Solution overview

The workflow has four moving parts, all on AWS:

Fine-tune the sparse encoder as an Amazon SageMaker AI training job on a single GPU instance. The base model is opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill, a 67M-parameter DistilBERT SPLADE encoder (BEIR average NDCG@10 0.517) that uses the v3 double-log activation log(1 + log(1 + ReLU(logits))), max-pooled over the sequence into a ~30,000-dimension sparse vector.
Package and deploy the resulting model to a SageMaker real-time endpoint behind a custom inference handler that returns the SPLADE {token: weight} map directly.
Wire the endpoint into Amazon OpenSearch Service with an ml-commons remote connector, so the sparse_encoding ingest pipeline and neural_sparse query call the endpoint transparently.
A/B evaluate the fine-tuned model against the unmodified base on the same corpus and query set, with a paired significance test — only the model differs.

The following diagram shows the end-to-end flow in three bands. Build is one-time: fine-tune → save to S3 → deploy one SageMaker endpoint. Index is the write path, and the only path that calls the model: OpenSearch's sparse_encoding ingest pipeline invokes the endpoint through the ml-commons connector to turn each document into a sparse vector, which it stores in a rank_features index. Query is the read path and is inference-free — because this is a doc-only model, a search request never calls the endpoint; the neural_sparse query expands the query terms locally (IDF token weights) and matches them against the stored document vectors in the index.

The challenge: why initial fine-tuning attempts reduced quality

Before presenting the recipe that worked, it is worth being precise about the three that didn't, because each failure motivated the next decision. We keep this brief — benchmarks, the one parameter that changed, and the takeaway — and reserve the full worked example for the knowledge-distillation run.

Our starting hypothesis was that the FLOPS sparsity regularizer — the loss term that keeps SPLADE vectors sparse — was the parameter to tune. Fine-tuning was, counter-intuitively, making each passage vector sparser, not denser: the contrastive InfoNCE objective and the FLOPS term work against each other, and at any non-trivial regularization weight the FLOPS term dominates the net token count. So we tuned it. All three runs shared the same backbone — InfoNCE contrastive loss on (query, positive, BM25 hard negatives) triples, lr=2e-5, batch_size=4, max_seq_len=512, 7 negatives, 1 epoch — and varied only the regularization, measured on a held-out set of 2,028 queries against an ~80,000-passage pooled corpus.

Run	What changed	Active doc tokens	NDCG@10	Recall@100	MRR
Base (off-the-shelf)	—	149	0.261	0.540	0.236
v1	`flops_lambda=0.05`, query + doc, no warmup	15	0.213	0.385	0.196
v2	`flops_lambda=0.008`	31	0.259	0.441	0.241
v3a	doc-only FLOPS, `λ=0.01`, quadratic warmup `T=300`, ℓ0-mask `t=200`, LR warmup	194	0.192	0.326	0.182

The results break down as follows:

v1 copied the dataset author's recommended flops_lambda=0.05 and collapsed each passage from 149 active tokens to 15. Every metric regressed significantly — Recall@100 fell 0.540 → 0.385 (p ≈ 4e-34).
v2 dropped the weight to 0.008 (~31 tokens). NDCG@10 and MRR recovered to statistical parity with the base, but Recall@100 was still down ~0.10. That split is the diagnostic signal: over-sparsification affects recall most, because every token dropped from a passage is a query the passage can no longer be retrieved by. The top-of-ranking metrics appeared acceptable while coverage remained degraded, so we continued investigating.
v3a applied the full recommended approach from the doc-v3-distill paper (arXiv:2504.14839): regularize documents only (never the query side of this asymmetric model), ramp the regularization weight in with a quadratic warmup, add the ℓ0-mask so already-sparse documents escape the penalty, and add an LR warmup-and-decay schedule. This met the sparsity target — ~194 active tokens, comparable to the base's 149, with no collapse. However, matching the base's token count is not equivalent to matching its quality, and v3a was the lowest-performing model of the three, with Recall@100 of 0.326 (p ≈ 2.5e-67). The sparsity was corrected; the relevance was not.

This was the key result. We had set every hyperparameter the papers recommended, observed the training loss converge and the token count reach its target — and the model was still worse. A token-level inspection of a real SEC passage explained why: the fine-tuned model's top-weighted query tokens were spurious numeric fragments (##22, 22nd, nu, valid, 21), displacing the financial concepts (operating, margin, cash, flow, basis, points) the base model had correctly weighted. Contrastive training on BM25 hard-negative triples had taught the model brittle lexical and numeric matching. The over-sparsification in v1 and v2 had inadvertently masked this — fewer tokens left less room for the spurious signal to appear.

The conclusion was clear: the regularizer was not the underlying problem. Token count was a symptom; the root cause was what the training data taught. Tuning sparsity only changed how much room that signal had to express itself. The parameter that mattered was the training objective itself — which is what led us to knowledge distillation.

The solution: knowledge distillation from a cross-encoder teacher

If the problem is that BM25 triples teach lexical overlap, the fix is a teacher that judges semantic relevance — whether a passage actually answers a query — and trains the student to imitate that judgment instead of the surface form of the negatives.

We use a cross-encoder as the teacher: cross-encoder/ms-marco-MiniLM-L-6-v2, the canonical SPLADE++/SPLADE-v3 teacher, light enough to score the ~150,000 (query, passage) pairs in our training set on one GPU in minutes. A cross-encoder reads the query and passage together and emits a single relevance score, so it captures interactions a bag-of-terms model cannot. We score all pairs offline, once, then distill into the sparse student during training.

One design decision warrants emphasis up front, because the incorrect choice produced a diverged run. There are two standard distillation losses for retrieval:

KL divergence asks the student to rank the passages the way the teacher does. It works on softmax(scores / T) — only the relative order of the scores matters, not their absolute size.
MarginMSE asks the student to reproduce the teacher's absolute score gap between the positive and each negative — for example, "the positive should out-score this negative by 0.6."

The two losses interact very differently with the sparsity regularizer, and that — not arithmetic — is the underlying issue. Our first KD attempt added MarginMSE (following SPLADE-v3's recipe) and diverged. The mechanism: a SPLADE relevance score is the dot product of two sparse vectors, so the only way to make a score larger is to place more weight on more tokens. When MarginMSE required specific absolute gaps, the student inflated its token weights to reach them — driving the average active-dimension count from ~540 to ~2,820 and working against the FLOPS regularizer (whose purpose is to shrink those weights) throughout. The two objectives optimize in opposite directions.

This might appear to be a units problem — raw cross-encoder margins (±11) are on a different scale than raw SPLADE dot products (in the tens) — and we did test the obvious fix, re-scaling the teacher margins by 30 to match OpenSearch's KD score_scale. It slowed the divergence but did not stop it, which isolates the cause: the conflict is not the scale constant, but that MarginMSE constrains absolute magnitudes at all, on a model whose magnitudes are deliberately controlled by the regularizer.

KL does not have this problem because it never references absolute magnitude — re-scaling all of a query's scores leaves their softmax unchanged, so KL does not require the student to inflate its weights, only to order the passages correctly. This makes KL-only the appropriate choice for an inference-free SPLADE model with FLOPS sparsity control; MarginMSE is a dense-retriever distillation technique, applicable where there is no sparsity penalty to conflict with, and is not suitable here.

The remaining components of the recipe are also necessary: KD supplies the correct signal, but we retain the v3a sparsity controls (doc-only FLOPS, quadratic warmup, ℓ0-mask, LR schedule) so the result remains sparse and deployable. The resulting model produces 36 active tokens per passage. Both are required — a sound objective and controlled sparsity. Each earlier run had only one of the two.

Prerequisites

To reproduce the walkthrough, you need:

An Amazon OpenSearch Service domain on engine 3.5 with the ml-commons plugin and fine-grained access control enabled.
Two Amazon S3 buckets (datasets and model artifacts) and a SageMaker execution role with AmazonSageMakerFullAccess.
The training dataset: oneryalcin/financial-filings-sparse-retrieval-training from the Hugging Face Hub — 18,247 training triples and 2,028 held-out test queries drawn from SEC filings and earnings transcripts, already in (query, positive, negatives[]) form (no synthetic generation needed).
Python 3.12 with torch, transformers, and sentence-transformers.

Prepare the dataset and score it with the cross-encoder teacher

The dataset ships triples but no relevance scores, so the first step is to add them. The teacher scorer reads each {query, pos, negs[]} row, scores every (query, doc) pair, and writes a new JSONL with a scores list aligned to [pos, neg_1, ...].

The key detail is per-query min-max normalization to [0, 1]. This places every query's targets on a stable, bounded scale (matching OpenSearch's KD score_scale), which keeps the KL term well-behaved and would also keep an optional MarginMSE term within a comparable range rather than diverging:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# ml/training/teacher_scores.py
from sentence_transformers import CrossEncoder

def score_triples(triples_path, out_path,
                  teacher="cross-encoder/ms-marco-MiniLM-L-6-v2", batch_size=256):
    model = CrossEncoder(teacher, max_length=512)
    # ... read rows ...
    for i in range(0, len(rows), 64):
        chunk = rows[i:i + 64]
        pairs, spans = [], []
        for r in chunk:
            docs = [r["pos"], *r["negs"]]
            spans.append((len(pairs), len(docs)))
            pairs.extend([[r["query"], d] for d in docs])
        scores = model.predict(pairs, batch_size=batch_size)   # flatten + score, scatter back
        for r, (start, ln) in zip(chunk, spans):
            raw = [float(s) for s in scores[start:start + ln]]
            lo, hi = min(raw), max(raw)
            # per-query min-max -> [0, 1]; scores[0] is the positive, scores[1:] the negatives
            r["scores"] = [(s - lo) / (hi - lo) if hi > lo else 0.5 for s in raw]
            out.write(json.dumps(r) + "\n")

This step runs inside the SageMaker training job. It is the only preprocessing the KD run adds over the InfoNCE runs.

Configure the training job

The entire experiment is driven from one config file. Here are the training hyperparameters for the winning knowledge-distillation run, verbatim from config/experiment.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
training:
  loss: kd                  # knowledge distillation (vs. infonce)
  kd_alpha: 1.0             # 1.0 => KL-only — the winner. (alpha<1 adds MarginMSE, which diverged.)
  kd_temp: 2.0              # KL softmax temperature
  kd_margin_scale: 30.0     # only used when alpha<1
  teacher: cross-encoder/ms-marco-MiniLM-L-6-v2

  epochs: 1                 # 1 pass over the ~18.3k train triples (~4,562 steps at batch 4)
  batch_size: 4             # batch 16 OOM'd the 22 GB A10G at seq 512 with 7 negatives
  lr: 2.0e-5
  max_seq_len: 512

  # --- sparsity recipe carried over from v3a ---
  flops_lambda: 0.01        # document-side regularization weight (never the query)
  flops_warmup_T: 300       # quadratic ramp: lambda * min(1, step/T)^2
  flops_l0_threshold: 200   # ℓ0-mask: docs already <=200 active dims escape the penalty
  warmup_ratio: 0.1         # linear LR warmup (10% of steps) then linear decay
  weight_decay: 0.01

  instance_type: ml.g5.xlarge

A few notes on these choices:

kd_alpha: 1.0 (KL-only) is the single most important setting, for the scale-compatibility reason described above.
negs_per_query: 7 (set in the triples section) follows the dataset's BM25 hard-negative mining. Each training step therefore encodes (1 query + 1 positive + 7 negatives) × 4 sequences of up to 512 tokens — which is what forces the small batch size on a single A10G.
flops_warmup_T: 300 lets the contrastive/KD signal organize the vectors for the first ~6.6% of the epoch before the sparsity pressure ramps to full strength, so the model never collapses to zero.

The student computes its own relevance scores as SPLADE sparse-vector dot products over the same [positive, negatives] set, and the KD loss matches them to the teacher's distribution. The total loss adds the document-only, ℓ0-masked, warmed-up FLOPS regularizer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# ml/training/finetune.py
def kd_loss(student, teacher, alpha=1.0, kl_temp=2.0, margin_scale=30.0):
    t_soft    = F.softmax(teacher / kl_temp, dim=1)
    s_logsoft = F.log_softmax(student / kl_temp, dim=1)
    kl = F.kl_div(s_logsoft, t_soft, reduction="batchmean") * (kl_temp ** 2)
    if alpha >= 1.0:
        return kl, kl, kl.new_zeros(())          # KL-only (default, scale-safe)
    # MarginMSE branch (alpha<1) — scale-sensitive; diverged for our model:
    s_margin = student[:, :1] - student[:, 1:]
    t_margin = (teacher[:, :1] - teacher[:, 1:]) * margin_scale
    return alpha * kl + (1.0 - alpha) * F.mse_loss(s_margin, t_margin), kl, ...

# in the training loop, per batch:
if loss == "kd" and "teacher" in batch:
    s_scores = student_scores(q_vec, pos_vec, neg_vec)        # [batch, 1 + n_negs]
    loss_rank, kl_t, mm_t = kd_loss(s_scores, teacher,
                                    alpha=kd_alpha, kl_temp=kd_temp, margin_scale=kd_margin_scale)
loss_reg   = flops_regularizer(pos_vec, neg_vec, l0_threshold=flops_l0_threshold)  # DOCS only
eff_lambda = lambda_at(step, flops_lambda, flops_warmup_T)    # quadratic warmup
loss_total = loss_rank + eff_lambda * loss_reg

During training we log the average active-dimension count per document every 20 steps, with a collapse alarm under 80. On a healthy KL-only run the KL term sits around 0.02 and the document length holds around 150 — you can watch the collapse not happen.

Launch the fine-tuning job on SageMaker AI

SageMaker SDK v3 dropped the framework-specific HuggingFace estimator, so we use the generic Estimator with an explicit HuggingFace Deep Learning Container image. The dataset triples and the pooled corpus (for IDF recomputation) arrive as two input channels:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# ml/training/launch_sagemaker_job.py
from sagemaker.estimator import Estimator
from sagemaker.inputs import TrainingInput

estimator = Estimator(
    image_uri="<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/"
              "huggingface-pytorch-training:2.8.0-transformers4.56.2-gpu-py312-cu129-ubuntu22.04-v1.2",
    entry_point="sm_entrypoint.py",
    source_dir="ml/training",
    role=execution_role,
    instance_type="ml.g5.xlarge",   # 1x NVIDIA A10G, 24 GB VRAM, 4 vCPU, 16 GiB RAM
    instance_count=1,
    hyperparameters={
        "base-model": "opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill",
        "loss": "kd", "kd-alpha": 1.0, "kd-temp": 2.0,
        "teacher": "cross-encoder/ms-marco-MiniLM-L-6-v2",
        "epochs": 1, "batch-size": 4, "lr": 2e-5, "max-seq-len": 512, "negs-per-query": 7,
        "flops-lambda": 0.01, "flops-warmup-t": 300, "flops-l0-threshold": 200,
        "warmup-ratio": 0.1, "weight-decay": 0.01,
    },
    output_path=f"s3://{artifacts_bucket}/models/finetuned",
)
estimator.fit(inputs={"triples": TrainingInput(triples_s3),
                      "corpus":  TrainingInput(corpus_s3)}, wait=False)

We ran the job on a single ml.g5.xlarge instance — one NVIDIA A10G GPU (24 GB VRAM, ~22 GB usable), 4 vCPUs, 16 GiB of RAM. This is deliberately modest: a 67M-parameter encoder, one epoch over ~18,000 triples, fits comfortably and completes in well under an hour. The one accommodation the single GPU forces is the small batch size. A SPLADE encoder projects every token over the full ~30,522-term vocabulary, so each example's nine 512-token sequences (query + positive + 7 negatives) produce large [batch, seq, vocab] logits tensors — roughly 40× the size a dense [batch, seq, 768] encoder would emit. At batch 16 those tensors, plus the activations retained for backprop, exceed the A10G's ~22 GB, so we use batch 4 and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation OOMs. No multi-GPU, no distributed training, no FSDP is needed at this model size.

Package and deploy to a SageMaker AI real-time endpoint

SageMaker doesn't auto-extract a named training channel's model.tar.gz, and our serving contract is specific: the ml-commons connector invokes the endpoint with no post-processing function, so the endpoint must return the sparse {token: weight} map directly. We bundle a custom inference handler into the model archive at code/inference.py:

1
2
3
4
5
6
7
8
# ml/inference/inference.py — the SPLADE pooling the endpoint serves
def _splade_max_pool(logits, attention_mask):
    # v3 double-log activation — MUST match training, or serving silently diverges
    activated = torch.log1p(torch.log1p(torch.relu(logits)))   # log(1 + log(1 + ReLU(x)))
    masked = activated * attention_mask.unsqueeze(-1)
    pooled, _ = masked.max(dim=1)                              # max-pool over the sequence
    return pooled
# predict_fn then maps non-zero, non-special vocab ids -> {wordpiece_token: rounded_weight}

package_model.py builds the model.tar.gz (model files at the root, handler under code/), and the CDK SageMakerStack stands up the endpoint from it with the HuggingFace inference DLC:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// infra/lib/sagemaker-stack.ts (abridged)
const model = new sagemaker.CfnModel(this, `${label}Model`, {
  executionRoleArn: props.executionRole.roleArn,
  primaryContainer: {
    image: props.cfg.infra.hf_inference_image,   // huggingface-pytorch-inference:2.6.0-...-gpu-py312
    modelDataUrl: props.artifactsBucket.s3UrlForObject(s3Key),
    environment: { SAGEMAKER_PROGRAM: 'inference.py',
                   SAGEMAKER_SUBMIT_DIRECTORY: '/opt/ml/model/code' },
  },
});
new sagemaker.CfnEndpointConfig(this, `${label}EndpointConfig`, {
  productionVariants: [{
    modelName: model.attrModelName, variantName: 'AllTraffic',
    initialInstanceCount: 1,
    instanceType: props.cfg.infra.sagemaker_inference_instance,   // ml.g5.xlarge
  }],
});

We deploy two endpoints — financial-sparse-base and financial-sparse-finetuned — so the A/B test is apples-to-apples.

Inference hardware. Each endpoint runs on ml.g5.xlarge (the same A10G GPU as training), one instance per endpoint. SPLADE inference is roughly 10–50× faster on this GPU than on a comparable CPU instance, which matters during bulk indexing. Because GPU endpoints are the dominant recurring cost in this PoC, we delete them when idle and recreate them per evaluation; if you keep them running 24/7, expect the endpoints and the OpenSearch domain to dominate the bill.

Wire the endpoint into Amazon OpenSearch Service

ml-commons connector and model registration are SigV4-signed REST calls against the domain's _plugins/_ml API. We provisioned our stack using CDK that creates a Python Lambda custom resource, doing the following:

Maps the execution role to ml_full_access.
Creates two remote connectors (base + fine-tuned) pointing at the SageMaker endpoints, using the sparse blueprint with no post_process_function (the endpoint already returns the sparse map).
Registers and deploys two remote models and captures their model_ids.
Creates two sparse_encoding ingest pipelines and two rank_features indices.
Publishes both model_ids to SSM Parameter Store so the evaluation tooling resolves them at runtime.

The connector's request body is simply {"inputs": ${parameters.input}}, and the response flows straight into the rank_features field that the neural_sparse query searches.

Evaluate: A/B test on held-out and human-verified data

The evaluation harness runs the same query set against both arm indices, computes NDCG@10, Recall@100, and MRR with pytrec_eval, and reports a paired t-test — only the model differs between runs.

One production detail surfaced here and is worth flagging, because it is easy to miss until a query fails. The fine-tuned KD model expands queries richly — sometimes past 1,024 tokens — and OpenSearch turns each query token into a boolean clause, exceeding the managed domain's fixed indices.query.bool.max_clause_count = 1024 (which Amazon OpenSearch Service does not let you raise). The fix is to encode the query client-side on the SageMaker endpoint and prune to the top 1,000 tokens before searching, applied identically to both arms so the comparison stays fair:

1
2
3
4
5
# ml/eval/run_ab.py
vec = json.loads(rt.invoke_endpoint(EndpointName=endpoint, ContentType="application/json",
                                    Body=json.dumps({"inputs": [query_text]}))["Body"].read())[0]
top = dict(sorted(vec.items(), key=lambda kv: kv[1], reverse=True)[:max_query_tokens])  # top-1000
body = {"size": size, "query": {"neural_sparse": {field: {"query_tokens": top}}}}

Results

On the held-out synthetic test set (2,028 queries, ~80,000-passage pooled corpus), knowledge distillation outperformed the off-the-shelf base on every metric — the first run to do so, after three regressions:

Reading the p-values. Each p comes from a paired t-test over the per-query metric (the same query scored by both models), testing whether the average difference is real or just noise. Smaller is stronger; anything below 0.05 is conventionally "statistically significant." Values like p ≈ 4e-55 indicate the difference is very unlikely to be due to chance — expected at 2,028 queries. They indicate the difference is real, not that it is large; read them alongside the effect sizes (the Δ column), not in place of them.

Metric	Base	v3a (best of the failed runs)	KD (KL-only)	Δ vs base
NDCG@10	0.261	0.192	0.368	+41% (p ≈ 4e-55)
Recall@100	0.540	0.326	0.617	+14% (p ≈ 4e-18)
MRR	0.236	0.182	0.342	+45% (p ≈ 2e-52)

KD improved 508 queries and degraded 147. A token-level inspection confirmed the mechanism: the KD model now associates figures with concepts — top tokens free, operating, margin, cash, flow, fiscal, 2021 — in a compact, deployable 36-token vector, rather than the spurious fragments v3a learned.

Validation on human-verified data

An improvement on a synthetically generated test set raises a reasonable question — is the model genuinely better, or has it learned the style of the synthetic queries? To determine this, we validated on FinanceBench : 113 human-verified, analytical questions (for example, "Is 3M a capital-intensive business based on its FY2022 data?"). The result was instructive:

Metric	Base	KD	Δ (n=113)
Recall@100	0.389	0.575	+0.186 (p ≈ 0.0008)
NDCG@10	0.193	0.193	tie (p ≈ 0.99)
MRR	0.171	0.170	tie (p ≈ 0.99)

On real, out-of-distribution questions, KD's recall advantage is substantial and consistent — it retrieves considerably more of the relevant filings into the candidate set, which is precisely a sparse retriever's role when it feeds a downstream reranker. The +41% NDCG, however, reduced to a tie.

We did not run an ablation to pin down why the ranking gain didn't transfer, so we offer it as a hypothesis rather than a conclusion. The synthetic test queries were generated by the same process as the training queries, so several effects could be in play and we can't separate them: the synthetic queries may simply be more lexically matchable (easier) than human analyst questions; the model may have learned patterns specific to the synthetic query style; or the synthetic train and test sets may share vocabulary the human questions don't. What we can say with confidence, because it holds on the independent human-verified set, is that the durable, deployable improvement is in recall — and for a candidate-generation stage feeding a reranker, recall is the metric that matters most. Treat any ranking-quality gain measured only on synthetic queries as unconfirmed until it is verified on real ones.

Clean up

To avoid ongoing charges, delete the two SageMaker endpoints (and their endpoint configs and models), delete or scale down the OpenSearch Service domain, and empty the S3 buckets. If you provisioned with the CDK, cdk destroy removes the SageMaker, OpenSearch, and connector stacks; the S3 buckets are configured to auto-delete their objects on stack removal for this PoC.

Conclusion

Fine-tuning a neural sparse retriever for a specialized domain is effective — but the objective is the parameter that matters, not the regularizer. We spent three runs tuning FLOPS sparsity and the model regressed progressively; the improvement came from changing what the model learns. Specifically:

Contrastive training on BM25 hard-negative triples can degrade quality, because the negatives teach lexical and numeric matching rather than relevance. A cross-encoder teacher (knowledge distillation) corrects the training signal.
Use KL-only distillation for sparse models. MarginMSE's absolute-magnitude targets conflict with the FLOPS sparsity objective and lead to weight-inflation divergence.
Both a sound objective and controlled sparsity are required. KD supplied the signal; the doc-only FLOPS + ℓ0-mask + warmup recipe kept the result at a deployable 36 tokens per passage.
Validate on an independent, human-verified set. Our synthetic results overstated the ranking-quality (NDCG@10) gain; the human-verified set showed that the durable improvement is in recall.

In summary: knowledge distillation turns a strong off-the-shelf sparse encoder into a meaningfully better domain retriever for financial filings — a clear, significant improvement for candidate generation (recall), validated on human-labeled questions, achieved on a single GPU in under an hour. The base model is a strong baseline; improving on it required the right objective, not additional regularization.

Builder Center