AWS Builder Center

SLM Finetuning for Indian financial news sentiment analysis

In this blog we discuss LoRA, QLoRA and full fine-tuning (FT) methods for a SLM on a single 24GB GPU to classify the Indian financial news sentiment. During the process we discuss the benchmarking results and the parameter efficient fine tuning (PEFT) methods during the process

SLM Finetuning for Indian financial news sentiment analysis

In this blog we discuss LoRA, QLoRA and full fine-tuning (FT) methods for a SLM on a single 24GB GPU to classify the Indian financial news sentiment. During the process we discuss the benchmarking results and the parameter efficient fine tuning (PEFT) methods during the process.
We took Qwen2.5-1.5B-Instruct SLM to fine tune on Indian financial news sentiment domain task on  g5.xlarge  (a single NVIDIA A10G with 24 GB). Benchmark was done on following configurations -
  • Base (few-shot) — no training that served as the baseline.
  • LoRA — train small low-rank adapters and freeze the rest.
  • QLoRA — LoRA on top of a 4-bit-quantized base.
  • Full fine-tuning — update all 1.54 billion parameters.
With same data, same prompt given below is the benchmark results.

Benchmark results

Technique Accuracy Macro-F1 Trainable params Peak VRAM Train time Artifact
-------------------- -------- -------- ---------------- --------- ---------- --------
Base (1-shot/class) 0.617 0.617 0 — — —
LoRA (r=16) 0.765 0.767 18.5 M (1.18%) 14.0 GB ~52 min 86 MB
QLoRA (4-bit NF4) 0.755 0.759 18.5 M 13.6 GB ~66 min 86 MB
Full fine-tune 0.780 0.783 1.54 B (100%) 11.3 GB* ~44 min 2,960 MB
Key result — LoRA reached ~98% of full fine-tuning's accuracy (0.765 vs 0.780) while training 1.18% of the parameters and producing an 86 MB adapter instead of a 2,960 MB checkpoint.

Indian Financial Market sentiment dataset

General sentiment models learn that "plunge," "saddled," and "miss" are negative and "beats," "surges," and "record" are positive. Financial news involves much more nuances that involves context. Few examples below -
  • "Rupee plunges to record low" — negative for the rupee, but the overall context matters.
  • "NPA-saddled banks" — domain jargon a general model has never weighted.
  • "Q3 profit beats estimates; raises guidance" — textbook positive but buried in numbers.
  • "RBI keeps repo rate unchanged, in line with expectations" — genuinely neutral, yet a base model would either classify it as positive or negative.
We used  kdave/Indian_Financial_News  that had 26,961 articles scraped from Indian outlets (MoneyControl, Economic Times, Financial Express). The data dataset covers RBI policy, the Sensex/Nifty, the rupee, NPAs, and LTCG. The use case was designed as classification of the sentiment into one of three classes: Negative / Neutral / Positive. Three classes give us an objective, automatable metric (accuracy + macro-F1).

End to end architecture

The whole point of the experiment is fairness, so the engineering is mostly about removing confounds.

Figure 1. End-to-end system architecture. A control-plane workstation provisions and orchestrates a single AWS g5.xlarge (A10G, 24 GB).
We discuss below four key patterns that are used in training and testing.

Pattern 1 — Instruction framing with the chat template

We frame classification as instruction following. Qwen2.5-1.5B is an instruction-tuned model.
<|im_start|>system
You are a financial sentiment classifier for Indian market news. Classify ... as exactly
one of: Negative, Neutral, Positive. Respond with only the single label word.<|im_end|>
<|im_start|>user
Classify the sentiment of the following Indian financial news.

News:
{article truncated to 1500 chars}

Sentiment (Negative, Neutral, or Positive):<|im_end|>
<|im_start|>assistant
Positive<|im_end|>
The base model gets one labeled example per class (few-shot) for baseline. The fine-tuned models are evaluated zero-shot — we don't need examples for fine-tuned models.

Pattern 2 — Completion-only label masking

If you train on the full sequence, the model spends its capacity learning to reproduce the article and not to emit the label. So we compute loss only on the label tokens and every prompt token is set to the ignore index  -100 :
tokens: [ system … user … "assistant\n" ] [ "Positive" <|im_end|> \n ]
labels: [ -100 -100 -100 … -100 ] [ real token ids ]
^ loss ignored ^ loss here only
A smoke test confirmed exactly 3 unmasked tokens per example. This pattern was used in three training runs (LoRA, QLoRA, full FT). The pattern was applied during tokenization before the Trainer sees the data.

Pattern 3 — A balanced test set and deterministic splits

The dataset ships a single unsplit  train , so we built our own with a fixed seed (42), stratified by class:
  • Test = 1,998 (666 per class) that was balanced by design so accuracy and macro-F1 are directly comparable and "predict the majority class" won't become a skill.
  • Validation = 498, Train = 8,000 (capped so the full training finishes in ~2.5 GPU-hours).
This pattern was used for same test set for scoring everyone, same train/validation rows for the three trained methods.

Pattern 4 — A paged 8-bit optimizer so full FT even fits

Full fine-tuning of 1.5 B parameters is dominated by optimizer state (Adam keeps two moments per weight). We use  paged_adamw_8bit  (bitsandbytes), which keeps optimizer state in 8-bit and pages it to host RAM. That is the specific trick that lets all 1.5 B parameters train inside 24 GB at all.

Figure 4. Training pipeline. A single code path drives all three methods: load the stratified splits, tokenize with label masking, then load_model_for_training(method) attaches LoRA, prepares the 4-bit base, or leaves the full model in bf16. The Hugging Face Trainer (paged 8-bit optimizer, bf16, gradient checkpointing, effective batch 16, 2 epochs) writes the artifact and a per-method metrics JSON.
This pattern was used for full fine tuning.

A brief about LoRA, QLoRA and full fine tuning

LoRA freezes the pretrained weight W and learns a low-rank update ΔW = B·A (rank r=16): h = Wx + (α/r)·B(Ax). Only the adapters train (18.46 M parameters (1.18%)), so the optimizer state is tiny and the saved artifact is just an 86 MB adapter. We adapt all seven projections of each decoder block (q,k,v,o attention + gate,up,down MLP), covering both how the model routes information and how it transforms features.

Figure 5. LoRA mechanism. The pretrained weight W stays frozen (gray) while a trainable low-rank detour B·A (rank r=16, scaled by α/r) is added to the output: h = Wx + (α/r)·B(Ax). Only A and B train — 18.46 M parameters (1.18%) — so the optimizer state is tiny and the artifact is an 86 MB adapter.
QLoRA does the same LoRA, but the frozen base is loaded in 4-bit NF4 (a "normal float" format matched to the roughly-Gaussian weight distribution), with double quantization to compress the quantization constants. It has the lowest memory footprint of any method here.

Figure 6. QLoRA flow. The ~3 GB bf16 base is quantized to a ~0.9 GB frozen 4-bit NF4 copy (double-quantized), dequantized on the fly to bf16 for each matmul, with the trainable LoRA adapters added on top. It has the lowest memory footprint of any method here (peak 13.6 GB) at the cost of about one accuracy point.
Full fine-tuning updates everything.

The neutral class gives the gains

Every model above is scored the same way. Given below is the evaluation pipeline.

Figure 7. Evaluation pipeline. The balanced 1,998-row test set is rendered into the chat prompt (zero-shot for fine-tuned models, 1-shot for the base), left-padded into batches of 32 and decoded greedily for at most 5 new tokens; a first-match parser maps the output to Negative/Neutral/Positive, and scikit-learn computes accuracy, macro-F1, per-class F1, and the confusion matrix.
The base model's real weakness is Neutral as it over-reads financial framing as directional. The confusion matrices tell the story.
Base model (1-shot), per-class F1: Negative 0.72 · Neutral 0.53 · Positive 0.60
Base confusion matrix Predicted
Neg Neu Pos
Actual Negative 498 138 30
Neutral 167 390 109 <- Neutral bleeds everywhere
Positive 51 271 344 <- Positive mistaken for Neutral
After LoRA, Neutral F1 climbs to 0.71 and Positive recall increases:
LoRA confusion matrix Predicted
Neg Neu Pos
Actual Negative 427 210 29
Neutral 23 545 98 <- Neutral recall fixed
Positive 3 106 557
Full fine-tuning squeezes out the best Negative recall and the top overall score (Neutral F1 0.73, accuracy 0.780). But notice how close LoRA and QLoRA already are — and at a fraction of the cost.
Technique Negative F1 Neutral F1 Positive F1
-------------- ----------- ---------- -----------
Base (1-shot) 0.721 0.532 0.599
LoRA 0.763 0.714 0.825
QLoRA 0.765 0.710 0.804
Full FT 0.801 0.725 0.822

The learning rate

The table given below provides various learning rate that is used.
Full FT learning rate Train loss Eval loss Accuracy
--------------------- --------------- --------- ----------------
1e-5 ~0.21 (plateau) 0.22 0.636 (underfit)
5e-5 0.18 0.21 0.780
At  1e-5 , full fine-tuning scored 0.636 that was underfit. When we increased it to  5e-5  did it converge and reach 0.780.
Meanwhile LoRA and QLoRA learned stably at  lr=2e-4  with no tuning at all.
Takeaway — PEFT methods are dramatically more robust to hyperparameters. Adapters inject a small set of high-learning-rate weights on top of a frozen, already-good model, so they converge fast and forgivingly. Full fine-tuning moves all the weight. If it is too low and it underfits in your epoch budget and if it is too high and it destabilizes or forgets its instruction-following. It demands a careful learning-rate search merely to match what LoRA gets for free.

When to choose what?

  • Reach for LoRA by default. Best quality-per-effort: ~98% of full FT accuracy, 1.18% of the parameters, an 86 MB adapter, and stable training with essentially no hyperparameter search.
  • Reach for QLoRA when memory is the binding constraint or when you want to fine-tune a larger base model on the same card. You trade about one accuracy point for the smallest footprint.
  • Reserve full fine-tuning for when the last 1–2 accuracy points genuinely matter and you can afford 80× the trainable parameters, 35× the artifact, and a learning-rate search.

Sample code

Examples

The notebook closes by loading the best adapter and classifying a few fresh headlines with the same prompt and parser used in training and evaluation:
  • "Sensex tanks 1,000 points as rupee hits record low against the dollar" → Negative
  • "Infosys beats estimates with strong Q3 profit; raises full-year guidance" → Positive
  • "RBI keeps repo rate unchanged, in line with market expectations" → Neutral
  • "NPA-saddled PSU banks see provisions rise sharply this quarter" → Negative
Note the third one: "rate unchanged, in line with expectations" is exactly the genuinely neutral framing the base model kept misreading as directional — and precisely where fine-tuning buys its biggest gain.

The bottom line

For a well-scoped domain task, parameter-efficient fine-tuning gets you almost all the way to full fine-tuning's quality for a fraction of the parameters, for a fraction of the storage, and a fraction of the hyperparameter heartache. We tested all 1.5 billion parameters on one 24 GB GPU and measured it.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article