Building a domain-expert chatbot for $2.03
Building a domain-expert chatbot and what I learned about model selection when every hour counts.
The Challenge
The Build Small Hackathon has a compelling premise: in a world obsessed with bigger and bigger models, go the other way. The rules are simple: keep total parameters under 32 billion, ship a Gradio app on Hugging Face Spaces, record a demo. The constraint forces you to be creative with fine-tuning, prompting, or multi-model pipelines rather than brute-forcing with scale.
I set out to build Tech Advisor, a chatbot fine-tuned on AWS DevOps Agent documentation that runs entirely off the grid. No cloud APIs at inference. No vector database. No RAG pipeline. Just 66 pages of documentation baked directly into a small model's weights via QLoRA fine-tuning.
The goal: prove that a model small enough to run on consumer hardware can serve as a legitimate domain expert, for a one-time cost of under $5.
Why Fine-Tuning Instead of RAG?
Before I get to model selection, this is the first question people ask. With 66 pages of documentation, why not just use RAG (retrieval-augmented generation) with a base model?
Because the hackathon's spirit is "off the grid": no external dependencies at inference. RAG needs a vector database, an embedding model, and a retrieval step on every query. Fine-tuning compresses the knowledge into the model itself. Once trained, the model is self-contained: download it, load it, ask questions. No infrastructure, no network, no ongoing costs.
The trade-off is clear: RAG handles updates better (just re-index), fine-tuning handles deployment better (just load the model). For a hackathon demo where the docs won't change, fine-tuning wins.
The Starting Point: Why I Needed to Fine-Tune at All
Here's what happens when you ask a base model (NVIDIA Nemotron Nano 12B v2 VL, no fine-tuning) about AWS DevOps Agent:
Q: Tell me about the AWS DevOps Agent"The AWS DevOps agent, also known as the AWS Systems Manager Agent (SSM Agent), is a software component that enables you to manage and automate the configuration of your EC2 instances..."
Completely wrong. AWS DevOps Agent is a new AI-powered service for autonomous incident response and proactive prevention. It has nothing to do with SSM Agent. The base model confidently hallucinated because it has no training data about this service (it launched after the model's knowledge cutoff).
This confirmed the need for fine-tuning: the model needs to learn domain knowledge it simply doesn't have.
The Two Candidates
NVIDIA's hackathon field guide recommended Nemotron 3 Nano 4B as "edge-ready reasoning in a tiny footprint." But I also found Llama-3.1-Nemotron-Nano-4B-v1.1: same brand, same size class, same publisher.
On paper, they look interchangeable. Under the hood, they could not be more different.
| Nemotron 3 Nano 4B | Llama-3.1-Nemotron-Nano-4B-v1.1 | |
|---|---|---|
| Architecture | Hybrid Mamba-Transformer (NVIDIA's own design) | Pure Transformer (Meta's Llama 3.1) |
| What NVIDIA did | Built from scratch with novel SSM layers | Took Llama 3.1, applied post-training (distillation + alignment) |
| Custom CUDA kernels needed | Yes (mamba-ssm, causal-conv1d) | No |
| Install method | Compiled from source against your CUDA/GPU | Standard pip install |
| Setup time | 30-60 minutes (if it works first try) | 2 minutes |
What Went Wrong With Nemotron 3 Nano
I started with Nemotron 3 Nano. It's NVIDIA's own architecture, so it felt like the natural choice for an NVIDIA-sponsored hackathon. Within minutes of trying to set it up on my EC2 training instance, I was in dependency trouble.
The Mamba Compilation Problem
Nemotron 3 Nano uses a hybrid Mamba-Transformer architecture. Mamba is a state-space model, a fundamentally different approach to sequence modeling than standard attention. It introduces a "selective scan" operation that doesn't exist in any standard deep learning library.
Standard Transformer operations (matrix multiplications, softmax, layer normalization) are all pre-compiled in PyTorch via cuBLAS and cuDNN. These libraries have been optimized over a decade. When you
pip install torch, you get all of that for free.Mamba's selective scan is a novel recurrence. Nobody had a pre-compiled kernel for it before the Mamba authors wrote one. So when you install
mamba-ssm, it compiles custom CUDA kernels from source on your machine.Here's what the Docker setup required:
1
2
3
4
5
6
FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel # Must be "devel", not "runtime"
RUN pip install --no-cache-dir packaging ninja
ENV TORCH_CUDA_ARCH_LIST="8.6" # Must match your exact GPU
RUN pip install --no-cache-dir --no-build-isolation causal-conv1d
RUN pip install --no-cache-dir --no-build-isolation mamba-ssmEvery line exists to solve a specific compilation problem:
devel image instead of runtime: The runtime Docker image (~4GB) includes enough to run pre-compiled CUDA code. But compiling new kernels requires nvcc (NVIDIA's CUDA compiler), development headers, and linker libraries, which only exist in the devel image (~8-12GB). That's double the container size just to install a dependency.TORCH_CUDA_ARCH_LIST="8.6": The CUDA compiler generates machine code for specific GPU compute capabilities. My A10G is 8.6. An A100 is 8.0. A V100 is 7.0. If you omit this flag, the compiler either builds for ALL architectures (20+ minutes of compilation) or picks a default that doesn't match your GPU, giving you a runtime crash: CUDA error: no kernel image is available for execution on the device.--no-build-isolation: PEP 517 normally builds packages in an isolated virtual environment. But mamba-ssm needs to call torch.utils.cpp_extension during installation, linking against your exact PyTorch and CUDA libraries. With isolation on, it can't find them. Build fails with cryptic errors.causal-conv1d: Mamba uses a specialized causal 1D convolution before its state-space layer. A standard torch.nn.Conv1d would work functionally but is slow for Mamba's specific pattern (causal masking, depthwise, short kernel). So there's a separate custom CUDA kernel that also compiles from source.In a hackathon where every hour counts, spending 30-60 minutes on infrastructure before even starting to train is a significant cost. And if something doesn't compile cleanly (version mismatch between CUDA toolkit and PyTorch, wrong architecture flag, missing header file) you can lose hours debugging.
The Switch to Llama-3.1-Nemotron-Nano
After the compilation friction, I pivoted to Llama-3.1-Nemotron-Nano-4B-v1.1. Same brand, same size, same publisher. Completely different experience.
1
pip install torch transformers peft bitsandbytes trl datasets accelerateDone. No compilation, no architecture flags, no devel images. Works on any machine with a GPU and a recent PyTorch install. I went from "fighting infrastructure" to "training the model" in under 2 minutes.
What's the Relationship Between These Models?
Both carry the "Nemotron" brand, but they represent different strategies:
- Nemotron 3 Nano = NVIDIA built their own architecture from scratch using Mamba (a novel alternative to attention). This is NVIDIA's research bet on state-space models.
- Llama-3.1-Nemotron-Nano = NVIDIA took Meta's open-weight Llama 3.1 model and applied their post-training pipeline: knowledge distillation from larger Nemotron models, RLHF/DPO alignment, and curated data. The architecture is 100% Meta's Llama. NVIDIA made it smarter through training.
The analogy: Nemotron 3 Nano is a car NVIDIA built from scratch with a novel engine. Llama-3.1-Nemotron-Nano is a battle-tested Toyota (Llama 3.1) that NVIDIA tuned, upgraded, and rebranded. The "rebranded Toyota" turned out to be the better daily driver.
The Training Pipeline (35 Minutes, $2.03 Total)
With the model selected, the actual fine-tuning was straightforward. The entire training script is about 100 lines of Python:
- Data preparation: 66 markdown files from AWS docs, automatically split into sections and converted to 1,230 instruction-response pairs
- Load model in 4-bit: BitsAndBytes compresses the model from ~9GB to ~3GB so it fits in GPU memory alongside gradients
- Freeze all 4.5 billion parameters: the base model stays untouched
- Attach LoRA adapters: small trainable matrices injected into the attention layers (q, k, v, o projections). Only 11.5 million trainable parameters, or 0.25% of the model
- Train: 3 epochs, loss dropped from 2.0 to 0.43
| Item | Duration | Cost |
|---|---|---|
| EC2 g5.xlarge (A10G, 24GB VRAM) | ~2 hours total (setup + training) | ~$2.02 |
| Actual training time | 35 minutes | ~$0.59 |
| EBS storage | 2 hours | ~$0.01 |
| Total | $2.03 |
After training, I merged the adapter back into the base model and pushed to Hugging Face Hub. The model now runs on ZeroGPU (free tier), where a GPU is allocated on-demand only when a user sends a message, then released. No dedicated GPU running 24/7.
A Nemotron-Specific Feature: /no_think
One advantage of the Nemotron family: the
/no_think system prompt prefix disables chain-of-thought reasoning tokens. Without it, the model would generate internal reasoning before answering (useful for complex problems, wasteful for factual Q&A). With it, responses are clean and fast. Just the answer, no preamble.The Economics Argument
This is what made the hackathon theme click for me:
| Tech Advisor (local) | GPT-4 API | Claude API | |
|---|---|---|---|
| Setup cost | $2.03 (one-time) | $0 | $0 |
| Per-query cost | $0 | $0.03-0.12 | $0.015-0.075 |
| After 100 queries | $2.03 total | $3-12 | $1.50-7.50 |
| After 1,000 queries | $2.03 total | $30-120 | $15-75 |
| Data privacy | 100% local | Sent to provider | Sent to provider |
| Internet required | No | Yes | Yes |
| Rate limits | None | Yes | Yes |
For a focused domain task (one service's documentation, one area of expertise) the economics of a small fine-tuned model are compelling. Two dollars. Once. Forever.
Lessons Learned
1. "Same brand" doesn't mean "same architecture"
Nemotron is a brand name spanning multiple generations and architectures. Always look under the hood. Check whether a model uses standard Transformer ops (pip install and go) or novel architectures (compilation required).
2. Novel architectures carry ecosystem tax
Mamba is genuinely interesting research. State-space models offer theoretical advantages for long sequences. But in a time-constrained project, "works with pip install" beats "theoretically elegant" every time. The ecosystem around standard Transformers is vastly more mature.
3. Post-training can matter more than architecture
A standard architecture with great post-training (distillation from larger models, RLHF alignment, curated data) can outperform a novel architecture with less post-training investment. The Llama-3.1 base is so strong that NVIDIA's refinements on top make it competitive without any of the Mamba operational complexity.
4. Small models are viable domain experts
A 4B-parameter model, fine-tuned for 35 minutes on targeted documentation, produces useful answers about that domain. You don't need 70B+ parameters or complex RAG pipelines for focused knowledge tasks. The base model hallucinated about AWS DevOps Agent; the fine-tuned version answers accurately.
5. The "off the grid" future is real for narrow tasks
The hackathon's tagline was "the future of AI doesn't have to live in someone else's data center." For broad general-purpose AI, large cloud models still win. But for narrow, domain-specific applications (a support bot for one product, an internal knowledge assistant for one team) small local models are already good enough, and the deployment story is incomparably simpler.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article