I Built an SSH Honeypot on AWS: What Bots and Bugs Taught Me
I put a fake SSH server on the public internet and watched who came knocking. How I built a Cowrie honeypot on AWS with Terraform, Ansible and Grafana, and the bugs that only showed up in the cloud
Why I built a trap for hackers
I put a fake SSH server on the public internet and watched who came knocking. Within hours of going live, it was found by scanners from several continents.
I am a junior DevOps engineer and a cybersecurity student. At work I write Terraform and Terragrunt every day, but I wanted a project where I own every layer: the network, the server, the configuration, the monitoring and the CI. A honeypot is a great excuse for that. It has to be exposed to the internet on purpose, so every security decision around it actually matters.
The result is a production-style SSH honeypot on AWS, built entirely as code. A single
terraform apply followed by ansible-playbook takes a clean AWS account to a working trap with dashboards. Here is how it works and, more importantly, what went wrong along the way.How it works
The trap is Cowrie, a medium-interaction SSH honeypot. Attackers get a convincing fake Linux shell, but Cowrie never executes their commands; it only emulates them and logs everything. One event travels through the stack like this:
- A bot connects to port 22, which is published straight into the Cowrie container.
- Cowrie writes a JSON line for every event: connection, client version, login attempt with username and password, command, download attempt.
- Grafana Alloy reads the log from a shared Docker volume, parses each line and ships it to Loki.
- Loki stores 30 days of events, with retention enforced by its compactor.
- Grafana shows dashboards for login attempts over time, top usernames and passwords, commands and downloads.
Two small decisions made a big difference. First, only the event type became a Loki label. IP addresses and passwords have huge cardinality, so I extract them at query time instead of indexing them. Second, Alloy uses the timestamp from the Cowrie event, not the time it read the line, so the timeline stays correct even if the pipeline lags.
The real SSH daemon runs on a separate port that only my IP can reach. Grafana is bound to localhost and opened through an SSH tunnel, so none of the monitoring is exposed to the internet.
The AWS side
Everything on AWS is Terraform, and most of it is about limiting the blast radius of a server that is attacked on purpose.
State. Terraform state lives in a versioned, encrypted S3 bucket with public access blocked and native S3 state locking. The provider is pinned with
allowed_account_ids, so Terraform cannot touch another account even if other credentials are in my environment. I sign in through IAM Identity Center with short-lived credentials, not access keys.Network. A dedicated VPC with one public subnet and no NAT gateway, to keep costs low. The security group is the core of the design:
| Rule | Port | Source / destination | Why |
|---|---|---|---|
| Inbound trap | 22 | Anyone | This is the bait |
| Inbound admin SSH | Custom port | My IP only (/32) | Real access to the server |
| Outbound | 80, 443 | Anywhere | Malware pulled by attackers cannot reach arbitrary ports |
Instance. A Graviton
t4g.small running Ubuntu, the smallest size that runs the whole stack. A few settings I now consider default for any exposed EC2 instance:- CPU credits in standard mode, so bot load cannot create surprise charges.
- Encrypted root volume.
- IMDSv2 required, with a hop limit of 1, so containers (including the honeypot) cannot reach the instance metadata service.
- The AMI is ignored after creation, so a new Ubuntu image cannot silently replace the server and wipe the collected data.
- An Elastic IP keeps the address stable.
The server itself is configured by five idempotent Ansible roles: common, SSH hardening, firewall (nftables, default drop), Docker, and the honeypot stack. Secrets sit in Ansible Vault. A second playbook run that reports zero changes is my test that every role is truly idempotent. A GitHub Actions pipeline runs six checks on every push, from ansible-lint and
terraform validate to Gitleaks scanning the full Git history.Lessons learned: bugs that only appeared in the cloud
Everything worked on my local VM. Then I deployed to AWS, and the real learning started.
Every attacker had the same IP. My first version redirected port 22 to Cowrie's port with an nftables rule. On AWS, every single connection came from
172.18.0.1, the Docker bridge gateway. The redirected traffic was accepted by Docker's userland proxy, which opened a new connection to the container and erased the source address. For a honeypot, that makes the data worthless. The fix was to publish Cowrie directly on port 22, so Docker's kernel-level NAT handles the traffic and keeps the real attacker IP.My firewall broke Docker. The usual nftables setup starts with
flush ruleset, which also wipes the rules Docker creates, and container networking silently stops working. My firewall role now manages only its own tables and reloads instead of restarting.sshd reads the first value, not the last. Cloud images ship their own sshd drop-in files, and one of them re-enabled password login. Because sshd uses the first value it reads, I named my hardening file
00-hardening.conf so it always wins.The
Port directive was ignored. Newer Ubuntu releases start sshd through socket activation, which does not care about Port in the config. The role switches back to the classic service, and it validates the config before every restart so a typo can never lock me out.The VM thought it lived in the past. After my local VM was suspended, its clock drifted by hours, and package installs failed because repository metadata seemed to come from the future. Time synchronisation is now part of the base role.
None of these were in any tutorial I followed. Each one changed the design, and each one taught me how a layer actually works rather than how it is configured.
First observations
The first visitors were not hackers but scanners. In the first day, most connections opened a TCP session and closed it immediately, without even sending an SSH client version.
That is the reconnaissance phase. Tools like masscan, zmap and internet-wide scanning services just check whether port 22 is open and add the address to a list. Brute-force botnets usually arrive later, working from those lists. Reading the Cowrie events as a funnel (connect, SSH handshake, login attempt, successful login) makes the difference between the two very clear, and it is now my favourite dashboard panel.
What's next
The next stage is the part I am most excited about: a live world map of attack origins, using GeoIP enrichment in Alloy. After that, I plan to add Telegram alerts for unusual activity, continuous deployment through GitHub Actions with OIDC access to AWS and Systems Manager instead of SSH, and automatic reputation checks for malware that attackers try to download.
If you are learning AWS, I really recommend a project like this. It is cheap, it is fun, and the internet will test your security groups for you within hours.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article