Split-Brain, Timeouts, and Silent Failures: Mastering STONITH for SAP HANA on AWS

A practical troubleshooting guide for STONITH fencing in SAP HANA Pacemaker clusters on SLES running on AWS. Covers real-world failure scenarios including split-brain, IAM denials, timeouts, and silent rejoins with exact configurations and testing procedures.

WARISH, SEKH MOHAMMAD

120XSAP 12XAWS 15xAZURE 12xORACLE 10XSCRUM 11XOCI 12XMS 5XSECURITY 2XSOLARIS 2XDB2

Published Jun 24, 2026

Last modified Jun 24, 2026

What is STONITH and Why Does It Matter?

STONITH stands for "Shoot The Other Node In The Head." In a two-node Pacemaker cluster, when one node cannot communicate with the other, both nodes believe they should be the Primary. This is a split-brain scenario, and for SAP HANA, split-brain means potential data corruption.

STONITH prevents this by giving the cluster a mechanism to forcibly shut down a node it cannot communicate with. On AWS, this means calling the EC2 API to stop the instance. No polite shutdown request. A hard stop.

Without working STONITH, your cluster is essentially running without a safety net. Pacemaker will refuse to perform automated failovers if STONITH is not enabled.

The AWS STONITH Architecture

On AWS with SLES, the fencing mechanism works like this:

Pacemaker detects a node is unresponsive
The surviving node invokes the STONITH resource agent
The agent calls the AWS EC2 API (StopInstances) to forcibly stop the failed node
Once the stop is confirmed, Pacemaker proceeds with failover

The resource agent depends on your SLES version:

SLES 15 SP5+: stonith:fence_aws (recommended, supports skip_os_shutdown=true)
SLES 15 SP3/SP4: stonith:external/ec2 (legacy, still supported)

The key difference: fence_aws with skip_os_shutdown=true stops the instance at the hypervisor level. The hung node does not get a chance to refuse the shutdown.

The Configuration That Actually Works

Based on AWS documentation for SAP HANA Pacemaker clusters on SLES:

Cluster Properties

1
2
3
4
crm configure property stonith-enabled="true"
crm configure property stonith-timeout="600"
crm configure property priority-fencing-delay="20"
crm configure property stonith-action="off"

stonith-enabled=true: Non-negotiable. Without this, no automated failovers in split-brain.
stonith-timeout=600: Max time Pacemaker waits for fence to complete. Set high because AWS API calls can be slow for degraded instances.
priority-fencing-delay=20: During network partition, BOTH nodes try to fence each other. This delay gives the Primary node (higher priority) a 20-second head start, ensuring the correct node survives.
stonith-action=off: Fenced nodes stay stopped until manually investigated. Prevents corrupted nodes from auto-rejoining. AWS recommends this over "reboot" for SAP HANA.

STONITH Resource Definition (SLES 15 SP5+)

1
2
3
4
5
6
7
8
9
10
crm configure primitive rsc_fence_aws stonith:fence_aws \
  params pcmk_host_map="hanahost01:i-0abc123def456;hanahost02:i-0def789ghi012" \
  region="us-east-1" \
  skip_os_shutdown="true" \
  pcmk_delay_max="10" \
  pcmk_reboot_timeout="600" \
  pcmk_reboot_retries="4" \
  op start interval="0" timeout="600" \
  op stop interval="0" timeout="180" \
  op monitor interval="300" timeout="60"

Key parameters:

pcmk_host_map: Maps hostnames to EC2 instance IDs. Wrong mapping = fencing targets wrong instance.
skip_os_shutdown=true: Hypervisor-level stop. No OS involvement.
pcmk_delay_max=10: Combined with priority-fencing-delay=20, gives Primary a 20-30s advantage.
pcmk_reboot_retries=4: Retries on failure instead of giving up.
monitor interval=300: Checks every 5 minutes that fencing agent can reach EC2 API.

Real-World Failure Scenarios and How to Fix Them

Scenario 1: "Timed out waiting to power OFF"

What happens: Fence agent calls StopInstances but instance does not stop within timeout. Reports "Unable to obtain correct plug status."

Root cause: Node kernel panic. Instance in degraded state.

Fix:

Use skip_os_shutdown=true (bypasses OS shutdown)
Increase pcmk_reboot_timeout to 600
Set pcmk_reboot_retries=4

Scenario 2: Fenced Node Silently Rejoins

What happens: Node gets fenced (stopped). Someone or automation restarts it. Pacemaker starts on boot. Two nodes both think they are Primary.

Fix:

Set stonith-action=off (node stays stopped)
After investigation, start Pacemaker manually: systemctl start pacemaker
Verify HANA replication before allowing node back: hdbnsutil -sr_state

Scenario 3: IAM Permission Denied During Fencing

What happens: Real failure occurs. Fencing agent gets AccessDenied from EC2 API. Fencing fails. No failover. Both nodes stuck.

Required IAM Policy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StopInstances",
        "ec2:StartInstances",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus"
      ],
      "Resource": "*"
    }
  ]
}

Fix:

Test fencing BEFORE go-live: crm node fence nodename
Check CloudTrail for AccessDenied events on StopInstances
Ensure both instance IDs are in the policy scope

Scenario 4: Split-Brain, Both Nodes Fence Each Other

What happens: Network partition between AZs. Both healthy nodes simultaneously fence each other. Both succeed, zero nodes running.

Fix:

Set priority-fencing-delay=20 at cluster property level
Set pcmk_delay_max=10 on STONITH resource
Primary (higher priority) gets 20-30s head start in the fence race
Verify: crm configure show property | grep priority-fencing

Scenario 5: Monitor Operation Fails

What happens: STONITH resource shows Failed Actions with monitor errors. Agent may not be usable when needed.

Root cause: Transient EC2 API throttle or metadata endpoint slowness.

Fix:

Monitor timeout at least 60s: op monitor interval="300" timeout="60"
Clean up: crm resource cleanup rsc_fence_aws
Check metadata endpoint is accessible from the instance
Note: Pacemaker does not require STONITH to be "started" to use it for fencing

Testing Your Fencing: Do It Before Production

Test 1: Manual fence

1
crm node fence hanahost02

Verify: Instance stops, failover occurs, Overlay IP moves.

Test 2: Hard stop from AWS

Stop EC2 instance via Console/CLI. Surviving node should detect, fence (no-op), and promote.

Test 3: Network isolation

Block traffic between AZs via Security Groups. Primary should survive.

Test 4: Kernel panic

1
echo c > /proc/sysrq-trigger

Surviving node should fence and failover within stonith-timeout seconds.

Quick Reference Commands

Task	Command
Check cluster status	crm status
Show STONITH config	crm configure show rsc_fence_aws
Test fence manually	crm node fence hostname
Check fence history	stonith_admin --history
Clean failed actions	crm resource cleanup rsc_fence_aws
Put cluster in maintenance	crm maintenance on
Check HANA replication	hdbnsutil -sr_state (as sidadm)
Verify IAM from instance	aws ec2 describe-instances --instance-ids i-xxx
Check Pacemaker logs	journalctl -u pacemaker
Check corosync connectivity	corosync-cfgtool -s

Final Thoughts

STONITH is not a "set it and forget it" component. It is the single most critical piece of your HA cluster. If fencing fails, failover fails. If failover fails, your SAP HANA production database is down with no automatic recovery.

Test it. Test it after OS patches. Test it after IAM policy changes. Test it after instance replacements (because instance IDs change and pcmk_host_map needs updating).

The five minutes you spend testing fencing today saves you the 3 AM call tomorrow.

References

SAP HANA Pacemaker Cluster Configuration on SLES: https://docs.aws.amazon.com/sap/latest/sap-hana/sap-hana-pacemaker-sles-cluster-config.html
EC2 Instance Configuration for SAP HANA Clusters: https://docs.aws.amazon.com/sap/latest/sap-hana/sap-hana-pacemaker-sles-ec2-configuration.html
Updated Guidance for SAP Pacemaker Configuration: https://aws.amazon.com/blogs/awsforsap/updated-guidance-for-sap-pacemaker-configuration-abap-sap-central-services-ascs/
Manual Deployment of SAP HANA on AWS with HA Clusters: https://docs.aws.amazon.com/sap/latest/sap-hana/sap-hana-on-aws-manual-deployment-of-sap-hana-on-aws-with-high-availability-clusters.html
Monitor SAP Pacemaker Clusters Using AWS Services: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/monitor-sap-rhel-pacemaker-clusters-by-using-aws-services.html

# sap # architecture # best-practices # disaster-recovery # resilience

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Split-Brain, Timeouts, and Silent Failures: Mastering STONITH for SAP HANA on AWS

WARISH, SEKH MOHAMMAD

What is STONITH and Why Does It Matter?

The AWS STONITH Architecture

The Configuration That Actually Works

Cluster Properties

STONITH Resource Definition (SLES 15 SP5+)

Real-World Failure Scenarios and How to Fix Them

Scenario 1: "Timed out waiting to power OFF"

Scenario 2: Fenced Node Silently Rejoins

Scenario 3: IAM Permission Denied During Fencing

Scenario 4: Split-Brain, Both Nodes Fence Each Other

Scenario 5: Monitor Operation Fails

Testing Your Fencing: Do It Before Production

Quick Reference Commands

Final Thoughts

References