Installing a Boomi Molecule on AWS EC2, Part 2: Unicast Clustering and the Web Server (with Common Mistakes)
Part 2 of 2. Switching a two-node Boomi Molecule from multicast to unicast, fixing MULTIPLE_HEAD_NODES, and configuring the Shared Web Server(with Common Mistakes to Avoid) Read Part 1 first.
This is part 2 of a two-part series. If you haven't done it yet, read Part 1 first. Part 1 covers building the EFS and EC2 infrastructure and running the Molecule installer. This part picks up once both nodes are installed and mounted on the same shared folder.
Part 5: Switch the cluster to unicast
Why this step exists
When a Molecule node starts, it needs to find the other nodes. Boomi's default is multicast. The node sends a "who's out there?" message to a special group address and port, and any other node on the same network replies. That works on an office LAN. AWS doesn't carry multicast traffic, though. The message goes nowhere, each node decides it's alone, and each one declares itself the head of its own one-node cluster.
Unicast replaces the broadcast with an explicit list: these are the nodes in this cluster, at these addresses, listening on TCP port 7800. Two settings control it:
| Setting in the platform | Property name in the config file | Value |
|---|---|---|
| Cluster Network Transport Type (older name: Cluster Communication Method) | com.boomi.container.cluster.type | UNICAST |
| Initial Hosts for Unicast | com.boomi.container.cluster.initialHosts | <NODE_1_PRIVATE_IP>[7800],<NODE_2_PRIVATE_IP>[7800] |
The list uses the private IPs, each followed by
[7800], separated by commas with no spaces. It doesn't have to include every node, just enough that at least one is always running. I listed both for simplicity.Set it in the platform, not in the file
The reliable way is through the platform UI, and it needs the Molecule to be online first. That creates a small chicken-and-egg sequence:
- Start node 1 only. Wait until it shows green in Atom Management. On its own it will run in multicast mode, which is fine for now because nothing else is trying to join.
- In Atom Management, click the Molecule, then open Properties and the Advanced tab.
- The Advanced tab has a long alphabetical dropdown. Scroll to the C section, select Cluster Network Transport Type, and set it to UNICAST.
- Open the dropdown again, scroll to the I section, select Initial Hosts for Unicast, and enter the list of private IPs.
- Save. The platform pushes the values to the runtime and writes them to the config file on EFS.
- Restart node 1 with
./atom restart. The cluster layer only switches transport after a restart.
Common mistake: editing the config file by hand while the Molecule is offline. That's exactly what I tried first, adding the two lines to
container.properties with echo. The lines were there when I checked. But when node 1 started, it registered with the platform, the platform pushed its own copy of the properties (which didn't have my lines), and my edits were overwritten. The node came up in multicast mode. The platform is the source of truth for runtime properties, so set them in the UI.Common mistake: looking for the old property name. Older guides call this setting "Cluster Communication Method." In current versions it's listed as Cluster Network Transport Type. The dropdown is alphabetical, so scroll rather than searching.
Leave Cluster Network Bind Address unset. The node binds to its private IP automatically, and the log confirms that with a line about
jgroups.bind_addr.Check which mode a node is really in
Don't rely on the settings page alone. Look at the ports the Java process has open:
1
sudo ss -tulnp | grep javass lists sockets. -t is TCP, -u is UDP, -l shows listening ports only, -n shows port numbers instead of names, and -p shows which process owns each one.| What you see | What it means |
|---|---|
A UDP line for 228.10.10.10:45588 | Still in multicast mode. The unicast settings haven't taken effect |
A TCP LISTEN line for <private-ip>:7800 | Unicast. The node is listening for the other nodes on the cluster port |
This check is what finally showed me the real problem, which I describe in Part 6.
Part 6: Start the nodes and form the cluster
Start node 1
On node 1:
1
2
cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom startThe
bin folder holds the control script. Despite the name, atom is also the script for Molecules. Wait about a minute, then check:1
./atom statusIt prints
atom is running. or atom is not running. In Atom Management, the Molecule should turn green.Start node 2
Node 2 never runs the installer, because the installation is already on the shared disk. It just starts the same script:
1
2
cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom startDo this only after node 1 is green, the unicast settings from Part 5 are in place, and node 1 has been restarted with them. Then wait two minutes and open Atom Management, then the Molecule, then Cluster Status.
What success looks like
You should see two rows, both Online, with a star next to exactly one Node ID. That star marks the head node. There should be no Cluster Issues tab. Take a screenshot for your records.
Problem: MULTIPLE_HEAD_NODES
This is the problem most people hit on their first AWS Molecule, and it's exactly what happened to me. The Cluster Issues tab showed a severe warning,
MULTIPLE_HEAD_NODES, with each node reporting a different head node.It means each node wrote a small "view snapshot" file into the
bin/views folder on the shared disk, recording who it thinks the cluster members are and who the head is. The platform compares those files. In my case each file said "I'm the head, and I'm the only member." So the two nodes weren't talking to each other at all. They were two separate one-node clusters sharing a disk. That's dangerous, because both will try to run the same scheduled work and write to the same execution folders.There are only three possible causes, so check them in this order, from cheapest to most involved:
- The nodes are still in multicast mode, so discovery messages never arrive. This was my actual cause.
- The network between the nodes is blocked, either because one instance is in the wrong security group or the self-referencing rule is missing.
- The nodes started too close together, or node 2 started before node 1 finished forming the cluster, and stale snapshot files kept the wrong state alive.
Start with a clean stop. Stop node 2, then node 1, so nothing changes while you investigate:
1
2
cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom stopCheck that the settings reached the runtime. The config file is on the shared disk, so either node will show the same thing:
1
grep -i cluster /mnt/boomi/Molecule_<YOUR_NAME>/conf/container.propertiesYou want to see
cluster.type=UNICAST and an initialHosts line. If they're missing, go back to Part 5 and set them in the UI. In my case they were there, which was misleading, and the next check showed why.Check the network in both directions. From node 2, run
ping -c 3 <NODE_1_PRIVATE_IP>, and from node 1, run ping -c 3 <NODE_2_PRIVATE_IP>. The -c 3 option sends three packets and stops. If you see 0% packet loss, the security group is letting the nodes reach each other. If you see 100% loss, one instance isn't in the shared security group. Fix it under EC2, then Instances, then Actions, then Security, then Change security groups.Check what the running node is actually doing. Start node 1 and wait a minute, then run
sudo ss -tulnp | grep java. In my case the output showed a UDP line for the multicast address and no TCP 7800 at all. So the node was still in multicast mode, even though the file said UNICAST. The platform sync had overwritten my hand edits, and setting the properties through the UI fixed it. Once I'd done that and restarted node 1, ss showed a TCP LISTEN on port 7800, and node 2 joined.You can also test the cluster port directly from the other node:
1
2
sudo yum install nmap-ncat -y
nc -zv <NODE_1_PRIVATE_IP> 7800nc -z opens a connection and closes it without sending data. -v tells you what happened. A "connection refused" while ss shows nothing on port 7800 means the service isn't listening, which is a configuration problem. A "connection refused" or timeout while ss does show 7800 means a firewall problem.Finish with an ordered restart, so no stale state survives:
1
2
3
4
5
6
7
8
9
10
11
12
13
# on node 2, then node 1
./atom stop
# once, from either node: clear the old snapshot files
rm -f /mnt/boomi/Molecule_<YOUR_NAME>/bin/views/*.dat
# on node 1
./atom start
# wait until green in Atom Management, then wait one more full minute
sudo ss -tlnp | grep 7800 # must show java on 7800 before you continue
# on node 2
./atom startDeleting the
.dat files is safe, because they're regenerated within a minute. Skipping the wait between node 1 and node 2 is what creates the race condition in cause 3.Problem: NODE_JAVA_HOME_MISMATCH
Right after the cluster formed, another warning appeared:
NODE_JAVA_HOME_MISMATCH, saying node 2 was out of sync with the current Java home.Read the bracket in the message first. It says node 2 is reporting that node 1 is the head. That's good news, because it means
MULTIPLE_HEAD_NODES is resolved.The remaining issue comes from the "store runtime assets locally" choice during installation. Each node keeps its own copy of the Java runtime. Node 1's copy was created by the installer, but node 2's was created when it first started and hadn't been marked current yet. Restarting node 2 refreshes it:
1
2
3
# on node 2
./atom stop
./atom startIf the warning comes back, force node 2 to rebuild its local copy from the shared installation:
1
2
3
./atom stop
rm -rf /home/ec2-user/boomi_local/*
./atom startThis deletes only node 2's local cache. It never touches the shared installation.
Part 7: Configure the Shared Web Server
A Molecule doesn't listen on any inbound HTTP port until you configure one. Until you do, the API Management page warns that no listener ports are set, and
curl http://localhost:9090 on a node says it can't connect.Set the port in the platform
In Atom Management, open the Molecule, then Shared Web Server. Check that the breadcrumb at the top names the Molecule, not an individual Atom. Under Listening Port Configuration, add a row with port
9090, SSL unchecked, and Authentication Type set to None. That's fine for testing only. Scroll to the bottom and save.Restart the nodes
A new listener port only opens at startup, so both nodes have to restart. In a cluster, restart one node at a time so the Molecule stays online:
1
2
3
4
5
6
7
8
9
# on node 2
./atom stop
# on node 1
./atom restart
# wait until green, plus one more minute
# on node 2
./atom startA Molecule node takes longer to come up than an Atom, so be patient.
Verify on each node
1
2
sudo ss -tlnp | grep 9090
curl http://localhost:9090You want a LISTEN line owned by java on port 9090. The curl should return an HTML page reading
Error 404 NOT_FOUND. That 404 is the correct result. It proves the web server is running. There's just nothing deployed at the root path yet.A note on the Base URL column: it shows node 1's hostname, but every node opens the port, so requests to node 2's IP on 9090 work the same way. A load balancer in front of the nodes is what gives clients one address. You don't need it for testing.
Problem: the port didn't open after the restart
On my first restart,
ss only showed port 7800 and a random port, and curl couldn't connect. The setting was in the config file, though.The container log explained why. The web server libraries loaded at one time, and about five minutes later the log printed the full property set, including
sharedServer.http.connector.0.port=9090. The property arrived after the restart, because I'd saved the change in the UI after restarting the node, not before. The runtime accepted the new value, but a listener port only binds at startup.The fix was another restart on each node, followed by a 90-second wait. The rule is: save first, restart second, then wait a full 90 seconds before you check.
If the port still doesn't show up after a correctly ordered save and restart, try deploying a listener process to the Molecule and check again. On some versions the shared web server doesn't bind until at least one Web Services Server process is deployed.
Part 8: Next steps
With the cluster online and the web server listening, the Molecule is ready for real work. These are the next things I plan to do.
- Deploy a test listener. A simple one is a Start step set to Web Services Server with a REST resource and POST, a Message step, and Return Documents. Deploy it to the Molecule's environment. Because the Molecule lives in a different account from my original Atom, I had to rebuild the process there.
- Load-test each node separately. Run Autocannon from inside a node, pointing it at each node's private IP one at a time. Then check Process Reporting. Executions should show both node IDs, which proves the cluster is actually sharing work.
- Put a load balancer in front. An Application Load Balancer with a target group containing both nodes on port 9090 gives you one stable URL and spreads requests across the nodes. That's the production shape of a Molecule. I cover this in the next post.
- Turn on HTTPS and authentication on the Shared Web Server before exposing anything beyond a test network.
Day-to-day operation
Stopping for the day
Order matters in a cluster. Stop the member node first and the head node last, so the head doesn't spend time re-electing itself. On node 2, run
./atom stop and then ./atom status to confirm it's down. Then do the same on node 1.Then, in the AWS Console, go to EC2, then Instances, select both nodes, and choose Instance state, then Stop instance. Stopping ends the compute charge, which is almost all of the cost. EFS, the security group, and the key pair are left alone.
Resuming
- Start both instances in the console. Their public IPs will have changed, so copy the new ones for SSH. The private IPs are the same, so the unicast list still works.
- On each node, run
df -h /mnt/boomito confirm EFS came back through/etc/fstab. If it didn't, runsudo mount -a. - On node 1, run
./atom startand wait for green plus one minute. - On node 2, run
./atom start. - Check Cluster Status.
Pause versus tear down
| Stop instances | Terminate instances and delete EFS | |
|---|---|---|
| Compute charge | None | None |
| Storage charge | Root disks of stopped instances plus EFS, a few rupees per day | None |
| Molecule preserved? | Yes, exactly as you left it | No, you'll need a full reinstall |
| When to choose it | Pausing for days or weeks | When the experiment is finished |
If you terminate, also release any Elastic IPs, delete leftover EBS volumes and snapshots, and remove the Molecule from Atom Management so it doesn't stay behind as an offline runtime.
Common mistakes, in the order they happen
| Stage | Mistake | Symptom | Fix |
|---|---|---|---|
| Before you start | Not checking the Molecule licence count | Installer fails at "Retrieving Container ID" and rolls back | Check Licensing first, free a slot, or use another account |
| Before you start | Downloading from the wrong Boomi account | Molecule shows up in the wrong account, or the token is rejected | Switch accounts, generate the token there, reuse the installer |
| Security group | Leaving the description blank | The form won't submit | Type any text |
| Security group | Adding self-referencing rules during creation | "CIDR block, a security group ID or a prefix list has to be specified" | Create with SSH and 9090 first, then add the two self-referencing rules by editing |
| Security group | Typing the group name in the Description column | Rule saves but does nothing | Put the group in the Source search box |
| EFS | Mounting before the targets are Available or before attaching the security group | Mount hangs, then "Connection timed out" | Wait for Available, then assign the group under Network, then Manage |
| EFS | Missing .efs in the DNS name, or stray backticks | "Failed to resolve server" | Copy the DNS name from the EFS console exactly |
| SSH | Using the private IP from your laptop | Connection reset by peer | Use the public IP |
| SSH | Your laptop's IP changed since the rule was created | Operation timed out on port 22 | Edit the SSH rule's source to My IP again |
| SSH | Running ssh or scp while already on a server | Key file not accessible, or cannot stat the local file | Read the prompt, and run ssh and scp from your laptop |
| Terminal tabs | Losing track of which tab is which node | Commands run on the wrong machine | Quit the app, reconnect deliberately, and check each prompt |
| scp | Leaving off :~/ at the end of the destination | Copy fails or goes to the wrong place | Use user@host:~/ |
| Installer | Typing an email or token at a [1] or [2] prompt | The prompt repeats | Type the digit |
| Installer | Choosing option 1 when you meant a token | Asked for a username and password | Press Ctrl+C, run again, answer 2 |
| Installer | Typing the environment name or ID instead of its number | The list reprints | Type the number in brackets |
| Installer | Typing y at a [1] or [2] prompt | The prompt repeats | Type 1 or 2 |
| Installer | Pressing Enter on a required field with empty brackets | "... is required," then loops back | Type the value |
| Installer | Accepting the default installation directory | Node 2 has nothing to start | Install under /mnt/boomi |
| Installer | Node 2's boomi_local folder not created | Node 2 fails to start or reports a missing local directory | mkdir -p /home/ec2-user/boomi_local on every node |
| Unicast | Leaving multicast on | MULTIPLE_HEAD_NODES, and the nodes never join | Set Cluster Network Transport Type to UNICAST and set the initial hosts |
| Unicast | Editing container.properties by hand | The lines are there, but the node is still in multicast mode | The platform overwrites the file on sync. Set the properties in the UI, then restart |
| Unicast | Searching for the old property name | Can't find "Cluster Communication Method" | It's now called Cluster Network Transport Type |
| Unicast | Using public IPs in the initial hosts list | Nodes can't connect on 7800 | Use private IPs |
| Start-up | Starting node 2 too soon after node 1 | MULTIPLE_HEAD_NODES from stale snapshots | Wait for green plus a minute, delete bin/views/*.dat, and restart in order |
| Cluster | NODE_JAVA_HOME_MISMATCH after joining | Severe issue on node 2 | Restart node 2, and if needed clear its boomi_local folder |
| Web server | Restarting before saving the port | 9090 never opens | Save first, restart, then wait 90 seconds |
| Web server | Treating the root-path 404 as a failure | Confusion | A 404 from / means the web server is up |
Command reference
On your laptop
| Command | What it does |
|---|---|
chmod 400 ~/Downloads/<KEY_FILE>.pem | Makes the key file readable only by you. SSH refuses keys with looser permissions. Run this once |
ssh -i ~/Downloads/<KEY_FILE>.pem ec2-user@<PUBLIC_IP> | Opens a terminal on a node. -i selects the key file |
scp -i ~/Downloads/<KEY_FILE>.pem <local-file> ec2-user@<PUBLIC_IP>:~/ | Copies a file from your laptop to the node's home folder |
On the nodes: setup
| Command | What it does |
|---|---|
sudo yum update -y | Updates the operating system packages. -y answers yes to prompts |
sudo yum install java-17-amazon-corretto nfs-utils -y | Installs a system Java and the NFS client tools |
sudo mkdir -p /mnt/boomi | Creates the folder the shared disk attaches to. -p creates parent folders and doesn't complain if the folder exists |
sudo mount -t nfs4 -o nfsvers=4.1 <EFS_DNS_NAME>:/ /mnt/boomi | Attaches the root of the EFS file system to /mnt/boomi using NFS 4.1 |
df -h /mnt/boomi | Shows what's mounted at that path and how much space it has |
echo "<EFS_DNS_NAME>:/ /mnt/boomi nfs4 nfsvers=4.1,_netdev 0 0" | sudo tee -a /etc/fstab | Adds a line to the boot-time mount table so the mount returns after a reboot |
sudo mount -a | Mounts everything listed in /etc/fstab that isn't already mounted |
sudo chown ec2-user:ec2-user /mnt/boomi | Makes ec2-user the owner of the shared folder |
touch <file>, ls <dir>, rm <file> | Create an empty file, list a folder, or delete a file |
mkdir -p /home/ec2-user/boomi_local | Creates the per-node local assets folder |
On the nodes: installer
| Command | What it does |
|---|---|
chmod u+x molecule_install64.sh | Gives the downloaded installer permission to run |
./molecule_install64.sh | Runs the installer. The ./ is needed because the script isn't on the PATH |
| Ctrl+C | Cancels a running command. It's safe to use before the installer's summary screen |
On the nodes: Molecule control
Run these from
/mnt/boomi/Molecule_<YOUR_NAME>/bin.| Command | What it does |
|---|---|
./atom start | Starts this node |
./atom stop | Stops this node cleanly |
./atom restart | Stops, then starts |
./atom status | Prints whether this node is running |
On the nodes: diagnostics
| Command | What it does |
|---|---|
sudo ss -tlnp | Lists TCP ports that are listening, and which process owns each |
sudo ss -tulnp | grep java | Shows the Java process's TCP and UDP ports. Multicast appears as UDP 228.10.10.10:45588. Unicast appears as TCP :7800 |
ping -c 3 <private-ip> | Sends three packets to the other node to test basic reachability |
nc -zv <private-ip> 7800 | Tests whether a TCP connection to the cluster port succeeds. Install it with sudo yum install nmap-ncat -y |
grep -i cluster .../conf/container.properties | Shows the cluster settings the runtime currently holds |
grep -r 9090 .../conf/ | Finds any file under conf that mentions the port |
grep -i -E "pattern1|pattern2" <logfile> | tail -30 | Pulls matching lines from a log. -E enables the | OR option. tail -30 shows the last 30 lines |
tail -50 <logfile> | Shows the last 50 lines of a file |
rm -f .../bin/views/*.dat | Deletes the cluster snapshot files. They regenerate on start. Use this to clear stale head-node state |
Appendix A: How to read installer prompts
Every question the console installer asks follows one of three patterns. Once you recognize them, most of the installer mistakes go away.
| Pattern | Example | What to type |
|---|---|---|
| Options with characters in brackets | User Name and Password [1, Enter], Token [2], or Yes [y, Enter], No [n] | Exactly the character in the brackets. Nothing else is accepted. The option marked Enter is the default |
| A default value in brackets | Choose an installation directory [/home/ec2-user/...] | Enter to accept the default, or type a replacement |
| Empty brackets | Token [] or Working Data Local Storage Directory (required) [] | There's no default. Type the value. Enter on its own gets rejected |
If a prompt keeps repeating, you've given it something it doesn't recognize. Look at the brackets again.
Appendix B: Where the logs are
All of these paths are under the shared installation, so you can read them from either node.
| Path | Contents |
|---|---|
/mnt/boomi/Molecule_<YOUR_NAME>/logs/YYYY_MM_DD.container.<node-id>.log | One container log per node per day. The node ID is the private IP with underscores, for example <NODE_ID>. This is the first place to look for cluster and web server problems |
/mnt/boomi/Molecule_<YOUR_NAME>/conf/container.properties | The runtime properties as last synced from the platform |
/mnt/boomi/Molecule_<YOUR_NAME>/bin/views/*.dat | Cluster snapshot files, one per node |
/home/ec2-user/boomi_local/ (on each node) | Per-node local copies of the Java runtime, libraries, and working data |
Useful search terms for the container log are
cluster, unicast, multicast, MULTIPLE_HEAD_NODES, jgroups.bind_addr, sharedServer, jetty, and 9090. Searching for Container Properties for account shows the full property set at the moment the platform pushed it, with a timestamp you can compare to your restart time.Closing
Getting the cluster to form was the hardest part. Most of the problems came from one thing: the nodes couldn't see each other, and the cause wasn't obvious from the error messages. The fix was switching to unicast through the UI, checking the actual open ports rather than trusting the settings page, and restarting in the right order.
In the next post I'll put an Application Load Balancer in front of these two nodes, so clients get one address and keep working when a node goes down.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article