November 8 2026 Sunday

Domino servers - removing eMudhra CA certificates using Salt

I was reading Daniel Nashed's post about removing these and decided to utilize Salt to remove the offending certificates from both Windows and Linux Domino servers. I will prefix this with the usual you use this at your own risk, and neither I nor STS take any responsibility if you do. It worked fine for me over a number of Domino servers (~10). Also, this will be far for perfect, so change as you see fit. It is my expertise and Claude's coding.

What is interesting is that this uncovered that Traveler also has the errant certificate(s) in it own keystore independent of Domino's. It addresses Traveler as well.

There are two zip files attached, one the includes Python 3 for Windows (Linux probably already has it installed), and one that does not. Choose whichever you need. You will need Python installed on the Windows Domino servers, at least for the duration of your cleanup. You can remove if after:

domino-ca-cleanup-WITHOUT-python-installer.zip (~21 KB)

domino-ca-cleanup-WITH-python-installer.zip (~35 MB)

Yes, you need a working Salt Master. Adjust the paths in the files to suit your install. It also expects every Domino minion to have a grain called roles to have a value named domino. There is also the ability to check inside Docker containers running Domino. For more detailed instructions than below read the domino_ca_removal_runbook.md file included in the zip files.

This runbook removes the compromised eMudhra/emSign G3 root from HCL Domino trust stores, using verified off-host backups and a gated, per-host process. It assumes you’ve unpacked the bundle domino-ca-cleanup-XXX.zip and installed its files.
1. Installation & Setup
Unpack the bundle (domino-ca-cleanup-.zip) and install files:
  • states/domino_ca/* -> States repo (GitFS root: states)
  • states/domino_ca/files/* -> files/ folder
  • pillar/top.sls, pillar/domino_ca.sls -> Central pillar repo
  • master/domino-rollout.sh -> /usr/local/sbin/domino-rollout.sh (chmod 0700)
  • master/sweep-domino-backups.sh -> /usr/local/sbin/sweep-domino-backups.sh (chmod 0700)
Verify bundle integrity (sha256sum -c MANIFEST.sha256 must show OK), then update Salt: salt-run fileserver.update salt-run git_pillar.update # If pillar uses git_pillar salt '*' saltutil.refresh_pillar
Configuration (domino_ca.sls):
  • fingerprints: Verify SHA-256 against vendor advisory.
  • Paths & Python: Set windows_python (C:\Python315\python.exe) and linux_python.
  • Keystore: Default storepass is changeit (PKCS12).
2. Prerequisites
  • Master: Enable file_recv: True in /etc/salt/master temporarily. Ensure /srv/backups has sufficient space.
  • Inventory Check: Run to confirm registry paths and domino grain: salt '*' cmd.run 'reg query "HKLM\SOFTWARE\HCL\Domino" /v Path' python_shell=True
  • Windows Minions: Install Python 3.15.2 to C:\Python315. Verify: salt '' cmd.run 'C:\Python315\python.exe -c "import encodings, sys; print(sys.version)"' python_shell=True
  • Domino State: Domino must be running (listening on TCP 1352) for the tool to allow modifications.
3. Execution Flow (Per-Host)
Run hosts sequentially (one at a time, no parallel batches).
Step 1: Confirm Host is Idle salt '' saltutil.running --out=json Expected: {"": []} (Clear stuck jobs via saltutil.kill_job if needed).
Step 2: Run the Rollout Script sudo /usr/local/sbin/domino-rollout.sh The script steps through gates interactively:
  1. Audit: Lists keystores, entry counts, and matches.
  2. G3-Only Gate: Stops if any match is not the G3 root.
  3. Backup Pass (y): Copies, verifies, sidecars, and pushes keystores to master.
  4. Sweep (y): Moves pushed files to /srv/backups// (failed=0 required).
  5. Dry Run: Validates match counts.
  6. Apply (y): Removes G3 aliases.
  7. Sweep Again (y): Moves apply backups to master (failed=0 required).
  8. Post-Audit: Verifies matches=0 and Domino JVM size decreased by 1.
  9. Restart Prompt: Answer y only after restarting Domino and verifying health.
Step 3: Restart & Verify Restart Domino normally. Check server logs for clean database opens and test dependent services (Mail flow, Traveler sync). Remediate non-mail hosts first, mail hosts last.
Step 4: Final Audit & Close-Out sudo salt '' state.apply states.domino_ca.audit --timeout=600 --out=json > final-.json sudo /usr/local/sbin/sweep-domino-backups.sh After fleet verification (matches=0 everywhere), disable file transfers on the Master: sudo sed -i 's/^file_recv: True/file_recv: False/' /etc/salt/master sudo systemctl restart salt-master Archive /srv/backups off-host and verify hashes.
4. Special Scenarios & Pitfalls
  • Stopped Servers: If Domino is down (port 1352 closed), the tool blocks modifications to protect live states. Manually invoke keytool via a Salt PowerShell script after backing up, then verify with audit.
  • Docker Domino: Use states.domino_ca.docker. Rebuild the container image without the G3 root, otherwise container recreation restores the certificate from the image layer.
  • Windows Paths & Quoting: Use .ps1 files served via Salt for paths containing spaces (e.g., C:\Program Files).
Darren Duke   |   November 8 2026 11:27:50 AM   |    domino salt security    |  
  |   Next Document   |   Previous Document

Comments (0)

No Comments Found


content : © 2008 - 2020 darren duke - design : © 2013 serdar basegmez   |   powered by hcl® domino® blog template