November 8 2026 Sunday
Domino servers - removing eMudhra CA certificates using Salt
I was reading Daniel Nashed's post about removing these and decided to utilize Salt to remove the offending certificates from both Windows and Linux Domino servers. I will prefix this with the usual you use this at your own risk, and neither I nor STS take any responsibility if you do. It worked fine for me over a number of Domino servers (~10). Also, this will be far for perfect, so change as you see fit. It is my expertise and Claude's coding.
What is interesting is that this uncovered that Traveler also has the errant certificate(s) in it own keystore independent of Domino's. It addresses Traveler as well.
There are two zip files attached, one the includes Python 3 for Windows (Linux probably already has it installed), and one that does not. Choose whichever you need. You will need Python installed on the Windows Domino servers, at least for the duration of your cleanup. You can remove if after:
domino-ca-cleanup-WITHOUT-python-installer.zip (~21 KB)
domino-ca-cleanup-WITH-python-installer.zip (~35 MB)
Yes, you need a working Salt Master. Adjust the paths in the files to suit your install. It also expects every Domino minion to have a grain called roles to have a value named domino. There is also the ability to check inside Docker containers running Domino. For more detailed instructions than below read the domino_ca_removal_runbook.md file included in the zip files.
This runbook removes the compromised eMudhra/emSign G3 root from HCL Domino trust stores, using verified off-host backups and a gated, per-host process. It assumes you’ve unpacked the bundle domino-ca-cleanup-XXX.zip and installed its files.
1. Installation & Setup
Unpack the bundle (domino-ca-cleanup-.zip) and install files:
Configuration (domino_ca.sls):
Run hosts sequentially (one at a time, no parallel batches).
Step 1: Confirm Host is Idle salt '' saltutil.running --out=json Expected: {"": []} (Clear stuck jobs via saltutil.kill_job if needed).
Step 2: Run the Rollout Script sudo /usr/local/sbin/domino-rollout.sh The script steps through gates interactively:
Step 4: Final Audit & Close-Out sudo salt '' state.apply states.domino_ca.audit --timeout=600 --out=json > final-.json sudo /usr/local/sbin/sweep-domino-backups.sh After fleet verification (matches=0 everywhere), disable file transfers on the Master: sudo sed -i 's/^file_recv: True/file_recv: False/' /etc/salt/master sudo systemctl restart salt-master Archive /srv/backups off-host and verify hashes.
4. Special Scenarios & Pitfalls
What is interesting is that this uncovered that Traveler also has the errant certificate(s) in it own keystore independent of Domino's. It addresses Traveler as well.
There are two zip files attached, one the includes Python 3 for Windows (Linux probably already has it installed), and one that does not. Choose whichever you need. You will need Python installed on the Windows Domino servers, at least for the duration of your cleanup. You can remove if after:
domino-ca-cleanup-WITHOUT-python-installer.zip (~21 KB)
domino-ca-cleanup-WITH-python-installer.zip (~35 MB)
Yes, you need a working Salt Master. Adjust the paths in the files to suit your install. It also expects every Domino minion to have a grain called roles to have a value named domino. There is also the ability to check inside Docker containers running Domino. For more detailed instructions than below read the domino_ca_removal_runbook.md file included in the zip files.
This runbook removes the compromised eMudhra/emSign G3 root from HCL Domino trust stores, using verified off-host backups and a gated, per-host process. It assumes you’ve unpacked the bundle domino-ca-cleanup-XXX.zip and installed its files.
1. Installation & Setup
Unpack the bundle (domino-ca-cleanup-.zip) and install files:
- states/domino_ca/* -> States repo (GitFS root: states)
- states/domino_ca/files/* -> files/ folder
- pillar/top.sls, pillar/domino_ca.sls -> Central pillar repo
- master/domino-rollout.sh -> /usr/local/sbin/domino-rollout.sh (chmod 0700)
- master/sweep-domino-backups.sh -> /usr/local/sbin/sweep-domino-backups.sh (chmod 0700)
Configuration (domino_ca.sls):
- fingerprints: Verify SHA-256 against vendor advisory.
- Paths & Python: Set windows_python (C:\Python315\python.exe) and linux_python.
- Keystore: Default storepass is changeit (PKCS12).
- Master: Enable file_recv: True in /etc/salt/master temporarily. Ensure /srv/backups has sufficient space.
- Inventory Check: Run to confirm registry paths and domino grain: salt '*' cmd.run 'reg query "HKLM\SOFTWARE\HCL\Domino" /v Path' python_shell=True
- Windows Minions: Install Python 3.15.2 to C:\Python315. Verify: salt '' cmd.run 'C:\Python315\python.exe -c "import encodings, sys; print(sys.version)"' python_shell=True
- Domino State: Domino must be running (listening on TCP 1352) for the tool to allow modifications.
Run hosts sequentially (one at a time, no parallel batches).
Step 1: Confirm Host is Idle salt '' saltutil.running --out=json Expected: {"": []} (Clear stuck jobs via saltutil.kill_job if needed).
Step 2: Run the Rollout Script sudo /usr/local/sbin/domino-rollout.sh The script steps through gates interactively:
- Audit: Lists keystores, entry counts, and matches.
- G3-Only Gate: Stops if any match is not the G3 root.
- Backup Pass (y): Copies, verifies, sidecars, and pushes keystores to master.
- Sweep (y): Moves pushed files to /srv/backups// (failed=0 required).
- Dry Run: Validates match counts.
- Apply (y): Removes G3 aliases.
- Sweep Again (y): Moves apply backups to master (failed=0 required).
- Post-Audit: Verifies matches=0 and Domino JVM size decreased by 1.
- Restart Prompt: Answer y only after restarting Domino and verifying health.
Step 4: Final Audit & Close-Out sudo salt '' state.apply states.domino_ca.audit --timeout=600 --out=json > final-.json sudo /usr/local/sbin/sweep-domino-backups.sh After fleet verification (matches=0 everywhere), disable file transfers on the Master: sudo sed -i 's/^file_recv: True/file_recv: False/' /etc/salt/master sudo systemctl restart salt-master Archive /srv/backups off-host and verify hashes.
4. Special Scenarios & Pitfalls
- Stopped Servers: If Domino is down (port 1352 closed), the tool blocks modifications to protect live states. Manually invoke keytool via a Salt PowerShell script after backing up, then verify with audit.
- Docker Domino: Use states.domino_ca.docker. Rebuild the container image without the G3 root, otherwise container recreation restores the certificate from the image layer.
- Windows Paths & Quoting: Use .ps1 files served via Salt for paths containing spaces (e.g., C:\Program Files).
October 25 2026 Sunday
PHP2Shell and hidden PHP applications and services - Salt to the rescue
I am a huge fan of Salt and its ability to transform an organization with its infrastructure-as-code capabilities. Yes, yes, it's owned by Broadcom, a company whose popularity is only slightly higher than Ebola, but they haven't effed it up quite yet. Even so, if you don't have an IaC solution in your stack, Salt's open-source model gives you staggering options.
I won't go into a deep dive on how Salt works here, except to say it's a master-minion relationship. The Salt Master server pushes configs (known as states) to the Salt minions under its purview, which can include enforcing firewalls, sending dynamic files using Jinja, and locking down endpoints. The control is practically endless.
But the real magic isn't just pushing configs; it's the Master's ability to actively query the minion fleet. And herein lies its true power over traditional agentless vulnerability scanners and static CMDBs. Standard security tools rely blindly on known OS package manager registries (rpm, dpkg, or Windows Installer databases) to inventory installed software. If a runtime or application is dropped in via a manual ZIP extraction, compiled from source, or tucked away inside a container layer, standard tools are completely blind to it. Salt turns your entire server fleet into an active, programmable query engine that pierces straight through those blind spots.
Case in point: I knew I had one PHP installation in a network I was using Salt to manage. One. I knew about one. Once I used Salt to query all the server-based minions and audit runtimes, I found an additional two installations. Each of these three PHP installs was vulnerable to PHP2Shell, and two of them had essentially an unknown existence. Remember, PHP may not be installed in a known path, or could be hiding in a container, a Snap application, or a random directory so finding them manually, let alone knowing they're there, is an exercise in futility. Salt doesn't just manage your infrastructure; it forces it to tell you the truth.
Again, I won't go into Salt installation at all here (ask your AI-du-jour), however here are two scripts I had Claude build me to find errant PHP installs in Docker containers and in Snap applications:
# Report the PHP version inside every running container on this minion.
# Output: host|container|image|php_version.
HOST=$(hostname)
for c in $(docker ps -q); do
NAME=$(docker inspect --format '{{.Name}}' "$c" | sed 's#^/##')
IMAGE=$(docker inspect --format '{{.Config.Image}}' "$c")
VER=$(docker exec "$c" sh -c '
if command -v php >/dev/null 2>&1; then
php -r "echo PHP_VERSION;"
elif command -v php-fpm >/dev/null 2>&1; then
php-fpm -v 2>/dev/null | head -1
else
echo NOT_FOUND
fi' 2>/dev/null)
echo "${HOST}|${NAME}|${IMAGE}|${VER:-EXEC_FAILED}"
done
To run this a Minion grain named Role is expected to contain "docker":
While none were found in the above containers (NOT_FOUND), we now *know* they don't include a hidden PHP install. No guessing, just knowledge.
Here is a Linux shell script that Salt can run to peer into Snap applications, such as Netcloud:
Which when ran will report any PHP versions lurking in Snap installs, and this one hits pay dirt as 8.2.26 is vulnerable to PHP2Shell:
Gives this output after scouring the minions:
Salt can do so much more than this, but this goes to show the kind of power it offers when you start to think outside the box.
I won't go into a deep dive on how Salt works here, except to say it's a master-minion relationship. The Salt Master server pushes configs (known as states) to the Salt minions under its purview, which can include enforcing firewalls, sending dynamic files using Jinja, and locking down endpoints. The control is practically endless.
But the real magic isn't just pushing configs; it's the Master's ability to actively query the minion fleet. And herein lies its true power over traditional agentless vulnerability scanners and static CMDBs. Standard security tools rely blindly on known OS package manager registries (rpm, dpkg, or Windows Installer databases) to inventory installed software. If a runtime or application is dropped in via a manual ZIP extraction, compiled from source, or tucked away inside a container layer, standard tools are completely blind to it. Salt turns your entire server fleet into an active, programmable query engine that pierces straight through those blind spots.
Salt doesn't just manage your infrastructure; it forces it to tell you the truth.
Case in point: I knew I had one PHP installation in a network I was using Salt to manage. One. I knew about one. Once I used Salt to query all the server-based minions and audit runtimes, I found an additional two installations. Each of these three PHP installs was vulnerable to PHP2Shell, and two of them had essentially an unknown existence. Remember, PHP may not be installed in a known path, or could be hiding in a container, a Snap application, or a random directory so finding them manually, let alone knowing they're there, is an exercise in futility. Salt doesn't just manage your infrastructure; it forces it to tell you the truth.
Again, I won't go into Salt installation at all here (ask your AI-du-jour), however here are two scripts I had Claude build me to find errant PHP installs in Docker containers and in Snap applications:
# Report the PHP version inside every running container on this minion.
# Output: host|container|image|php_version.
HOST=$(hostname)
for c in $(docker ps -q); do
NAME=$(docker inspect --format '{{.Name}}' "$c" | sed 's#^/##')
IMAGE=$(docker inspect --format '{{.Config.Image}}' "$c")
VER=$(docker exec "$c" sh -c '
if command -v php >/dev/null 2>&1; then
php -r "echo PHP_VERSION;"
elif command -v php-fpm >/dev/null 2>&1; then
php-fpm -v 2>/dev/null | head -1
else
echo NOT_FOUND
fi' 2>/dev/null)
echo "${HOST}|${NAME}|${IMAGE}|${VER:-EXEC_FAILED}"
done
To run this a Minion grain named Role is expected to contain "docker":
[root@saltmaster user@hostname]# salt -G 'roles:docker' cmd.script salt://scripts/php_in_containers.sh --out=txt
aichat.STS.LOCAL: {'pid': 12054, 'retcode': 0, 'stdout': 'aichat.sts.local|open-webui|ghcr.io/open-webui/open-webui:main|NOT_FOUND\naichat.sts.local|llama-phi4-mini|ghcr.io/ggml-org/llama.cpp:server-cuda|NOT_FOUND\naichat.sts.local|comfyui|ghcr.io/ai-dock/comfyui:latest-cuda|NOT_FOUND', 'stderr': ''}
someproxy: {'pid': 188039, 'retcode': 0, 'stdout': 'someproxy|netdisco-backend|netdisco/netdisco:latest-backend|NOT_FOUND\nsomeproxy|netdisco-web|netdisco/netdisco:latest-web|NOT_FOUND\nsomeproxy|netdisco-postgresql|netdisco/netdisco:latest-postgresql|NOT_FOUND', 'stderr': ''}While none were found in the above containers (NOT_FOUND), we now *know* they don't include a hidden PHP install. No guessing, just knowledge.
Here is a Linux shell script that Salt can run to peer into Snap applications, such as Netcloud:
#!/bin/sh
# snap_php.sh
# Reports PHP inside installed snaps. Where a snap has an app wrapper in
# /snap/bin (e.g. nextcloud.occ), PHP runs through `snap run --shell`, which
# supplies the snap's own library environment. Otherwise the script runs any
# PHP binary it finds directly.
# Output: host|snap|method|app_or_path|version
HOST=$(hostname -f 2>/dev/null || hostname)
for current in /snap/*/current; do
[ -d "$current" ] || continue
name=$(basename "$(dirname "$current")")
case "$name" in
core|core18|core20|core22|core24|snapd|bare|bin) continue ;;
esac
# Preferred: run PHP inside the snap's environment through an app wrapper
app=""
[ -x "/snap/bin/${name}.php" ] && app="${name}.php"
[ -z "$app" ] && [ -x "/snap/bin/${name}.occ" ] && app="${name}.occ"
if [ -n "$app" ]; then
v=$(snap run --shell "$app" -c 'php -r "echo PHP_VERSION;"' 2>/dev/null)
echo "${HOST}|${name}|snap-run|${app}|${v:-UNKNOWN}"
continue
fi
# Fallback: run any PHP binary found in the snap directly
find "$current/" -type f \( -name 'php' -o -name 'php[0-9]*' -o -name 'php-fpm*' \) 2>/dev/null |
while read -r bin; do
[ -x "$bin" ] || continue
case "$(basename "$bin")" in
php-fpm*) v=$("$bin" -v 2>/dev/null | head -n 1) ;;
*) v=$("$bin" -r 'echo PHP_VERSION;' 2>/dev/null) ;;
esac
echo "${HOST}|${name}|direct|${bin}|${v:-UNKNOWN}"
done
doneWhich when ran will report any PHP versions lurking in Snap installs, and this one hits pay dirt as 8.2.26 is vulnerable to PHP2Shell:
[root@salt user@host]# salt -G 'roles:nextcloud' cmd.script salt://scripts/php_in_snap.sh --out=txtGives this output after scouring the minions:
my-nextcloud.domain.com: {'pid': 2389795, 'retcode': 0, 'stdout': 'my-nextcloud.domain.com|nextcloud|snap-run|nextcloud.occ|8.2.26', 'stderr': ''} Salt can do so much more than this, but this goes to show the kind of power it offers when you start to think outside the box.
October 11 2025 Saturday
Type 2 Hypervisors and the evils of "E" cores
In this post I want to give some pointers on making VirtualBox (or any Type 2 hypervisor) run “better” on CPUs that use Big-Little CPU architecture.
First and foremost, this information is provided without warranty, guarantee of correctness, or applicability to your individual situation. As always with information, YMMV (your mileage may vary). Again, any attempt you make is at your own risk. I am simply trying to help.
OK, what is Big-Little? It’s an Intel technology (also found on recent AMD CPUs) that divides CPU cores into two tiers: Big (performance or “P” cores) and Little (efficiency or “E” cores). E cores are typically used for background tasks or processes that don’t require immediate attention, such as dormant browser tabs.
My regular desktop PC has an Intel i9-13900T CPU, which features eight P cores and 16 E cores. You can see this by checking the CPU specs from your CPU manufacturer:
https://www.intel.com/content/www/us/en/products/sku/230498/intel-core-i913900t-processor-36m-cache-up-to-5-30-ghz/specifications.html
Now, Windows doesn't really tell you which is which, so in Task Manager, you can’t distinguish. But for anything virtualization, E cores ARE SIMPLY ABYSMAL. If a VM is moved to an E core, it pretty much stops responding. The moral of the story is that VMs need to run on P cores.
“That’s great news, Darren, but how does one accomplish this?” I hear you ask. There are ways. Again, do these at your own risk and YMMV. Personally, I use a program called “Process Lasso”. This lets me tag specific EXE files and processes to ONLY run on P cores and keep them off the dreaded E cores. You can also disable E core in the BIOS settings of most PCs, but that’s overkill (unless no other way works for you).
Process Lasso is available here: https://bitsum.com/
Process Lasso will actually show you the type of cores (E cores have the leaf):
Process Lasso reports I have 16 P cores, but since they are hyperthreaded, I effectively have 8. I’ll let you Google it if you really want to blow your mind, should you really want to know about hyperthreading.
If you download, install, and run Process Lasso as an administrator, you can pin specific applications or processes to specific cores. This is called processor affinity and can be seen here, where some tasks are on CPU 0-15 (P cores) and some are on CPU 0-31, which is every core on the PC, P or E, whichever the OS decides to place them on:
As you can see from the above, I used Process Lasso to tie VirtualBox (and VBoxSVC, and VBoxHeadless) to 0-15, meaning only ever run them on P cores. Once you do this, the performance difference of the running VMs is DRASTIC. I created a profile called “P-cores-only”, but you can at least test it by deselecting “E” cores like this:
Hopefully, after pinning the VirtualBox processes to P cores, your VMs now respond and freeze or crash less often. Again, YMMV.
First and foremost, this information is provided without warranty, guarantee of correctness, or applicability to your individual situation. As always with information, YMMV (your mileage may vary). Again, any attempt you make is at your own risk. I am simply trying to help.
OK, what is Big-Little? It’s an Intel technology (also found on recent AMD CPUs) that divides CPU cores into two tiers: Big (performance or “P” cores) and Little (efficiency or “E” cores). E cores are typically used for background tasks or processes that don’t require immediate attention, such as dormant browser tabs.
My regular desktop PC has an Intel i9-13900T CPU, which features eight P cores and 16 E cores. You can see this by checking the CPU specs from your CPU manufacturer:
https://www.intel.com/content/www/us/en/products/sku/230498/intel-core-i913900t-processor-36m-cache-up-to-5-30-ghz/specifications.html
Now, Windows doesn't really tell you which is which, so in Task Manager, you can’t distinguish. But for anything virtualization, E cores ARE SIMPLY ABYSMAL. If a VM is moved to an E core, it pretty much stops responding. The moral of the story is that VMs need to run on P cores.
“That’s great news, Darren, but how does one accomplish this?” I hear you ask. There are ways. Again, do these at your own risk and YMMV. Personally, I use a program called “Process Lasso”. This lets me tag specific EXE files and processes to ONLY run on P cores and keep them off the dreaded E cores. You can also disable E core in the BIOS settings of most PCs, but that’s overkill (unless no other way works for you).
Process Lasso is available here: https://bitsum.com/
Process Lasso will actually show you the type of cores (E cores have the leaf):
Process Lasso reports I have 16 P cores, but since they are hyperthreaded, I effectively have 8. I’ll let you Google it if you really want to blow your mind, should you really want to know about hyperthreading.
If you download, install, and run Process Lasso as an administrator, you can pin specific applications or processes to specific cores. This is called processor affinity and can be seen here, where some tasks are on CPU 0-15 (P cores) and some are on CPU 0-31, which is every core on the PC, P or E, whichever the OS decides to place them on:
As you can see from the above, I used Process Lasso to tie VirtualBox (and VBoxSVC, and VBoxHeadless) to 0-15, meaning only ever run them on P cores. Once you do this, the performance difference of the running VMs is DRASTIC. I created a profile called “P-cores-only”, but you can at least test it by deselecting “E” cores like this:
Hopefully, after pinning the VirtualBox processes to P cores, your VMs now respond and freeze or crash less often. Again, YMMV.
January 3 2025 Friday
Domino Router bug - seems to also affect server availability index
I was waiting for the other shoe to drop for this bug. Surely the errant code wasn't only in the router task. Well, it seems that it's NOT only the router.
After working with a customer on fail-over issues in a cluster we came across this interesting availability index "issue". On a server patched for the router bug (or that is un-patched server that has not been rebooted) the "show ai" command behaves as expected, the XF, Hits and AI min and max are populated:
However, on a rebooted, un-patched server AI is completely and utterly blank:
So, if you're clustering and using any type load balancing, or restricting of server access using AI, you may see issues on un-patched Domino servers.
After working with a customer on fail-over issues in a cluster we came across this interesting availability index "issue". On a server patched for the router bug (or that is un-patched server that has not been rebooted) the "show ai" command behaves as expected, the XF, Hits and AI min and max are populated:
However, on a rebooted, un-patched server AI is completely and utterly blank:
So, if you're clustering and using any type load balancing, or restricting of server access using AI, you may see issues on un-patched Domino servers.
March 21 2024 Thursday
OpenNTF March 2024 Webinar - Me, today, presenting Domino security
Today (March 21, 2024), although it will be available on the YouTube after, I am presenting as part of the monthly OpenNTF webinar series.
It's at 11am eastern, and details are here:
https://openntf.org/webinars
It's at 11am eastern, and details are here:
https://openntf.org/webinars
September 4 2023 Monday
Collabsphere 2023 - Great new Domino features since 9.0.1 FP8
The first in-person event for a number of years was held at the stunning Chicago Botanic Gardens. Yet another fantastic event by Richard Moy and team (and LeeAnn).
HCL's openness is also still kind of strange to me after all the barren IBM years. Now, if only they can provide public notice of customers having to run DLUA in order to get a renewal.....still, the product side is knocking the ball out the park.
So here the SlideShare link to the pres....
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8-2023-edpptx
HCL's openness is also still kind of strange to me after all the barren IBM years. Now, if only they can provide public notice of customers having to run DLUA in order to get a renewal.....still, the product side is knocking the ball out the park.
So here the SlideShare link to the pres....
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8-2023-edpptx
Darren Duke
|
September 4 2023 03:35:56 PM
|
collabsphere conference domino notes presentations
|
Comments [1]
April 16 2023 Sunday
Ransomware Prevention Part 11 - Let’s talk about Service Accounts
Part 11 - Let's talk about Service Accounts
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Service Accounts. They exist everywhere. Most have common (and scary) attributes such as passwords that haven't changed in a few years, if not a decade or more. Service Account passwords are rarely, if ever, changed because of of the havoc the can be created when they are. And there are many organizations that have no earthly idea where or how many times these accounts are used. Finally many are local admins on server, or (queue scary music) are domain admin level accounts.
Given any of the above attributes, let alone several of those attributes being present in a single service account, it should come as no surprised that they represent an adversary hitting a gold mine if the can compromise one of them. So how does one protect these God level accounts? Read on.....
In all my years of doing this, I have yet to find an actual reason to have any service account listed as a domain admin. Zero. Nada. Ziltch. If you have a service account in Domain Admins you are simply doing your job wrong.
Managed Service Accounts
The preferred way to create, manage and use service accounts is utilize Managed Service Accounts (MSA). These MSA accounts come in two distinct flavors, stand-alone MSA accounts (sMSA aka MSA), and group MSA accounts (gMSA) and were first introduced in Windows 2008 R2 with gMSA accounts being added with 2012. The only significant difference between the two types is that a sMSA account can only ever be used on a named, single server. it cannot be "assigned for use" on two servers at the same time (note, I said servers, not services!). gMSA accounts on the other hand can get used across several names servers, so a shared account if you will.
For security reasons, sMSA accounts should always be your default choice. gMSA accounts have specific use cases, the one I see the most is using a single gMSA account on several ADFS servers in an ADFS farm.
MSA accounts in general address many, if not all, of the issues with traditional service accounts, namely:
"Darren, this sounds perfect!" Well, yes and no.
That being said, MSA accounts can and should be used anywhere and everywhere you can. For management of MSA accounts, ManageEngine have a helpful free tool available here: https://www.manageengine.com/products/free-windows-active-directory-tools/free-active-directory-service-account-management-reporting-tool.html. Microsoft has detailed MSA documentation and a quick Google will show you how to set them up.
Traditional Service Accounts
As mentioned above, you may locate services that simply won't work with MSA accounts. If you have those then you're left with the traditional service account way, which is simply a user account. These accounts, while nowhere near as secure as MSA accounts are, can have increased security.
Traditional service accounts should be prevented from interactive logins. While MSA accounts have this prevention enabled by default, traditional service accounts need to be set up for this via a GPO
Traditional service accounts need long, complex passwords. I'd look at a minimum of 32 characters.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Service Accounts. They exist everywhere. Most have common (and scary) attributes such as passwords that haven't changed in a few years, if not a decade or more. Service Account passwords are rarely, if ever, changed because of of the havoc the can be created when they are. And there are many organizations that have no earthly idea where or how many times these accounts are used. Finally many are local admins on server, or (queue scary music) are domain admin level accounts.
Service Account passwords are rarely, if ever, changed
Given any of the above attributes, let alone several of those attributes being present in a single service account, it should come as no surprised that they represent an adversary hitting a gold mine if the can compromise one of them. So how does one protect these God level accounts? Read on.....
In all my years of doing this, I have yet to find an actual reason to have any service account listed as a domain admin. Zero. Nada. Ziltch. If you have a service account in Domain Admins you are simply doing your job wrong.
Service Accounts should never, ever be a Domain Admin
Managed Service Accounts
The preferred way to create, manage and use service accounts is utilize Managed Service Accounts (MSA). These MSA accounts come in two distinct flavors, stand-alone MSA accounts (sMSA aka MSA), and group MSA accounts (gMSA) and were first introduced in Windows 2008 R2 with gMSA accounts being added with 2012. The only significant difference between the two types is that a sMSA account can only ever be used on a named, single server. it cannot be "assigned for use" on two servers at the same time (note, I said servers, not services!). gMSA accounts on the other hand can get used across several names servers, so a shared account if you will.
For security reasons, sMSA accounts should always be your default choice. gMSA accounts have specific use cases, the one I see the most is using a single gMSA account on several ADFS servers in an ADFS farm.
MSA accounts in general address many, if not all, of the issues with traditional service accounts, namely:
- Automatic changing of passwords by AD ever 30 days or whatever your AD machine password expiration is. NO manual intervention is needed.
- Password complexity is high, 240 bytes make brute forcing difficult
- MSA accounts have to be specifically assigned to a server before a server can use it.
- MSA accounts are prevented from interactive user logins.
"Darren, this sounds perfect!" Well, yes and no.
- Not every service you have running can use MSA accounts. It's gotten better over the years, but it's still trial and error.
- They are an absolute pain in the backside to create and manage the first time you try.
- You still need to reduce the MSA account to least privileged access.
That being said, MSA accounts can and should be used anywhere and everywhere you can. For management of MSA accounts, ManageEngine have a helpful free tool available here: https://www.manageengine.com/products/free-windows-active-directory-tools/free-active-directory-service-account-management-reporting-tool.html. Microsoft has detailed MSA documentation and a quick Google will show you how to set them up.
Traditional Service Accounts
As mentioned above, you may locate services that simply won't work with MSA accounts. If you have those then you're left with the traditional service account way, which is simply a user account. These accounts, while nowhere near as secure as MSA accounts are, can have increased security.
Traditional service accounts should be prevented from interactive logins. While MSA accounts have this prevention enabled by default, traditional service accounts need to be set up for this via a GPO
Traditional service accounts need long, complex passwords. I'd look at a minimum of 32 characters.
December 9 2022 Friday
Domino 12.0.2 - no support for Windows 2016? Really?
While talking with a customer today I was informed HCL told them Windows Server 2016 wasn't supported for Domino 12.0,2 (apparently due to some technical limitation). I thought there was no way this was correct, so off I go to HCL's support web site, and low and behold, no Windows 2016 listed as a supported OS!
From https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101447
From https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101447
October 21 2022 Friday
My Collabsphere 2022 presentation - Great new Domino features since 9.0.1FP8
It's available over on slideshare. Link is below.
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8pptx
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8pptx
June 30 2022 Thursday
Domino 12.0.2 adds VSS backup support
Since BackupExec ceased support for Domino backup APIs after v14 there have been very few backup utilities that integrate with Domino natively. The fix was always for IBM to add VSS support to Windows Domino installs (the vast, vast majority of installs I see *are* on Windows). But IBM (along with 1000's of other fixes they should have and could have done) choose not to.
HCL have finally fixed this oversight (and by oversight I mean complete dereliction of duty from IBM). I fully admit I was worried when HCL went all chips in and bought it all from IBM, but boy have they been adding stuff that has been sorely missing from the product. VSS support included.
The best part is that (for backup at least, restores are a tad more finicky so be sure to read the docs) there is no setup on your side once 12.0.2 ships and you install it. It is available today in FlexNet as a preview release, not gold code yet so you've been warned. Here is what happens on the Domino side (I have logging turned up) when Veeam backs up my 12.0.2 Windows server with Veeam "application aware processing" turned on:
HCL have finally fixed this oversight (and by oversight I mean complete dereliction of duty from IBM). I fully admit I was worried when HCL went all chips in and bought it all from IBM, but boy have they been adding stuff that has been sorely missing from the product. VSS support included.
The best part is that (for backup at least, restores are a tad more finicky so be sure to read the docs) there is no setup on your side once 12.0.2 ships and you install it. It is available today in FlexNet as a preview release, not gold code yet so you've been warned. Here is what happens on the Domino side (I have logging turned up) when Veeam backs up my 12.0.2 Windows server with Veeam "application aware processing" turned on:
