Well, any file really, but this script is targeted at keeping the configuration files, ssl certificates and modsecurity rules in sync across a cluster of reverse proxy servers. While this script is targeted at Apache (or Nginx) on Ubuntu a few changed lines and it should work on any distro, If any changes are detected from the primary then the backup apache server is restarted.

For this particular example I have two apache reverse proxies running in HA (using pacemaker, et al), and I only every want to edit the configuration files on one server and have them copied to the other. No more forgetting to copy apache configs (which you only ever find out about when one goes down).

As usual, use at your own risk and YMMV unless you are paying me. It's far from perfect, so feel free change as you will.

Here we go:
  1. Never test in production. Just saying.
  2. There are two servers, primary and backup. Primary is the source of the files, backup is the target. So any changes on primary will be reflected on backup at whatever cron frequency you see fit (I use 5 minutes).
  3. I'm using a user called rsyuser. This user is created on both servers and has read and write access to all the files/folders in question at both ends (I created a group and added this new user and root to it).
  4. The rsyuser user also has a SSH key created in order to do this password-less. This SSH key is created on  the backup server using this command on the backup server logged in as rsyuser ssh-keygen -t rsa -b 2048 (make sure just to hit enter, and do not enter a password). Once created, copy both files to the root users ssh folder (/root/.ssh) and make the files you just copied owned by rsyuser and leave the group as root .
  5. The public key of this user is copied to the primary server with this command from the backup
    ssh-copy-id -i /home/rsyuser/.ssh/id_rsa.pub rsyuser@your_primary_server_ip_address
  6. You probably want to make sure you can now copy between the servers by manually running rsync from the backup and get files from the primary before you go any further.....
  7. Change the script to your IP addresses, services, and user and  place the script below in /home/rsyuser/ and make it executable (+x).
# while technically this is for Ubuntu servers,
# you can easily change the paths to match CentOS, RHEL, etc

# Darren Duke

# Provide as-is. No warranty or guarantee is implied nor given

#Change the remoteIP to be your primary server IP address

#Change this to your user that you are using

#for nginx change the line below to 'nginx'


# Add to each array the folders you wish to sync
sourceFilesArray=("/etc/apache2/sites-available/*.conf" "/etc/apache2/sites-enabled/*.conf" "/etc/modsecurity/modsecurity.conf" "/etc/apache2/sts_ssl/2018/*" "/etc/apache2/sts_ssl/*")

targetFilesArray=("/etc/apache2/sites-available/" "/etc/apache2/sites-enabled/" "/etc/modsecurity/modsecurity.conf" "/etc/apache2/sts_ssl/2018/" "/etc/apache2/sts_ssl/")



for i in "${sourceFilesArray[@]}"


RSYNC_COMMAND=$(rsync -alizhe ssh $rsyncusername@$remoteIP:$i ${targetFilesArray[count]})

if [ $? -eq 0 ]; then

# Connected, we're doing something right!

       echo $i

if [ -n "${RSYNC_COMMAND}" ]; then

        echo 'in -n, so set restartApache so later it can be restarted'



           echo 'in else, so no changes were made'



# Error....hum!

       exit 1


echo $count



echo "restart $serviceName: " $restartApache
if [ $restartApache -eq 1 ]; then

service $serviceName restart

echo "$serviceName restarted";


As you will notice there are two arrays in the script, one lists the files you want to copy from the primary (handily called sourceFilesArray), the other is this target folder of file that sourceArray{x] will get copied to (also handily called targetFilesArray). One of the really cool things about rsync (the engine behind all of this) is that is will take a wildcard and expand it, real-time as the script is executing. That is why you will see some sourceArray elements having a wildcard (*) in them. This saves you a ton of array elements.

Once you have it working the way you want, simply create a cronjob on the backup server:

*/2     *       *       *       *       /home/rsyuser/rsync_apache.sh
Darren Duke   |   November 20 2018 03:49:08 AM   |    apache  nginx  security  linux    |   Comments [0]

October 15 2018 Monday

Domino 10 adds (nee TDI) IBM SDI 7.2

While downloading Domino 10 the other day I also saw this in the "Domino 10" download list I had searched for:

Image:Domino 10 adds (nee TDI) IBM SDI 7.2

In previous versions of Domino you only had entitlement rights to the now aging Tivoli Directory Integrator 7.1.1. I had asked a while back (maybe 18 months) about access to (the now then renamed to) IBM Security Directory Integrator 7.2 and was told there were no plans. Well now that we have a whole new Domino release someone realized that was a dumb decision and IBM have revved the entitlement to 7.2.

Once you get TDS 7.2 go get the latest fix pack too (as the time of writing FP5) from
IBM Fix Central:

Image:Domino 10 adds (nee TDI) IBM SDI 7.2
Darren Duke   |   October 15 2018 10:18:26 AM   |    domino  tdi  10    |   Comments [0]

September 27 2018 Thursday

iNotes, ADFS and 2FA - the movie

Quite a while back (3 years!!!) I demo'd a completely password-less Notes client. The logical follow on to this is iNotes. But maybe I want more a more secure iNotes, say with 2FA maybe? Well using a product called Duo and ADFS and Domino we are able to do this.

For assistance with this contact
Lisa. Duo is here.
Darren Duke   |   September 27 2018 06:20:49 AM   |    domino  inotes  security  duo  saml  wfl    |   Comments [0]

September 19 2018 Wednesday

Domino logging to Syslog

First off....I know right, a *blog post*! Who knew.......

So you have Domino.

And you have a syslog server where everything except Domino is logged to.

You want Domino to play along with everything else. What can you do?

For starters there is this event handler type in events4.nsf:

Image:Domino logging to Syslog

But in typical IBM fashion the documentation for the above is practically non-existent on how this works . Nor does there seem to be a way to specify a remote syslog server. So I would presume (and dear reader feel free to leave a comment with links, etc if you know them) that this only works with a Domino server running on Linux and AIX. Now with Linux I could syslog locally and then punt my logs to another syslog server. So theoretically this could do what we want on Linux platforms.

But what about Windows I hear you ask? Well here we need to use a combination of things. Specifically this event handler:

Image:Domino logging to Syslog

That puts my Domino server logs into the Windows Event viewer like so:

Image:Domino logging to Syslog

First stage ignition completed! Now to punt this over to a syslog server. For this I use
NXLog Communty Edition installed on the Windows servers and configure this to throw Windows Events out to my syslog server (in my case Nagios Log Server which is free for <500MB/day although I have used GrayLog before as well that has an free version and an Enterprise with is free <5GB/day and also IBM's QRadar with also has a community edition).

With NXLog conifgured to send Windows Event Logs to my syslog server I now have my Domino logs (from Windows Domino servers at least) in an infinitely more usable format (any my security folks maybe happier too. Final stage ignition complete!

So from this (Domino console and log.nsf):

Image:Domino logging to Syslog

To this (Nagios Log Server):

Image:Domino logging to Syslog

And with the use a quick elastic search, all my Domino server logs in one place:

Image:Domino logging to Syslog

So there you have it. Domino logging to a syslog server.
Darren Duke   |   September 19 2018 05:42:29 AM   |    domino    |   Comments [4]

Here's my CollabSphere 2018 presentation from the conference in Ann Arbor, MI. Again, a huge thank you to Richard and Leann Moy for organizing another spectacular conference.

Darren Duke   |   July 26 2018 10:28:18 AM   |    domino  mwlug  collabsphere  presentation  security    |   Comments [1]

June 12 2018 Tuesday

10 Year Blogaversary

I really wasn't keeping track of this, but I've had some issues with the blog lately and went down the archive section and low and behold the first post was on July 2nd 2008. 10 fricken years. An entire decade of ranting and raving and hopefully occasionally useful stuff......That's not to say I've really kept up the pace from the early years (in fact the frequency has plummeted), nor has it been all smelling of roses (more on that later).

So what have the last 1,685 posts taught me? Well, first off I've never moved the blog off of Domino and a true testament to the genius that is Steve Castledine (I do find the occasional bug but I can usually work around that). The UI was changed back in 2013 (thanks to Sedar's work with bootstrap) and the location move to Prominic but it's still the same NSF and same data. I wonder how many different blog platforms Stuart has tried in the last decade? It is behind NginX these days and the blog was SSL'd a while back.  

So what else has happened here since that first posting? Well....
  • 1,685 blog posts.
  • 2 podcasts have been born.
  • 1 podcast has been put to rest.
  • 8 end of year (snark) reviews have been published.
  • At least 13 conference presentations have been given (11 are linked on the blog), but that number is probably closer to 17.
  • There are 13 drafts that never saw the light of day. One or two are pretty scathing.

Now for some interesting product things....

First BlackBerry post - July 8, 2008
Last BlackBerry post - March 9, 2011

First iPhone post - March 9, 2011 (the exact same post as the last BlackBerry one!)

First Quickr post - August 21, 2008
Last Quickr post - August 25, 2017 (almost a year after IBM support ended)

First Symphony post - June 11, 2009
Last Symphony post - June 11, 2009 (only one post, a tad prescient)

First Let's Encrypt post - October 1, 2015

First Veeam post - June 15, 2009

I don't track any other stats any more, like page reads and that kind of stuff (if there is any Google Analytics stuff in here, it's just because I've not gotten around to taking it out...I'm egotistical and a tad narcissistic but not *that* egotistical and narcissistic), but you can tell the popular posts usually by either comments on the blog or off-line messages like email or phone calls. So what was the most active blog posts? There are two and they are both IBM related, and remember when I said above it was not all plain sailing....

The first was a lamenting, somewhat ranty yet absolutely correct post where IBM were all fired up about mail.next and I mentioned that mail.now should take precedence. This culminated in IBM threatening to revoke (my non-existent) IBM Champion status and I believe they also threatened to sue STS and boot us out of the business partner program. Why you may be wondering what caused all this kerfuffle? This sentence, “no one in my organization will tell me the truth because I’m a vindictive asshole, so don’t cross me or I will end your career" in reference to what an IBM executive may think to themselves.

Now, IBMers usually won't call me directly, so they'll go via Lisa. Call they did, and all they seemed to want to know was who possibly could this IBM executive be? I wasn't anyone I actually know personally so if you've ever had a conversation with me, it's not you. I also wasn't any of the names mentioned in the calls Lisa got involved in. Boy was she pissed at me, but I think IBM using threats pissed her off more. Buy her a drink at the next conference you see her at and get the full story.

The second was also IBM Champion related. It is also one of the funniest things I've ever done "form-submission-wise". I filled in an IBM Champion self nomination for myself and actually submitted it. You have to see it to really get it. I still chortle to myself on the odd occasion I go read that modern day classic.  

Honorable mentions also go to the posts about IBM killing 9.0.2 and it's slightly older sibling 9.0.2 Where for art thou?.

What other upsides are there to this decade long journey? Well, I've not written or updated a resume/CV for years. The hiring process for new customers goes something like this:

Step 1 - Customer Google's something. Google kindly returns one of my blog entries.
Step 2 - The customer contacts STS asking if we know this stuff. Lisa searches the blog, sends several articles on the topic to the customer.
Step 3 - Profit.

Seriously, that's pretty much how this works now, and while we'll never really know how much impact the blog has had to STS revenue, I think it's up there. And Lisa never wanted me to start a blog in the first place! True story. Also a true story, the search was broken in the blog (due to a security HTTP header) and that's how I realized it had been 10 years.

Finally there is nothing quite like Googling something and your own blog post has the answer to the question. It's equal parts effing cool and worrying that you used to know this and no longer even remember writing the answer.

Here's to another 10 years......
Darren Duke   |   June 12 2018 05:10:19 AM   |    misc    |   Comments [4]

It's a three day weekend in the States so I'm busy patching servers and installing fix pack and intermediate fixes to Domino servers. I came across an issue that a customer had mentioned to me a few weeks back while on-site that I see all the time but think nothing of. Basically they stop all the IBM Domino services and any backup agents and they still can't install a FP or IF. They called support and were told to boot into safe mode and do the install (wtf support?).

Here's there error (and we've all seen it);

Image:You’re installing a new fix pack or IF on Domino and Windows won’t let you

So how do you fix this without booting into safe mode (again, wtf support?)? Well, provided you've stopped the IBM Domino services (both of them) and any backup agent servers then the lightly culprit is this little bugger, the Windows Management Instrumentation Service:

Image:You’re installing a new fix pack or IF on Domino and Windows won’t let you

Stop this service (it may restart again, so if it does disable it until the upgrade completes, then re-enable and restart it).

You may see this, if you do click OK, it's not going to hurt anything to stop these services while upgrading:

Image:You’re installing a new fix pack or IF on Domino and Windows won’t let you

Once Windows Management Instrumentation is stopped, voila, Domino upgrades successfully:

Image:You’re installing a new fix pack or IF on Domino and Windows won’t let you

No need to boot into safe mode (again, again, wtf support?).

If you did disable the Windows Management Instrumentation service because it restarted, remember to go back and re-enable it and restart it.

Darren Duke   |   May 28 2018 04:38:43 AM   |    domino  upgrade    |   Comments [3]

If you are using Apache as either a HTTP server or a reverse proxy (like my free one) then it is relatively simple to disable TLS1.0 and TLS1.1. Basically this should do it:

SSLProtocol TLSv1.2

Except if you're using SNI
. With Apache you may see a strange phenomenon where no matter what you do to a virtual server config TLS1.0 and TLS1.1 still remain active. This has to do with an OpenSSL bug outlined here

What this bug means is that if you have multiple virtual hosts (either in a single config or as multiple configs), Apache can only use the SSLProtocol setting from the first host it loads. Which one will load first? No idea. But if you change each and every virtual host to the above an reload Apache, voila, TLS1.0 and TLS1.1 are now disabled.
Darren Duke   |   March 12 2018 10:15:02 AM   |    apache  security  ssl    |   Comments [0]

We've all heard the swirl around IBM's reversal on releasing a Domino 10 (a really good reversal of a really bad decision), but let's not forget until 10 ships it's still feature packs that distribute new features and fixes and feature pack 10 (FP10) is a biggie.....

FP10 completes IBM's promise to finally provide Java 8 in both the client and DDE (new in FP10) and on the server (release in FP8...I think....). So because of this quite massive change (and Eclipse has been updated to 4 dot something from 3 dot something) I would test this more than I generally would especially if you are doing anything Ecplisy or Javery stuff in your clients or apps.

Also if you are updating your Traveler server to Domino 9.0.1 FP10 be sure to install Traveler first!. You have been warned.

Get your FP10 here

Domino :

Notes :


It is also worth pointing out that with FP9 there were new templates released for mail9, pubnames and pernames. These are not installed when you upgrade to FP9 or FP10 and need to be installed separately. These add some new features like running mail rules on existing mail.
Darren Duke   |   January 31 2018 07:17:44 AM   |    901  domino  lotus  lotus notes  notes    |   Comments [4]

Last month, in December 2017 a new (well old, but new) vulnerability was discovered in TLS, the ROBOT attack (Return Of Bleichenbacher's Oracle Threat) and yes, your Domino servers are probably susceptible to it. To avoid re-posting everything from that article go read it them come back.

Your back? OK. So you need to disable any and all RSA encryption ciphers. So here goes (all tests were done on a 9.0.1 FP9 server).....

Before a custom SSLCipherSpec:

Image:How to prevent ROBOT (Return Of Bleichenbacher’s Oracle Threat) on Domino servers

If you look at the ciphers, sure enough there are RSA ciphers in there (TLS_RSA_xxxxxx) for both TLS 1.2 and TLS 1.0 respectively:

Image:How to prevent ROBOT (Return Of Bleichenbacher’s Oracle Threat) on Domino servers

Image:How to prevent ROBOT (Return Of Bleichenbacher’s Oracle Threat) on Domino servers

OK, Houston we have a problem. To rectify it use this SSLCipherSpec


Note, I added this via the Domino Console):

set config SSLCIPHERSPEC= C030009FC02F009EC028006BC0140039C0270067C013C0140039C013

Results of the SSL Test after restarting HTTP:

Image:How to prevent ROBOT (Return Of Bleichenbacher’s Oracle Threat) on Domino servers

And the errant ciphers are gone:

Image:How to prevent ROBOT (Return Of Bleichenbacher’s Oracle Threat) on Domino servers
Darren Duke   |   January 15 2018 01:51:20 PM   |    domino  domino security  security    |   Comments [3]