Veeam v11 changed from a powershell snap-in to a powershell module. As such it broke everything.

In v10 and earlier Veeam PS you probably loaded it something like this:



$snaps = Get-PSSnapin
foreach($snap in $snaps){if($snap.name -eq "VeeamPSSnapin"){$exflag = 1}}
if($exflag -ne 1){
      Add-PSSnapin -name VeeamPSSnapin -erroraction silentlycontinue
      if($error -ne $null){write-host "CRITICAL - Could not load Veeam snapin";exit 2}
}


....rest of your existing code


Well, Veeam decided "due to popular demand" to change this and broke everything. After struggling for a few days to figure out the secret sauce (I had a lot of trouble invoking the new module non-interactively) I hit pay dirt.


For Veeam v11 simply change the above code to this:



$VeeamPath = “C:\Program Files\Veeam\Backup and Replication\Console”
$env:PSPath = $env:PSPath + “$([System.IO.Path]::PathSeparator)$VeeamPath”
Import-Module -DisableNameChecking Veeam.Backup.PowerShell

Connect-VBRServer


...rest of your existing code


Now, I don't have any error checking code in there yet, but this may help some people when the upgrade.

FYI the secret sauce for non-interactive was adding the explicit  Veeam Path (the top two lines of the new code). If your module install path is different adjust accordingly. You can probably achieve the same "fix" by manually adding the Veeam Console path to the local environment variable "PSModulePath" on the Veeam server, I haven't tried that yet and the code way of adding the path is more flexible when I'm copying code around to different systems:


Image:Upgraded to Veeam v11 and now all your Veeam related powershell scripts are broke?
Darren Duke   |   April 23 2021 03:57:25 AM   |    veeam  security    |   Comments [0]

Back by popular demand......

On Friday 29th of January I will be hosting an hour long webinar to provide some real-world proven tips and tricks on preventing and surviving a ransomware attack.

It's at 1pm Eastern time. To reserve a spot simply email info@simplified-tech.com.


It could save you $100,000's.
Darren Duke   |   January 22 2021 08:39:26 AM   |    security    |   Comments [0]

This Friday 11th of December I will be hosting an hour long webinar to provide some real-world proven tips and tricks on preventing and surviving a ransomware attack.

It's at 1pm Eastern time. To reserve a spot simply email info@simplified-tech.com.

It could save you $100,000's.
Darren Duke   |   December 9 2020 10:35:43 AM   |    security    |   Comments [0]

I do a fair bit of ADFS so I know my way around it pretty well. But when I have to delve into the world of ADFS Transforms I wake up in cold sweats. Here's the business case....we need to take the SAMAccount name of a user and truncate it to only the first eight characters in length before passing it off as the NameID to a 3rd party application. So truncate jhowstoday to jhowstod. Simple right?

Why the eight character limit I hear you ask? No idea. Must have been written in DOS.


I thought this would be pretty straight forward, but no. There are some examples in the internet about truncating in an ADFS transform using regex but none, absolutely none worked for me. After coming up with such a blank and then flailing around from more then a handful of hours I thought I would add it here in case anyone else has to stumble through the hell I just put myself through.....


Also, it doesn't seem to appear that SAMAccount name is exposed in ADFS as an incoming option, but UPN is (at least in 2012 R2). So now I need two transform rules, so yay!


OK, here's the finished product, we'll pull apart each claim. The transform starts and the top and works down, so "1" is first and "2" is second and last. "2" is where the NameID will pop out of....

Image:ADFS Transformations how does one truncate? - also known as Regular Expressions are evil

Rule 1 - Extract SAMAccountName

c:[Type == "
http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
=> add(store = "Active Directory", types = ("temp:samaccountname"), query = ";sAMAccountName;{0}", param = c.Value);



In the above I essentially do an AD query and store the SAMAccount name in temp:samaccountname to I can get it in rule 2.


Rule 2 - Truncate to <= 8

c:[Type == "temp:samaccountname"]

=> issue(Type = "
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier",
Value = RegexReplace(c.Value, "^([a-zA-Z]{8})+.+", "$1"));


Rule 2 is interesting. I take the previously stored temp:samaccountname and pass it through the ADFS transform regex engine. I'm presuming your SAMAccount name is only letters here [a-zA-Z] so adjust accordingly. The regex (I hate regex) actually matches the first group (in regex that's everything between the parenthesis in the query so [a-zA-Z]{8}). So in the
jhowstoday example it matches the jhowstod. It then takes that group 1 value, jhowstod. and replaces the entire  original text, the $1 in that command. Finally the returned regex is assigned as a NameID and passed out as there are no more rules.

It's easier to visualize using an online regex editor. I use
https://regex101.com/. Here is the breakdown:

Image:ADFS Transformations how does one truncate? - also known as Regular Expressions are evil

It looks pretty simple, but trust me, it was far from. Hopefully this will someday help someone.
Darren Duke   |   September 25 2020 03:29:53 PM   |    securitty  adfs  saml    |   Comments [0]

As most will have hopefully read by now the ZEROLOGON vulnerability in Windows (CVE-2020-1472) is pretty wild. Basically by passing Windows netlogon process bypassing a series of zeros to it. Fun times!

Microsoft did issue a patch in their August 2020 updates here
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1472, however I'm skeptical that people actually read the FAQ at the bottom of the article:

Image:ZEROLOGON and why you may not actually be protected

Notice that link in the highlighted section? Here it is
https://support.microsoft.com/en-us/help/4557222/how-to-manage-the-changes-in-netlogon-secure-channel-connections-assoc

Which leads us to this nugget:


Image:ZEROLOGON and why you may not actually be protected

So your Domain Controllers are not fully protected? Admittedly this is a pretty confusing list of bullet points but it does seem to suggest the patch reports as opposed to enforces non-Windows machines. My assumption seems to be backed up by this paragraph below that mentions enforcement will being with the February 2021 patch Tuesday:


Image:ZEROLOGON and why you may not actually be protected
OK, so you installed the patch and your secure right? From Windows devices? Looks like. From non-Windows? I don't believe so unless you add this registry entry otherwise you are flying pants down until February when Microsoft will do it for you:


Image:ZEROLOGON and why you may not actually be protected
Darren Duke   |   September 25 2020 11:08:39 AM   |    security    |   Comments [0]

UPDATE 07/24/2020 - if you are still having issues after these fixes, head over to Tom Hillebrand's blog http://www.surfdomino.com/web.nsf/pages/ST11FP1

I'll admit it. I wasn't exactly sure where the migration to HCL was going to end up. From a features perspective and moving the stagnant products forward it have been nothing short of astonishing. From a documentation and sales viewpoint it has been anything but. Point in case Sametime v11.

Sametime 11 was the promised land. Sametime without the need for DB2 and WAS. Sametime installable on a single server again. Sametime with persistent chat and multi device support. Enough to make even the customers still on 8.0.2 (the last non-WAS version) to stand up like a meerkat and take notice. So here's my tome.....


As a general rule when I have server hanging off the full internet I put it in a DMZ. Additionally whenever possible when I put a server in the DMZ it's going to be Linux. So Domino and Sametime on Linux it is. CentOS 8 to be exact.  It is here the Homer-esque story begins....


I'll start off with this statement. IBM documentation was always woeful. IBM was a veritable Mark Twain when compared to HCL. Good Lord. It is nothing short of atrocious. Especially for Sametime. What we have here is a failure to communicate.


First the good news, I have just gotten it working after several different attempts and more than 8 hours of trying, Almost a WAS like effort. Now, I'm no Linux Deus, but I can get by well enough and boy did I struggle. All because HCL hasn't provided even the bare minimum of documentation for the install process and even more important, the OS pre-requisites need to make the product work.


You get this:


Image:Sametime v11 on CentOS 8....head meet desk

A PDF that is somewhat helpful if you don't know MongoDB (I don't) but from an installation and use knowledge dump it is useless.

So here's how I eventually got the promised land of Sametime v11 to work on CentOS 8. I already had Domino working and installed Sametime and Mongo per the guide. But nada. Nothing. Sametime seemed to start and I was able to log in and while chats can be sent the are blank at the both the recipient and senders clients. Great. In the forum there is mention that this field (not at all mentioned in either in the documentation or added by the installer) in the stconfig database:


Image:Sametime v11 on CentOS 8....head meet desk

Upon install the Chat Log Flag below is not set to "relax" which is (apparently) required to make it work, but set to "off" (why HCL?). I changed it per the forum post but no change in my tragic outcome of blank messages:


Image:Sametime v11 on CentOS 8....head meet desk

So if it's not a Sametime issue I figured it must be Mongo, as all the chats are stored there to allow for the new persistent chat. Hum. I don't know Mongo and Sametime has radically changed since I used to do it regularly (I stopped at WAS like most customers did). Another forum post that pointed me to a "trace" folder under the DomData. Off I go and find this file:


Image:Sametime v11 on CentOS 8....head meet desk

Upon opening it and looking around I find this nugget when ever I restart Sametime:


Image:Sametime v11 on CentOS 8....head meet desk

Hum, libstchatloggingmongodb won't start? Sametime uses Mongo for persistent chat, so...... OK, so where is it and what is it? The Linux locate (part of mlocate) and ldd (part of the core OS) are your friends. Be sure to run updatedb often for locate to be useful.


First, does libstchatloggingmongodb file exist?

locate libstchatloggingmongodb


Yes:


Image:Sametime v11 on CentOS 8....head meet desk

So the file does exist. Let's find out what the file needs:


ldd /usr/lib64/libstchatloggingmongodb.so


Image:Sametime v11 on CentOS 8....head meet desk

Well, there's a clue "not found" eh?. A fair number of the libraries that libstchatloggingmongodb.so needs are either not installed or not linked to the correct path (in my case both, although mostly the latter).


I then go down each missing file and find out if it's installed (if not figure out which packages need and them installed). Here's an example of the libwinux.so file that was installed but not linked by the Sametime installer (why not HCL?). Note, I changed the Sametime related paths from 11000000 to latest in the hope that a future Domino upgrade won't break all the symlinks again:

locate libwinux.so
ln -s /opt/hcl/domino/notes/latest/linux/libwinux.so /usr/lib64/libwinux.so
ldd libstchatloggingmongodb.so


Image:Sametime v11 on CentOS 8....head meet desk

Rinse and repeat until you have all the missing ones linked or installed. However for the life of me I could not get libssl.so.10 and libcrypto.so.10 fixed until I figured out I needed compat-openssl110
:

yum install compat-openssl110


Image:Sametime v11 on CentOS 8....head meet desk

Once that was installed and all the links established the new ldd command looks like this:


Image:Sametime v11 on CentOS 8....head meet desk

With these in place. Sametime v11 on CentOS is finally working. Now to figure out why the embedded client shows me as "away" while I'm actually typing and why you have to kill an entire chat to get it to populate the "chats" view in Notes.. Fun times!


Image:Sametime v11 on CentOS 8....head meet desk

Oh, and to get file transfer to work was another Herculean effort of editing XML files (the trick is which one and what attribute). But still, Sametime without WAS and DB2 is glorious, It just took a long time to get to that point. Thanks HCL for the software and seemingly giving a crap. Now can we have stellar (or maybe just useful) documentation so everyone can install and use the excellent strides you have made?

I'm sure there is a better way,but the above actually worked for me. so while not exhaustive, here's most of the link commands I used for Sametime v11 FP1 on CentOS 8 after Domino, Sametime and Mongo were installed (and remember to change "off to "relax" in the stconfig):


yum install compat-openssl10

ln -s /opt/hcl/domino/notes/latest/linux/libnotes.so /usr/lib64/libnotes.so

ln -s /opt/hcl/domino/notes/latest/linux/libndgts.so /usr/lib64/libndgts.so

ln -s /opt/hcl/domino/notes/latest/linux/libxmlproc.so /usr/lib64/libxmlproc.so

ln -s /opt/hcl/domino/notes/latest/sticc/libgsk8iccs_64.so /usr/lib64/libgsk8iccs_64.so

ln -s /opt/hcl/domino/notes/latest/linux/libstchatloggingmongodb.so /usr/lib64/libstchatloggingmongodb.so

ln -s /opt/hcl/domino/notes/latest/linux/libmongoclient.so /usr/lib64/libmongoclient.so

ln -s /opt/hcl/domino/notes/latest/linux/libwinux.so /usr/lib64/libwinux.so

ln -s /opt/hcl/domino/notes/latest/linux/libubq.so /usr/lib64/libubq.so

ln -s /opt/hcl/domino/notes/latest/linux/libwinux.so /usr/lib64/libwinux.so

ln -s /opt/hcl/domino/notes/latest/linux/libstts.so /usr/lib64/libstts.so

ln -s /opt/hcl/domino/notes/latest/linux/libboost_chrono.so.1.63.0 /usr/lib64/libboost_chrono.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_date_time.so.1.63.0 /usr/lib64/libboost_date_time.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_filesystem.so.1.63.0 /usr/lib64/libboost_filesystem.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_regex.so.1.63.0 /usr/lib64/libboost_regex.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_system.so.1.63.0 /usr/lib64/libboost_system.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_system.so.1.63.0 /usr/lib64/libboost_system.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/libboost_thread.so.1.63.0 /usr/lib64/libboost_thread.so.1.63.0

ln -s /opt/hcl/domino/notes/latest/linux/STOpenSSL/libssl.so.1.0.0 /usr/lib64/libssl.so.1.0.0
Darren Duke   |   May 29 2020 07:57:55 AM   |    sametime    |   Comments [0]

No meaningful blog posts in ages, then 257 in the space of 3 days. Yeah, COVID quarantine is a killer.

Anyway, apparently ID Vaults stop working after 10 years. Not the best decision ever made but the head shed at IBM when 8.5 shipped. I only discovered this when trying to reset a vaulted password. What's even worse is the error of this type of failure. It's not something useful like:


Your ID Vault Trust Certificate is expired as it lasts for 10 years, see technote xxxxxx



(admittedly Domino related IBM-hosted technotes are about as much use as an orange colored President, but meaningful error messages are)

But no. Not useful. All there was this verbal splat of English:


Server Error: The address book does not contain a cross certificate capable of validating the public key.


Image:ID Vault Trust Certificates expire after 10 years. AKA that was a stupid decision and breaks ID Vault

OK. After a few hours of repeatedly hitting my head against a wall checking every certificate under the sun and some Googling that made me start to doubt my ability in seach-fu I hit pay dirt.
This article from Fabio Di Paola was the solution. Essentially this, Vault Trust Certificates expire after 10 years. Genius decision by the ID Vault creators. Not. Many 8.5 installs are coming up to, or have passed their 10 year anniversary when initial ID Vaults would have been installed, so this may help some folks.

Fabio also has the steps to fix it, but it's basically this:


First check if your Vault certificate is expired if so manually delete your old Vault Trust Certificates from the NAB (say what now Darren????!!!!) then re-add the Organization back to the vault via the Manage Vault action. Viola, error banished, passwords are resettable. Bird sing, sky blue.


See Fabio's article for more information if you need it.
Darren Duke   |   April 15 2020 02:08:12 PM   |    domino security    |   Comments [0]

In yesterday's post about AES NSF local encryption, I quipped that you should be able to locally encrypt a replica with AES 128, but that you'd have to do it manually.  Well, you can't. I probably should have tested this theory like HCL should have tested this feature.

Here's what happens:


Go to an existing database on a server and enable AES 128 encryption (yes, the option is there):


Image:Changing NSF encryption on a server to AES? No, you’re not.

Compact said database:


Image:Changing NSF encryption on a server to AES? No, you’re not.

Take another peak at the encryption settings:


Image:Changing NSF encryption on a server to AES? No, you’re not.

"Strong"? WTF? No matter what I do I can't get an existing on server database to encrypt with the new AES setting.

At first I thought this was a Domino version issue. Nope. Servers are 11.0.1. Then I thought maybe an OS difference, after all one of my main servers is Windows, the other CentOS. Nope. Same behavior on both.

Hum. Next up some notes.ini settings can change these settings (see
Ben's post here), but none of those are present on the server.

Lastly was an idea that maybe a local desktop policy was interfering but I could find no evidence of that after several tries (and as mentioned yesterday, the desktop policy settings document is not showing AES as an option and this should only be for local anyway, not server).

So my conclusion for now is that you can only encrypt NEW DATABASES with AES 128 on a Domino server
(or it could be AES and reporting wrong, but new NSFs that are encrypted don't exhibit that behavior, so I very much doubt it is that).
Darren Duke   |   April 14 2020 06:28:32 AM   |    domino  security    |   Comments [0]

One of the new features added with 11.0.1 is 128 bit AES local encryption. Kudos for HCL doing stuff IBM could of and should of done a decade ago. But there are a few things missing.

If you encrypt a NSF with AES encryption like so (this is on a Domino server to get "at-rest" encryption):


Image:Creating a replica of an AES encrypted NSF - some issues

If you then create another replica of the NSF, AES is not an option (only strong):


Image:Creating a replica of an AES encrypted NSF - some issues

I tested against two 11.0.1 servers and a 11.0.1 server and 11.0.1 client (everything was 11.0.1 didn't change the options).


So what can you do if you require AES encryption on all replicas? I guess you can replicate without encryption and then encrypt once it's at the new location (update 04/14/2020, no, you're not). Unfortunately it's also not a setting in a policy yet either so if you were hoping to use this with a Notes client prepare to be disappointed for a while:


Image:Creating a replica of an AES encrypted NSF - some issues

PubNames template is 11.0.1 as well.


One step forward and half a step back is still a big improvement over IBM's approach of no steps forward at all.
Darren Duke   |   April 13 2020 11:04:26 AM   |    domino    |   Comments [0]

Updated 3/31/2020 for 11.0.1

A few weeks ago it dawned on me that there doesn't seem to be a useful (at least that I know of) list of the features added to Domino over the last few releases. Then Lisa asked me for this list for a STS customer. While this is not definitive, it is a least a starting point to see what new features were added when, and maybe more importantly, what was added. This is a tad difficult to collate due to IBM's seriously disastrous decision to go the "feature pack" way for a while.


It should also be noted that as of 10 and higher IBM/HCL have reverted back to fix packs being fix packs. So no new features should be getting added outside of a release revision (for example no new features in 10.0 FP1, but there are new features in 10.0.1).


So here goes:


9.0.1 (IBM)


FP8
  1. NIFNSF
  2. Java 8 on server
  3. Summary 16MB
  4. SAML ADFS 3.0

FP9
  1. New mail, contacts and pubnames templates (Some features below require these new templates)
  2. View open speed increase on TX logged NSFs
  3. Mail forwarding restrictions
  4. Inline view indexing
  5. Run mail rules on existing messges
  6. REST API improvements

FP10
  1. Java 8 in client and DDE
  2. Eclipse update 4.6.2 (from 3.4.2)

10.0 (IBM)
  1. Touch screen supoort
  2. Custom colors
  3. New mail template
  4. Dynamic indexing for highly changing views
  5. Symmetrical clusters
  6. Document deletion logging
  7. Dead mail automatic processing
  8. New ODS 53, 256GB NSF size support
  9. Node.js support
  10. DQL (Domino Query Language)

10.0.1 (IBM)
  1. AUT - automatic client update tool
  2. Marvel Client Essentials included
  3. SSL cipher improvements
  4. One time mail signatures

11.0 (HCL)
  1. New licensing model
  2. New mail template
  3. OpenSSL Use
  4. InstallAnywehre use
  5. DirectorySync
  6. Use ID Vault password as HTTP password
  7. AdoptOpenJDK 8 with OpenJ9
  8. DAOS 2-tier storage
  9. Nomad for iPad released

11.0.1 (HCL)
  1. Swiftfile integration is standard. Can be disabled
  2. New Directory Sync feature allow multiple AD users to be registered at once
  3. Subject Alternate Name support in TLS certs.
  4. SNI Domino HTTP support. Server Name Indication (disabled by default)
  5. DAOS tier 2 storage
  6. Docker image in UBI format
  7. Java 1.8.0.242
  8. Save (well export) document to PDF
  9. 128 bit AES local encryption
  10. Updated templates (including mail and pubnames)
  11. Cross Domain support for Traveler to get ID's from ID Vault
Darren Duke   |   February 29 2020 03:46:06 AM   |    domino    |   Comments [2]