Domino SSL ciphers set in the Domino Server document are ONLY applicable to HTTP. Not SMTP, LDAP, et al.... Doh. You can set with note.ini— Darren Duke (@darrenduke) https://twitter.com/darrenduke/status/560186157930381312">January 27, 2015
Now, I'm back in the office it's time to address this. So based on that session it seems as if LDAP, SMTP, DIIOP, POP3 and IMAP (and Remote debug monitor?) protocols do not adhere to the cipher list in the server document (there was no mention of internet sites documents, but I would presume they are affected by this issue too). That means even if you indicate that the server should only use, say AES ciphers like this:
then any protocol that is not HTTP or HTTPS seems not to restrict this to AES. It would seem to allow RC4 and any other cipher allowed by the server (one would hope that with the TLS fix that also disabled low quality ciphers that none, 40 and 56 bit would get disabled by this but I can neither confirm nor deny this).
There is however a server notes.ini you can use that apparently does work on these errant protocols, but note, this also overrides anything in the server document FOR ALL PROTOCOLS so test first:
Use the NOTES.INI setting SSLCipherSpec to specify SSL restrictions for all protocols. Ciphers are specified by a 2-digit code. You can add as many ciphers as you need.
For example, to enable 3DES and RC4128SHA ciphers, enter the following line in the NOTES.INI file:
where 05 = 3DES and 0A = RC4128SHA.
So that's handy right? What about the two digit hex values? Well this is where you hit a bit of a brick wall. All I can find is this list:
(SSL users only) Determines which SSL-compliant cipher to use to encrypt files on the server. Specification numbers correspond to the following ciphers:
01 - SSL_RSA_WITH_NULL_MD5
02 - SSL_RSA_WITH_NULL_SHA
03 - SSL_RSA_EXPORT_WITH_RC4_40_MD5
04 - SSL_RSA_WITH_RC4_128_MD5
05 - SSL_RSA_WITH_RC4_128_SHA
06 - SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5
09 - SSL_RSA_WITH_DES_CBC_SHA
0A - SSL_RSA_WITH_3DES_EDE_CBC_SHA
0B - SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
0C - SSL_DH_anon_WITH_RC4_128_MD5
0D - SSL_DH_anon_WITH_DES_CBC_SHA
To enter multiple ciphers, enter each cipher specification value, including leading zeros. Do not include spaces between values. For example:
SSLCipherSpec = 01020A
from http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLCipherSpec. What you will no doubt notice is the absence of any AES ciphers in that list. My guess is a quick PMR (sic!) will get you all the values, but I can't seem to find them anywhere so I cracked open DDE and took a look:
So it looks to me as if AES 128 is "2F" and AES 256 is "35". I would still suggest you PMR this as you use any information on this blog at your own risk (see what I did there?).
So if you want to limit all protocols on a server to just RC4 128 SHA-1and MD5 and AES 128 and 256, which will give you a pretty good spread of clients then add this:
Again this will override anything in the server or internet sites documents. After sitting through Daniel and David's session there should be a lot more to come in the next few weeks, like TLS 1.2, new ciphers, SSLv2 handshake's (to prevent some issues people are having with STARTTLS between Domino and other servers) and other things I can no longer recall.
I was fully expecting to write a "what a train wreck" review before I went. I was not expecting to say I had a metric shit ton of fun. But I did. And based on other posts I've perused it seems almost everyone else did. There are far more eloquent reviews elsewhere, so this will be bare bones.
First the "ups", in no particular order:
- Much, much improved OGS. Flow, demos, people who care.....And a quintet, who doesn't like quintets?
- It doesn't seem to matter how many people don't turn up every year, BALD still rocks with people new and old
- Verse, and yes, a feature actually made me moist (more in a post on that later)
- No theme park. Hardly ever went anyway. Gave more time to "proper" socializing
- The 3rd (or 4th??) annual day-after-ConnectED-sphere breakfast. If you ever get a chance to have breakfast with any of these, do it : Lisa Duke, Julian Woodward, Ben Poole, Mat Newman, Amanda Bauman and Tony Holder. A truly mind bending experience, and not for the faint of heart
- No law enforcement was called as we left the Dolphin rotunda at 4AM on Thursday AM
- Math(s) during the CGS. Who doesn't like math(s)?
- Size does matter. Smaller *is* better
- Coming soon (or not), move the Domino view indexes out of the NSF....also called NIF something or other
- They are taking Domino security seriously again
- The Penumbra Dinner and Lisa getting her "captain hat" to go with "the Duke family yacht"
- Domino4Wine from the great folks at Prominic....run DDE and Admin on OSX and/Linux. See more here http://vimeo.com/117342115
Now the "downs"
- I'd guess there were more "technical" sessions than last year, but the admin in me was still left yearning. If you're into Xpages, then you are A-OK. Anything else, the outlook was a tad glum. There was lots of Verse, but not really a lot of *actual* Verse. If IBM wanted to leave me gagging for more, it worked. I really shouldn't have had to sit through three or four near identical sessions to figure this out
- Same price, less days
- New hotel price was $70 per night more expensive. 25% more expensive
- If you know the "Carry On..." films/movies, you'll know what I mean when I say the conference had a bit of that kind of feel to it. That maybe an "up" though, so who knows
- Tickets for booze, although IBM did seem to succumb to "social shame" and reverse that decision, no doubt to Mark Roden's dismay....he had at least 20+ tickets. Again, could technically be an "up"
- The required BlueMix slide in pretty much every presentation I sat through. Still a bit hazy on what BlueMix is
- Beyond the Everyday still makes me think of that movie with Emily Blunt and Tom Cruise, neither of whom presented
- The sheer amount of time between sessions....some were 45-60 minutes apart
Now for the synopsis....
To me, it doesn't matter what you call it, but I think it would matter where you hold it. Vegas? Nah, not really me. Florida-sphere for me given the chance.
I do think Verse has some legs if IBM can get it out soon with features that make it killer. There is one feature I really, really like but it won't be there until Verse 1+. I am worried about seeing the words "IBM" and "freemium" in a sentence. They have divested themselves of any business deemed a "commodity", so they don't have the skills and I doubt they'll have the stomach for consumer long term. If it's just to garner press, then yeah....otherwise color me nonplussed. Verse on-prem? My guess is 6-12 months behind what they are aiming for if they are trying to make it "simple", although I am getting that "Quickr feeling" about this, you know, the one where the rug gets pulled out from under you.....yeah, that one.
I was pleasantly surprised that IBM didn't just completely paper over the elephant in the CGS room by indicating that the decision for next year had not been made (that's what I heard all week by the way). Subsequent "rumor" has it that we may not hear until May-ish.
Finally, IBM needs to be very careful with the loyal (?) folks still using their stuff on-prem. They need to get 9.0.2 out or roll out the 9.0.2 features in fix packs or other avenues. If 9.0.2 doesn't ship until Q4-2015 then it will be two-ish years since 9.0.1. That's just bad form and sends the wrong message. IBM must multi-task effectively, for it is they who have decided to go multi-client.
So there you have it. Maybe the best ConnectED-sphere yet. I'm still contemplating how much of that is down to the attendees and how much is down to IBM.
Anyway, every Linux host should have it's own unique host SSH key to ensure security and authenticity of the server you are connecting to. When you create a server from an OVF that doesn't happen automatically. In fact you get the SSH host key that is on the OVA at time of creation (in this case mine).....potentially opening you up to man in the middle attacks (potentially.....although unlikely).
Here's how to do it......Log in into the Proxy server as root (either via VMware console or SSH into the host using Putty) and issue the following commands:
rm -rf /etc/ssh/ssh_host_*
service ssh restart
Here's the expected output from the above commands....
Once you do this, try logging in again via SSH (again I use Putty) and you should see a warning about a potential security breach and that this could be a bad thing (see below), it's not as we meant to create a new key, so click Yes;
Over that past few weeks I've been banging my head against the wall trying to figure out why a Traveler server that has been relocated behind a proxy would not work (it was a standalone server that was working fine before it was moved behind the proxy). Everything seemed fine, except one couldn't get to the Traveler log on page and/or add devices to the servers. Existing users worked flawlessly. Needless to say this was extremely aggravating. I'd install another, new Traveler server and put it behind the same proxy and it would work fine. WTF?
So after much digging and testing (I've create more Traveler servers in the last week than I have in the rest of my career), I think I have finally managed to work out what the issue is. I would say this is definitely a bug with Traveler, but given the issues I've had lately with PMR's I'm in no mood to play that game Russian Roulette so I decided to figured out a fix on my own (probably and utterly unsupported but it works for me - oh, it is supported!!!).
So first, let's outline the problem/bug I found in Traveler.....
If your Traveler server Domino common name, lets say Traveler/Blah (so traveler) is the same as the hostname in the proxy URL, lets say traveler.blah.com (so traveler) then Traveler throws a wobbler. Do not pass go, do not collect $200 if you will. You will be unable to get to the Traveler web page on the device or any browser for that matter.
Now, use any other Domino server name and as long as it's not the hostname part of URL you use on the proxy then you are fine. This is what I was inadvertently doing when I was creating new Traveler servers to test on. Doh.
So a Domino server name of LNT1/Blah and a proxy URL of traveler.blah.com.....fine! No issues, proxy works fine. "Argh!" I hear you say, "Darren, I have a Traveler server called Traveler/Blah and also use the URL traveler.blah.com to get to it....so I can't proxy it then?". Well yes......
You have two options.....
1 - Install Domino with IHS (on Windows of course) and use IHS. I don't think IHS will have this issue. Hum, IHS no longer supported for a front end to Domino, see this IBM technote.
2 - (and this is the part that is probably unsupported) I was able to build a new Domino server (exact same Domino and Traveler versions as the one you want to proxy) using the Domino server name LNT1/BLAH and copied over the LotusTraveler.nsf and traveler folders out of the existing servers data folder to the new one. I then configured the proxy to use traveler.blah.com and low and behold, it works!
I've been running on the new server for a few days and so far neither iOS users nor Android users have reported any issues so, fingers crossed, this may well work. Who knew?
Here's the new flow:
Internet -> Proxy using host name traveler.blah.com -> Domino Traveler server called LNT1/Blah with a host name of lnt1.blah.com
Now, fair warning this may not yet be supported by IBM so if you choose to do this, you do it at your own risk (while under NDA on this, it was stated that is unsupported so YMMV).
According to this post on the Domino wiki, you can use this server notes.ini setting to fully disable SSLv3 but still keep TLS working.:
If you need this, test it before you put it into production. I have not yet done this, but everyone I know that has has had no issues so far. Again YMMV.
Chrome started 32, ended 39
IE....11 and 11
IBM finally realized the 2015 plan was imploding. Except they "realized" this in 2014, so the immense damage of the plan has already been done. Oh, and I doubt they will stop the culling.
Talking of IBM....support. Oh, how I used to use thee as a unique selling point. Now? I routinely have 14+ day periods when the assignee of the PMR doesn't respond to multiple emails. At first I thought this was just me....it isn't.
POODLE! The *secure* internet broke. Now, anyone who watches 24 will know that Chloe O'Brian could get by any encryption scheme with little more than an abacus and calculator, but low-and-behold, now so can the entire world.
Speaking of IBM, they fixed (or enhanced?) Domino!! The finally implemented TLS albeit giving us the 1999 version (TLS1.0). Way to phone it in IBM. Still IBM did show that then can actually achieve impressive feats in a relatively small period of time. I'm sure the people who achieved this are soon to be moved to positions where they can do less good.
We were told Quickr was getting Office 2013 and Domino 9 support. I'm really beginning to doubt this. Vendors seemingly live to tell customer one thing, do another, and never inform them of the (not in the customers best interest) switch. It's shameful at times.
I can still count on the fingers of no hands how many CCM implementations I've seen.
This year there were 96 IBM ICS Champions. Up from 76 this time last year. I'm still batting a 1000 for non-selection, and I'm quite proud of that. The selection process is about as transparent as IBM sales figures. There does seem to be far fewer "WTF" moments when reading the list this year, so kudos to the selection committee for putting forth less politically motivated selections.
I wrote a (now infamous) blog post about Mail.Next. There was a lot of consternation from the "powers-that-be" inside of IBM about being called a "vindictive arsehole" and much less about the point of the post. I did say IBM fixed TLS right? Oh, and I was threatened with revocation of my Champion status (see above).....you can't make this shit up.
Speaking of Mail.Next, it got a name....IBM Verse! Get it? Me neither. Based on the a fore mentioned blog post, it has been said I'm ad-Verse.
In an effort to market Mail.Next, or maybe create something "viral" many IBMers blacked out their eyes on their social media avatars. To me it just looked like they mistakenly used their Adult Friend Finder picture by accident. Still, you can get plenty of viral stuff from Adult Friend Finder so I can see the allure.
Connect, I don't even recall going. Either that's a bad sign for me or the conference. Or both.
Speaking of not-Lotusphere, in order to fix all the shortcomings with Connect IBM did what IBM do in moments of crisis, they renamed it. To "ConnectED". Get it? Yeah, neither did I. The only way to Google this for the longest time was to Google "Connect 2014". Given the extreme lack of information even this close to the event, one can assume that the next rename will the "Dis-ConnectED". In reality I would be shocked if ConnectED 2015 lasted to ConnectED 2016, both in name and in conference location.
Lenovo purchased IBM's Intel Server division. Just after IBM pushed hard into the cloud and bough Softlayer. Luckily for IBM neither of those things require lots and lots of servers.
Speaking of cloud, the race to the bottom continues....cheaper storage wars are ongoing between Microsoft, Amazon and Google. Soon they may well be paying us to used their stuff!
Apple released the iPhone 6 Plus. Most likely as an answer to cries for a bigger iPhone. I would suspect that they are the most returned device that Apple has had, but that just makes Apple right, so they'll quite happily eat the cost.
Oracle continue to be the most vilified IT vendor in the galaxy. Not due to how many servers Oracle requires, but due to *still* incorporating the Ask.com toolbar and home page in the Java desktop installs.
I'm kind of starting to miss This Week In Lotus. That only took a few years! There has been a lot of fodder that we could have munched on lately, so maybe that is all it is.
Technologies that made 2014 included ASUS 28" LED monitors, ACS smart cards and Veeam 8.
Technologies I thought would make 2014....Hawthorn.
Go scan your servers at SSL Labs.
Anyway, provided you are using 9.0.1 FP IF1 (the TLS fix that IBM provided a while back) the apparent Domino fix is to disable AES and 3DES ciphers and run with only RC4:
With those changes you go from an "F" to a "B" on SSL Labs. Here is the server with AES and/or 3DES enabled:
Here is a Domino server with just RC4 enabled:
Oh, and F5's are also at risk.....
If you're on anything less than 9 then you don't get TLS so you're not affected by this.....oh, the irony. Still it would be very beneficial to IBM's public perception to get TLS 1.2 and better ciphers into Domino ASAP. Fixing this stuff once a decade is not cutting it. As you can see above RC4 is not to hot these days.
As Adam Langley puts it:
This seems like a good moment to reiterate that everything less than TLS 1.2 with an AEAD cipher suite is cryptographically broken
So, IBM, the ball is in your court again.......
I will be updating my free proxy soon, but that is not affected by this issue, it still gets a "B".
Now just implementing these fixes may not completely protect you, (thinking BEAST attack here) unless you also disable both AES ciphers in Domino. Basically these are the two ciphers you want enabled:
It's worth pointing out that with the TLS1.0 fix IBM also addressed a long time pet peeve of mine, low quality ciphers:
- SSL renegotiation has been disabled
- All weak (<128 bits) cipher suites have been disabled
A good move but you still should really disable 128bit AES ciphers too. Indeed if you have 40 or 56 bit ciphers enabled then the Domino console throws this out:
If you *do* need to enable these low quality ones then you are doing something really, really wrong.
Disable the AES ones and you may just end up with a "B" grade at https://www.ssllabs.com/ssltest/index.html (the server below was also upgraded to SHA2)
If you want an "A" use my proxy server as Apache does support TLS1.2.
Now, IBM, you have brought us into 1999 with TLS 1.0, when do we get to the heady heights of 2008 and TLS 1.2? All kidding aside IBM have shown here that they can *still* do amazingly good work in a pretty short period of time. Once the SHTF it took less than 90 days for this to be addressed. Hopefully this is the shape of things to come and this sense of urgency for security will remain and not be left on the shelf until the next end-of-the-world-security-failure scenario.
I won't hold my breath for TLS 1.2 support, but I will cross my fingers.
Well, yes there is. Using a old Domino feature called Indirect Files, copy and paste and Excel. Let me show you how......
If you're on Domino 9 make sure to add the following to your server notes.ini, This will prevent compact from failing by preventing the Router from delivering mail to a compacting NSF (oh, how I wish we had this back in 8.5!!!):
If you are not on 9, you have to do it the hard way.....either the server needs to be down, you can quit the router or you can try and try again until you get the files compacted. Pick your poison, but I'd suggest the last option until you only have a handful of errant NSF's to do.
Either way, this is how I'd tackle this issue.
1) Go to the Files tab in Domino Administrator, mail folder and sort by the DAOS State tab (in the example below I only have two NSF's, in reality I'd do these manually, but if I had a few more I'd use the process I'm outlining here):
2) You are interested in any mail file with a status of anything other than Read/Write. So select all those mail files matching this criteria, go to the Edit menu and select Copy :
3) Open Excel (or your preferred spreadsheet app.....Symphony anyone? Bueller? ) and paste in the contents, you should end up with this:
4) We're only really interested in the Physical Path column, you can delete everything else. In a new column enter this formula (in my example I've deleted all columns except Physical Path and moved it to the A column):
Copy that cell (in my case B1) to the rest of the rows and you should end up with something like this (also remove the title row):
5) Now select the column with the mail file names as relative to Domino (not the file names, so in my case column B). Once selected "copy" to the clipboard:
6) Now it's time to pay attention.... create a new blank workbook and paste the column using the Paste Special option:
7) Select Paste Values:
8) Voila, you now have a new workbook with just relative file names of mail files that are not DAOS'd. Save this new file as MS-DOS (*.txt) but use the suffix .IND for the file:
9) Get this file on your Domino server. In my case it's on the D: drive called FiletoDAOS.IND. Once there run the following command from the Domino console:
load compact -c -i -ZU -v -n -daos on D:\FileToDAOS.IND
What this does is use an Indirect file (our .IND file) as the source of files to compact....if successful you should see the results on the Domino console:
10) If your using Domino 9 and have that notes.ini setting chances are you will get the majority compacted....if your using anything less than R9 you may have to repeat these steps several times (each time getting less and less files to be DAOS'd).
There you go. You will now slowly whittle away at the non-DAOS'd mail files on your server. Sure beats typing on the console....especially if you have more than a few mail files to do.
11) To see if you still have any left to do, look at the mail folder in Domino Admin again (remember to F9). In my case all were completed.....:
If yours are not, go back to step 1 and whittle away again......