April 16 2023 Sunday
Ransomware Prevention Part 11 - Let’s talk about Service Accounts
Part 11 - Let's talk about Service Accounts
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Service Accounts. They exist everywhere. Most have common (and scary) attributes such as passwords that haven't changed in a few years, if not a decade or more. Service Account passwords are rarely, if ever, changed because of of the havoc the can be created when they are. And there are many organizations that have no earthly idea where or how many times these accounts are used. Finally many are local admins on server, or (queue scary music) are domain admin level accounts.
Given any of the above attributes, let alone several of those attributes being present in a single service account, it should come as no surprised that they represent an adversary hitting a gold mine if the can compromise one of them. So how does one protect these God level accounts? Read on.....
In all my years of doing this, I have yet to find an actual reason to have any service account listed as a domain admin. Zero. Nada. Ziltch. If you have a service account in Domain Admins you are simply doing your job wrong.
Managed Service Accounts
The preferred way to create, manage and use service accounts is utilize Managed Service Accounts (MSA). These MSA accounts come in two distinct flavors, stand-alone MSA accounts (sMSA aka MSA), and group MSA accounts (gMSA) and were first introduced in Windows 2008 R2 with gMSA accounts being added with 2012. The only significant difference between the two types is that a sMSA account can only ever be used on a named, single server. it cannot be "assigned for use" on two servers at the same time (note, I said servers, not services!). gMSA accounts on the other hand can get used across several names servers, so a shared account if you will.
For security reasons, sMSA accounts should always be your default choice. gMSA accounts have specific use cases, the one I see the most is using a single gMSA account on several ADFS servers in an ADFS farm.
MSA accounts in general address many, if not all, of the issues with traditional service accounts, namely:
"Darren, this sounds perfect!" Well, yes and no.
That being said, MSA accounts can and should be used anywhere and everywhere you can. For management of MSA accounts, ManageEngine have a helpful free tool available here: https://www.manageengine.com/products/free-windows-active-directory-tools/free-active-directory-service-account-management-reporting-tool.html. Microsoft has detailed MSA documentation and a quick Google will show you how to set them up.
Traditional Service Accounts
As mentioned above, you may locate services that simply won't work with MSA accounts. If you have those then you're left with the traditional service account way, which is simply a user account. These accounts, while nowhere near as secure as MSA accounts are, can have increased security.
Traditional service accounts should be prevented from interactive logins. While MSA accounts have this prevention enabled by default, traditional service accounts need to be set up for this via a GPO
Traditional service accounts need long, complex passwords. I'd look at a minimum of 32 characters.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Service Accounts. They exist everywhere. Most have common (and scary) attributes such as passwords that haven't changed in a few years, if not a decade or more. Service Account passwords are rarely, if ever, changed because of of the havoc the can be created when they are. And there are many organizations that have no earthly idea where or how many times these accounts are used. Finally many are local admins on server, or (queue scary music) are domain admin level accounts.
Service Account passwords are rarely, if ever, changed
Given any of the above attributes, let alone several of those attributes being present in a single service account, it should come as no surprised that they represent an adversary hitting a gold mine if the can compromise one of them. So how does one protect these God level accounts? Read on.....
In all my years of doing this, I have yet to find an actual reason to have any service account listed as a domain admin. Zero. Nada. Ziltch. If you have a service account in Domain Admins you are simply doing your job wrong.
Service Accounts should never, ever be a Domain Admin
Managed Service Accounts
The preferred way to create, manage and use service accounts is utilize Managed Service Accounts (MSA). These MSA accounts come in two distinct flavors, stand-alone MSA accounts (sMSA aka MSA), and group MSA accounts (gMSA) and were first introduced in Windows 2008 R2 with gMSA accounts being added with 2012. The only significant difference between the two types is that a sMSA account can only ever be used on a named, single server. it cannot be "assigned for use" on two servers at the same time (note, I said servers, not services!). gMSA accounts on the other hand can get used across several names servers, so a shared account if you will.
For security reasons, sMSA accounts should always be your default choice. gMSA accounts have specific use cases, the one I see the most is using a single gMSA account on several ADFS servers in an ADFS farm.
MSA accounts in general address many, if not all, of the issues with traditional service accounts, namely:
- Automatic changing of passwords by AD ever 30 days or whatever your AD machine password expiration is. NO manual intervention is needed.
- Password complexity is high, 240 bytes make brute forcing difficult
- MSA accounts have to be specifically assigned to a server before a server can use it.
- MSA accounts are prevented from interactive user logins.
"Darren, this sounds perfect!" Well, yes and no.
- Not every service you have running can use MSA accounts. It's gotten better over the years, but it's still trial and error.
- They are an absolute pain in the backside to create and manage the first time you try.
- You still need to reduce the MSA account to least privileged access.
That being said, MSA accounts can and should be used anywhere and everywhere you can. For management of MSA accounts, ManageEngine have a helpful free tool available here: https://www.manageengine.com/products/free-windows-active-directory-tools/free-active-directory-service-account-management-reporting-tool.html. Microsoft has detailed MSA documentation and a quick Google will show you how to set them up.
Traditional Service Accounts
As mentioned above, you may locate services that simply won't work with MSA accounts. If you have those then you're left with the traditional service account way, which is simply a user account. These accounts, while nowhere near as secure as MSA accounts are, can have increased security.
Traditional service accounts should be prevented from interactive logins. While MSA accounts have this prevention enabled by default, traditional service accounts need to be set up for this via a GPO
Traditional service accounts need long, complex passwords. I'd look at a minimum of 32 characters.
December 9 2022 Friday
Domino 12.0.2 - no support for Windows 2016? Really?
While talking with a customer today I was informed HCL told them Windows Server 2016 wasn't supported for Domino 12.0,2 (apparently due to some technical limitation). I thought there was no way this was correct, so off I go to HCL's support web site, and low and behold, no Windows 2016 listed as a supported OS!
From https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101447
From https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101447
October 21 2022 Friday
My Collabsphere 2022 presentation - Great new Domino features since 9.0.1FP8
It's available over on slideshare. Link is below.
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8pptx
https://www.slideshare.net/darrenduke/great-new-domino-features-since-901fp8pptx
June 30 2022 Thursday
Domino 12.0.2 adds VSS backup support
Since BackupExec ceased support for Domino backup APIs after v14 there have been very few backup utilities that integrate with Domino natively. The fix was always for IBM to add VSS support to Windows Domino installs (the vast, vast majority of installs I see *are* on Windows). But IBM (along with 1000's of other fixes they should have and could have done) choose not to.
HCL have finally fixed this oversight (and by oversight I mean complete dereliction of duty from IBM). I fully admit I was worried when HCL went all chips in and bought it all from IBM, but boy have they been adding stuff that has been sorely missing from the product. VSS support included.
The best part is that (for backup at least, restores are a tad more finicky so be sure to read the docs) there is no setup on your side once 12.0.2 ships and you install it. It is available today in FlexNet as a preview release, not gold code yet so you've been warned. Here is what happens on the Domino side (I have logging turned up) when Veeam backs up my 12.0.2 Windows server with Veeam "application aware processing" turned on:
HCL have finally fixed this oversight (and by oversight I mean complete dereliction of duty from IBM). I fully admit I was worried when HCL went all chips in and bought it all from IBM, but boy have they been adding stuff that has been sorely missing from the product. VSS support included.
The best part is that (for backup at least, restores are a tad more finicky so be sure to read the docs) there is no setup on your side once 12.0.2 ships and you install it. It is available today in FlexNet as a preview release, not gold code yet so you've been warned. Here is what happens on the Domino side (I have logging turned up) when Veeam backs up my 12.0.2 Windows server with Veeam "application aware processing" turned on:
June 9 2022 Thursday
Ransomware Prevention Part 10 - Credential Guard, the feature you didn’t know existed
Part 10 - Credential Guard, the feature you didn't know existed
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
This series is now over a year in the making.....I hope a reader or two still exists.
Certain versions of Windows have a special feature called Credential Guard. Due to Microsoft not being, well, particularly into security this feature is not present in Home nor Pro versionS of Windows. I view this a travesty, but hey Microsoft makes tons of money so why should they care. It does exist in Enterprise and Education desktop Windows and also in Windows Server since 2016. If you have looked at doing Windows 10 Enterprise before, but haven't found a killer feature, then this is it (and LTSC). If you have never looked at buying Windows 10 Enterprise, this is the feature that should make you look into it.
Not only is is not widely available, it's woefully marketed. Did you even know about this? One of the two most important tools in the hacker prevention tool chest? (the other being SRP, aka part 6 in this series)
Credential Guard does what it says on the box, it protects credentials. Specifically in-memory credentials. These are stored in such a way as to be accessible to hackers once they compromise the device ("pass the hash" is the usual name for this type of hack). If you have no idea what I'm talking about go watch this video that uses the MimiKatz tool to extract in-memory credentials (password hashes to be exact) out of thin air....
https://youtu.be/bTYR_xYSDIk
Scared yet? If you're not then you're in the wrong job. Go read a gardening blog or take up knitting.
What the above video shows is how easy it is to effectively harvest credentials from Windows OSes. Credential Guard addresses this Windows "feature". It also worth noting that some CPUs now also have this type of protection built in, specifically AMD Ryzen Pro CPUs can have a similar protection enabled in BIOS. But on the Pro line.
OK, so I have Windows 10 Enterprise, or 2019/2022 server. How do I get this level of protection? For starters, VMs are a bit different, so I'll cover those later. Second, laptops with VPN clients are different so read this all before you enable it on laptops. Even a standard desktop OS it's a lot of convoluted steps. Thankfully Microsoft do provide a PowerShell script to simplify enabling it. They way it works it also a bit convoluted. The setup even more so. The PowerShell (see two paragraphs down) is a God send.
See, the "fix" Microsoft came up with is to install a Hyper-V machine on the device in question, lock it down and encrypt it and store the credentials in that Hyper-V instance. So now you have two PCs. Kind of. If you really want to know more about how it works see here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-how-it-works
The PowerShell readiness/enablement script is here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/dg-readiness-tool
The above script needs to be ran in an admin PowerShell with the -capable switch, then you need to reboot and run it again with -capable. Be sure to check out the pre-reqs as well (UEFI, enable virtualization technologies in BIOS, etc.)
If the script says you can enable it, run it with -enable. A reboot and few auto reboots later, Credential Guard is installed. Note, if you use a VPN client on the device in question, chances are the VPN is not fully CG compatible, so be sure to check, If the VPN client is not CG compatible run with -enable -cg (so not just -enable, add the -cg).
Here I'm running the script with the -capable switch to see if my PC can indeed enable CG.....
I need to reboot, then I run the -capable again:
In the above screen shot I have highlighted an issue. In this case it is very likely a VPN shim driver as I'm running it on a laptop with a VPN client so I will run the "-enable -cg" flags to enable *only* CG (just "-enable", so without "-cg" does get me better security, but experience tells me it will stop my VPN client for working.)
Above, we can now see Hyper-V and IOMMU have been enabled. Time to reboot again....and then rerun the PowerShell with the -ready switch:
As you can see I now have Credential Guard enabled. The yellow warning are because I chose to *only* enable CG and not the other option as that would croak my VPN client. MimiKatz has now been taken to the vet and euthanized and the password hashes are no longer accessible to hackers. You can see this in action on this video:
https://youtu.be/urqXgBbVyWY
Once enabled my LSA credentials are not longer stored in-memory in plain text. This also adds another Windows process, LSALSO which is the new credential handler:
If this were a LAN PC, and hence no need for the -cg switch (I'm presuming a LAN connected PC doesn't need to VPN into the LAN.....) the -ready check should show this after I ran a straight -enable switch. Below is Windows 2022 Server after the -enable with all features green:
OK, servers. Physical servers are enabled the same as desktops. VMware Windows guests are different. These need to enabled in the VM options under Virtualization Based Security (VBS) and then the PowerShell ran as desktop. This feature is available in vCenter 6.7 and higher. At the time of writing I'm still not getting Windows Server 2016 to work even though it should, 2019 and 2022 are both fine. YMMV. As always take a snapshot for the VM before jacking with it. Checking the VBS box will enable IOMMU and UEFI (you should already be using UEFI anyway). Here's the check box in question for VMs (note, you only see this if you specifically set the guest OS version, i.e. Windows Server 2022 in the General Options section, if you leave VMWare Tools to figure it out this check box does not appear):
It is not lost on me the irony of a Windows VM running a Windows VM in order to secure it's credentials. Nested VMs like this used to be a big serious no-no but with the advent of CG/VBS most of the real-world arguments are around performance. I haven't seen an perceptible performance degradation on any VM, but again YMMV.
I'm a really big believer in doing Credential Guard whenever and wherever possible. If it's a 2019 or greater server and I've built it chances are it's CG protected. All of the STS desktops and laptops are CG enabled as well, although you do need Windows 10 Enterprise or Education to enable it. If you want to talk about getting on the Windows 10 Enterprise bus, drop Lisa a line and we can talk about it. It's a worth having if for no other reason than CG.
This is just the basics of Credential Guard so be sure to check out the additional mitigations you can also take to further secure your environment here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/additional-mitigations
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
This series is now over a year in the making.....I hope a reader or two still exists.
Certain versions of Windows have a special feature called Credential Guard. Due to Microsoft not being, well, particularly into security this feature is not present in Home nor Pro versionS of Windows. I view this a travesty, but hey Microsoft makes tons of money so why should they care. It does exist in Enterprise and Education desktop Windows and also in Windows Server since 2016. If you have looked at doing Windows 10 Enterprise before, but haven't found a killer feature, then this is it (and LTSC). If you have never looked at buying Windows 10 Enterprise, this is the feature that should make you look into it.
Not only is is not widely available, it's woefully marketed. Did you even know about this? One of the two most important tools in the hacker prevention tool chest? (the other being SRP, aka part 6 in this series)
Credential Guard does what it says on the box, it protects credentials. Specifically in-memory credentials. These are stored in such a way as to be accessible to hackers once they compromise the device ("pass the hash" is the usual name for this type of hack). If you have no idea what I'm talking about go watch this video that uses the MimiKatz tool to extract in-memory credentials (password hashes to be exact) out of thin air....
https://youtu.be/bTYR_xYSDIk
Scared yet? If you're not then you're in the wrong job. Go read a gardening blog or take up knitting.
What the above video shows is how easy it is to effectively harvest credentials from Windows OSes. Credential Guard addresses this Windows "feature". It also worth noting that some CPUs now also have this type of protection built in, specifically AMD Ryzen Pro CPUs can have a similar protection enabled in BIOS. But on the Pro line.
If a hacker can harvest a domain admin account you are toast. They have already won. Just take down your tent and go home. Find a good gardening blog or take up knitting. Your job is to prevent that from happening......
OK, so I have Windows 10 Enterprise, or 2019/2022 server. How do I get this level of protection? For starters, VMs are a bit different, so I'll cover those later. Second, laptops with VPN clients are different so read this all before you enable it on laptops. Even a standard desktop OS it's a lot of convoluted steps. Thankfully Microsoft do provide a PowerShell script to simplify enabling it. They way it works it also a bit convoluted. The setup even more so. The PowerShell (see two paragraphs down) is a God send.
See, the "fix" Microsoft came up with is to install a Hyper-V machine on the device in question, lock it down and encrypt it and store the credentials in that Hyper-V instance. So now you have two PCs. Kind of. If you really want to know more about how it works see here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-how-it-works
The PowerShell readiness/enablement script is here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/dg-readiness-tool
The above script needs to be ran in an admin PowerShell with the -capable switch, then you need to reboot and run it again with -capable. Be sure to check out the pre-reqs as well (UEFI, enable virtualization technologies in BIOS, etc.)
If the script says you can enable it, run it with -enable. A reboot and few auto reboots later, Credential Guard is installed. Note, if you use a VPN client on the device in question, chances are the VPN is not fully CG compatible, so be sure to check, If the VPN client is not CG compatible run with -enable -cg (so not just -enable, add the -cg).
Here I'm running the script with the -capable switch to see if my PC can indeed enable CG.....
I need to reboot, then I run the -capable again:
In the above screen shot I have highlighted an issue. In this case it is very likely a VPN shim driver as I'm running it on a laptop with a VPN client so I will run the "-enable -cg" flags to enable *only* CG (just "-enable", so without "-cg" does get me better security, but experience tells me it will stop my VPN client for working.)
Above, we can now see Hyper-V and IOMMU have been enabled. Time to reboot again....and then rerun the PowerShell with the -ready switch:
As you can see I now have Credential Guard enabled. The yellow warning are because I chose to *only* enable CG and not the other option as that would croak my VPN client. MimiKatz has now been taken to the vet and euthanized and the password hashes are no longer accessible to hackers. You can see this in action on this video:
https://youtu.be/urqXgBbVyWY
Once enabled my LSA credentials are not longer stored in-memory in plain text. This also adds another Windows process, LSALSO which is the new credential handler:
If this were a LAN PC, and hence no need for the -cg switch (I'm presuming a LAN connected PC doesn't need to VPN into the LAN.....) the -ready check should show this after I ran a straight -enable switch. Below is Windows 2022 Server after the -enable with all features green:
OK, servers. Physical servers are enabled the same as desktops. VMware Windows guests are different. These need to enabled in the VM options under Virtualization Based Security (VBS) and then the PowerShell ran as desktop. This feature is available in vCenter 6.7 and higher. At the time of writing I'm still not getting Windows Server 2016 to work even though it should, 2019 and 2022 are both fine. YMMV. As always take a snapshot for the VM before jacking with it. Checking the VBS box will enable IOMMU and UEFI (you should already be using UEFI anyway). Here's the check box in question for VMs (note, you only see this if you specifically set the guest OS version, i.e. Windows Server 2022 in the General Options section, if you leave VMWare Tools to figure it out this check box does not appear):
It is not lost on me the irony of a Windows VM running a Windows VM in order to secure it's credentials. Nested VMs like this used to be a big serious no-no but with the advent of CG/VBS most of the real-world arguments are around performance. I haven't seen an perceptible performance degradation on any VM, but again YMMV.
I'm a really big believer in doing Credential Guard whenever and wherever possible. If it's a 2019 or greater server and I've built it chances are it's CG protected. All of the STS desktops and laptops are CG enabled as well, although you do need Windows 10 Enterprise or Education to enable it. If you want to talk about getting on the Windows 10 Enterprise bus, drop Lisa a line and we can talk about it. It's a worth having if for no other reason than CG.
This is just the basics of Credential Guard so be sure to check out the additional mitigations you can also take to further secure your environment here: https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/additional-mitigations
March 24 2022 Thursday
Ransomware Prevention Part 9 - More semi-easy stuff
Part 9 - More semi-easy stuff
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Well, it's been a while since part 8, but this series generated a lot of work for STS, so I'm not complaining. After doing some of this stuff for a while now I have some more suggestions for you to implement.....
HTA - the malware distribution engine Microsoft provides free of charge
I always forget organizations still have and use Internet Explorer. Thanks Microsoft! Even though it goes end of life in June 2022, it's still used in a lot of places. A lot. Not only did MS foist IE on us, but they also bundled on top a thing called HTML Application (HTA for short, see https://en.wikipedia.org/wiki/HTML_Application). And this thing keeps giving, and giving, and giving. In short it allows a bad actor to trick your users into possibly opening a bad application. See HTAs don't use the security model of the browser. No, HTA's are fully trusted applications that use the IE engine and allows VBScript and JScript to run with access to the file system and the registry. Without supervision. FYI, file system access is how ransomware encrypts shared drives, You've just connected the dots right? From the Wikipedia article above:
Then this gem....
Holy $%*# Batman.
In all my years doing this I don't recall seeing a single practical use for HTAs. The only time I've seen it used is to distribute malware via a script based attack. To block this super villain sized nastiness do as many of these things as you can:
1. Block all .HTA files from running. Period. Use a software restriction policy, add it to your endpoint protection. Add it everywhere.
2. Block mshta.exe from running. Again, use a software restriction policy, add it to your endpoint protection. Add it everywhere.
3. Block the MIME type application/hta.
4. Block files containing MTA:Application header.
5. Delete the .hta file association in Windows,
Do all of the above as HTAs are slippery little buggers. Oh, and just deleting mshta.exe is probably not sufficient as it can be dropped again using a different name. You really need as many of the above as you can figure out how to implement.
If you thought HTAs were bad.....two words: Velvet. Sweatshop. Read on.....
Somewhat harder....Password protected Office files
Look, I know it's hard. People like to think they are in the know by password protecting Excel files. It feeds their superiority complex. However, what you may not know is that if said password is "VelvetSweatshop" (no quotes) it is exceedingly special in Excel. And not in a good way.
Password protected files in Office are actually also encrypted files in Office (the encryption level is based on the Office version, https://en.wikipedia.org/wiki/Microsoft_Office_password_protection). So by being encrypted, these files essentially bypass any and all scanning by your security systems as the scanners cannot see inside the files to analyze them. Still with me? Because this is about to get very, very interesting....
When a user opens a password protected Excel file, Excel (all by itself) tries to use the password "VelvetSweatshop" to decrypt it. And if that is the password on the file, it does and it happily opens the file. Without any user intervention an Excel file protected and encrypted with the password "VelvetSweatshop" opens. Again, in big text.....
And again.....
Yeah, yeah...the daily thought of "what the eff are Microsoft thinking?" shouts inside your head yet again. Yes Excel has a default password. Kinda like crappy Wifi routers, right? Go try it, it's equal parts insane and, well, insane. It still works as of Excel 2201.
So if you can't scan a password protected Office file and Excel will happily open the file and (drum roll please.....), and lets say for kicks and giggles you also have Office macro support enabled (and you do, everyone does)..... then an unsuspecting user can open an Excel file which has neutered all of your high priced security systems, which then runs code (the Office macro), the bad actor could now have a foot hold in your environment. All because of a default password that exists in Excel.
Oh, and if you think the yellow bar at the top of Excel asking if the user wants to trust the macro, and that said user won't click "absolutely", I have some DarrenCoin to sell you.
Which brings us to....
Somewhat, somewhat harder.....Disable Office Macros from running
I their defence, Microsoft is adding extra security in a few months to Office to prevent internet downloaded Office macros from running. That's a step, but after reading about HTAs and Velvet Sweaters are you going to trust Microsoft?
You simply (ha, not that simple at all actually) need to disable Macro support in each and every Office product that has it. If your business processes are so complicated that they require Office Macro support, simplify your process by firing the idiot who designed it and then disable macro support.
Conclusion
Doing the above will seriously raise your security. You'd be better than most, if not all, of your peers. The older the version of Office you are on, the more vulnerable to these attacks you will be. Just saying.....
And if you want to kick your superiority complex into high gear, go ask your security people about HTA and Velvet Sweaters and when they look at you funny, send them this post. They won't sleep for days.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said all the way back in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Well, it's been a while since part 8, but this series generated a lot of work for STS, so I'm not complaining. After doing some of this stuff for a while now I have some more suggestions for you to implement.....
HTA - the malware distribution engine Microsoft provides free of charge
I always forget organizations still have and use Internet Explorer. Thanks Microsoft! Even though it goes end of life in June 2022, it's still used in a lot of places. A lot. Not only did MS foist IE on us, but they also bundled on top a thing called HTML Application (HTA for short, see https://en.wikipedia.org/wiki/HTML_Application). And this thing keeps giving, and giving, and giving. In short it allows a bad actor to trick your users into possibly opening a bad application. See HTAs don't use the security model of the browser. No, HTA's are fully trusted applications that use the IE engine and allows VBScript and JScript to run with access to the file system and the registry. Without supervision. FYI, file system access is how ransomware encrypts shared drives, You've just connected the dots right? From the Wikipedia article above:
When a regular HTML file is executed, the execution is confined to the security model of the web browser. This means it is confined to communicating with the server, manipulating the page's object model (usually to validate forms and/or create interesting visual effects) and reading or writing cookies.
Then this gem....
On the other hand, an HTA runs as a fully trusted application and therefore has more privileges than a normal HTML file; for example, an HTA can create, edit and remove files and registry entries. Although HTAs run in this 'trusted' environment, querying Active Directory can be subject to Internet Explorer Zone logic and associated error messages.
Holy $%*# Batman.
In all my years doing this I don't recall seeing a single practical use for HTAs. The only time I've seen it used is to distribute malware via a script based attack. To block this super villain sized nastiness do as many of these things as you can:
1. Block all .HTA files from running. Period. Use a software restriction policy, add it to your endpoint protection. Add it everywhere.
2. Block mshta.exe from running. Again, use a software restriction policy, add it to your endpoint protection. Add it everywhere.
3. Block the MIME type application/hta.
4. Block files containing MTA:Application header.
5. Delete the .hta file association in Windows,
Do all of the above as HTAs are slippery little buggers. Oh, and just deleting mshta.exe is probably not sufficient as it can be dropped again using a different name. You really need as many of the above as you can figure out how to implement.
If you thought HTAs were bad.....two words: Velvet. Sweatshop. Read on.....
Somewhat harder....Password protected Office files
Look, I know it's hard. People like to think they are in the know by password protecting Excel files. It feeds their superiority complex. However, what you may not know is that if said password is "VelvetSweatshop" (no quotes) it is exceedingly special in Excel. And not in a good way.
Password protected files in Office are actually also encrypted files in Office (the encryption level is based on the Office version, https://en.wikipedia.org/wiki/Microsoft_Office_password_protection). So by being encrypted, these files essentially bypass any and all scanning by your security systems as the scanners cannot see inside the files to analyze them. Still with me? Because this is about to get very, very interesting....
When a user opens a password protected Excel file, Excel (all by itself) tries to use the password "VelvetSweatshop" to decrypt it. And if that is the password on the file, it does and it happily opens the file. Without any user intervention an Excel file protected and encrypted with the password "VelvetSweatshop" opens. Again, in big text.....
Without any user intervention an Excel file protected and encrypted with the password "VelvetSweatshop" opens.
And again.....
Without any user intervention an Excel file protected and encrypted with the password "VelvetSweatshop" opens.
Yeah, yeah...the daily thought of "what the eff are Microsoft thinking?" shouts inside your head yet again. Yes Excel has a default password. Kinda like crappy Wifi routers, right? Go try it, it's equal parts insane and, well, insane. It still works as of Excel 2201.
So if you can't scan a password protected Office file and Excel will happily open the file and (drum roll please.....), and lets say for kicks and giggles you also have Office macro support enabled (and you do, everyone does)..... then an unsuspecting user can open an Excel file which has neutered all of your high priced security systems, which then runs code (the Office macro), the bad actor could now have a foot hold in your environment. All because of a default password that exists in Excel.
Oh, and if you think the yellow bar at the top of Excel asking if the user wants to trust the macro, and that said user won't click "absolutely", I have some DarrenCoin to sell you.
Which brings us to....
Somewhat, somewhat harder.....Disable Office Macros from running
I their defence, Microsoft is adding extra security in a few months to Office to prevent internet downloaded Office macros from running. That's a step, but after reading about HTAs and Velvet Sweaters are you going to trust Microsoft?
You simply (ha, not that simple at all actually) need to disable Macro support in each and every Office product that has it. If your business processes are so complicated that they require Office Macro support, simplify your process by firing the idiot who designed it and then disable macro support.
Conclusion
Doing the above will seriously raise your security. You'd be better than most, if not all, of your peers. The older the version of Office you are on, the more vulnerable to these attacks you will be. Just saying.....
And if you want to kick your superiority complex into high gear, go ask your security people about HTA and Velvet Sweaters and when they look at you funny, send them this post. They won't sleep for days.
July 15 2021 Thursday
Ransomware Prevention Part 8 - Backup and Recovery
Part 8 - Backup and recovery
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
This post is a bit different from the other posts, in that the previous 7 parts were tools and techniques to help prevent the attacks from ever happening (aka the best case scenario). Even if you follow all 7 posts down to the letter, there is still a possibility ransomware will get through your (now) multi-layered defenses. After all you have to be correct every time for everything. Mr and Mrs Hacker only have get it correct once. So plan for the worst and hope for the best. Not the other way round. So this post will cover how to actually put your organization in a place to recover as best as possible were the unthinkable to happen.
While you could pay the ransom, the Sophos - State of Ransomware 2021 report indicates only 8% of paying victims claimed to recover everything. 4% got nothing at all for their payment. On average only 65% of data is restored after an ransomware incident after paying the ransom, so one third of the data is gone, like the snap in Avengers: Infinity Wars, but for data. The average ransom payment was $170,404 USD. But the entire bill for rectifying the attack comes in at a whopping $1,850.000 USD.
What I'm about to cover cannot be done with a $100 Microcenter USB external drive and Windows Backup (well, maybe it can, but it shouldn't). Yes, for real backup and recovery build outs can be relatively expensive, but they are far, far less expensive than the average $1,850,000 that it currently costs were you to pay up and all the other things you now have to fix. And once you get hit, YOU WILL BE DOING THIS ANYWAY, so make the argument to do it now. It's not if you will get hit, it's when. And just because you have been hit doesn't mean you won't get hit again. I really wish they'd spend more time on probability in math(s) class.
Alas, sometimes you need a really bad experience to understand the obviously (now with the benefit of hindsight) stupid things you previously did. Exhibit number 1:
So let me start the meat of this post with the most important thing you will ever read in terms of recovering from a ransomware attack.....
No, I'm serious....this includes password and decryption keys as well. So once again, to the chorus.....
Never. Ever. The stories I have heard....."we had backups but they got encrypted as well"...."we had off-site backups and we even encrypted them for reason x,y,z, however the the private key/password (usually just a text file stored in a "secure" IT Windows file share) was encrypted by the ransomware so our backups are useless". It goes on and on and on. It's extremely common for an organization who gets ransomwared who also has backups that are about as useful as an ashtray on a motorbike. Far more common than you would ever imagine. So plan. And have a plan for when the plan won't work. Print actual copies of any keys you use and put them in a very safe place. Make sure you are not the only one who knows them.
Don't be the guy above that puts temporary hose ramps on a train track. Let's try to save you from that, eh?
For the most part this article will cover Veeam, mainly because of all the systems I've used, it's the easiest and does what it says. You solution du-jour may or may not be able to do the following. If it can't consider changing.
Also this is backup and recovery. Not high availability. Those are two very different things that are != (or <>) to each other at all. While a given product maybe able to do both, I'm not covering both here. HA is a paying gig and track down Lisa if you're interested in that.
Now for the second most important thing to understand about backups.....automate. When humans are involved with backups they fail. All the time. When humans are not involved with backups they fail far less often.
Recognize that not everything needs to be backed up and recoverable
There is some stuff is critical to your organization. Without it you simply cannot function. Back those up. Everything else is optional and is a function of cost vs PIA to rebuild it. For example, SQL servers and AD, sure. But if I had a pretty sizable Tenable install with one or more Nessus Linux scanners feeding it, do I really need to backup *all* the Nessus scanner devices? I would argue no. The value is in the Tenable reports that are harvested from the Nessus scanners. I can rebuild the Nessus scanners at a later date, or just back up one or two of them. Needless to say, the more you back up the more time it takes. Additionally you are taking precious backup resources from other more critical systems.
Frequency and Tagging
Give serious thought to the frequency you need to backup a given device. Break out your backups into these frequencies. Some stuff you want daily, others weekly or even monthly or quarterly. I may backup a given domain controller daily, but others maybe able to be backed up weekly. Also tag the stuff you don't want backed up. Then there is no confusion as to who is to blame when all hell breaks lose and that VM is not in the backup.
Tagging VMs is a way to combat the age old issue of forgetting to add something to the backup. Tagged objects can then be added automatically to backups. Both VMware and HyperV can do this (requiring vCenter and SCVMM respectively). In vCenter create folders for each backup frequency and add a tag to that folder and move VMs to the required folder. Then have Veeam back up that tag. SCVMM is much less user-friendly as you have to tag each VM independently.
Here's a vCenter folder tagged (meaning everything in that folder is also dynamically tagged when Veeam comes looking):
And here is the corresponding Veeam job that adds VMs that match the tag at every execution. Truly dynamic and now you don't need to edit your backup job everytime someone adds a VM. Simply move the VM to the required folder in vCenter and the next time that job runs, the new VM is added to the backup.
SCVMM is a per VM setting, but Veeam is still the same, dynamically adding VMs with associated tag at backup execution time. You cannot set this in HyperV settings, only in SCVMM settings:
Don't forget to backup assets that you will need *during* the recovery. Your PC for example. Also backup and store off-disk the Veeam configuration. You really don't want to have to install a new Veeam server and have it index all the backups across all your different storage tiers. That can add a long time the recovery.
Yes, you do need three tiers of backups
Every knows this already, yet few do it. It's a bit like exercise, we all *know* we should do it and it's not a secret, but doing *it* is a whole different matter. Multi-tier backups are like that. We *know* to do. The majority just don't. And by multi-tier I don't just mean cloud. Cloud for restoring has significant issue which I'll get to later. Just don't go thinking you've avoided all the backup pitfalls by using cloud. Because you haven't.
So a Darren approve system would go something like this....
Backup Location 1: Local disk. Dedicated *only* to the backup system. Not on a shared SAN with everything else. That's simply moronic and your asking for trouble with that approach. Lots and lots of storage. For Veeam your going to want format the storage as ReFS. Local disk has lots of advantages:
It does have one pretty big disadvantage:
Build for restore speed
Look, once your hit and you are confident you have good, restorable backups, it's now a time sink, a waiting game if you will. Create restore job, wait, wait, wait. Create restore job wait, wait, wait. The shorter your restore time, the faster you'll be back up and running. So from a restore perspective build the fastest backbone you can. At a minimum I'm talking 10Gb. See 10Gb is literally 10x faster than 1Gb. In real life 10Gb is 5x to 7x faster than 1Gb. That is still a huge factor. See:
10TB restore at 1Gb = ~22 hours
10TB restore at 10Gb = ~4-5 hours
And trust me, when you get hit, 10TB is a tiny amount to restore. If you have 4 VMs hitting 10TB each, on a 1Gb network you'll be up in approx one work week. On a 10Gb network, that is now restored inside of a day.
So this brings me back to the woeful cloud speeds during a restore. Even if your cloud provider were to give you a 10Gb feed back (which I very, very much doubt), can your internet connection back feed that kind of speed through to your virtual hosts? This is why you want recents close at hand and on a very fast backbone.
Restore speed is why the idiot CEO of Colonial Pipeline paid the ransom, thinking that somehow paying for and getting a decryption key would be speedier than restoring the backups they were already restoring. It's CEOs like this one that make ransomware such a lucrative crime.
Did you backup the pre-detonated ransomware? Are you now going to inadvertently restore it?
One of the tricks the ransomware tricksty hobbittes have in their quiver is to let the encryption engine sit dormant for a period of time before detonating, in hopes of contaminating your backups, so when you restore, boom, another no good very bad day for you. While this is a risk for you, it's also a risk for them as the longer they delay their attack the more likely you are to discover it. pre-encryption. That's not to say it's not a real threat, because it is. And the backup vendors are now integrating scanning directly into the restore process to ensure you don't inadvertently reinfect yourself.
In Veeam's case this feature is called Veeam Secure Restore. There could be some setup involved depending on your requirements so make sure you know what they are before you need it. It will add time to the restore as the virtual disk is mounted and scanned prior to full VM restore, but if you need this level of assurance, it is now available.
Configs, keys and the like
This is where I now extol the virtues of the cloud. You want to backup any and all configuration settings that you may need during a restore. I strongly suggest they be kept in secure cloud location. For example. you can have Veeam backup it's own config DB, ship it via SFTP to a SAN, etc. then ship that off to an AWS bucket. There are a multitude of ways of doing this, but again, automate it. Humans are generally useless when it comes to backup tasks.
Monitor
Yes, Veeam will send emails to you when a job succeeds, fails, burps, has a baby or bar mitzvah. etc. but you, as a general rule won't read them. So use something else to monitor your entire backup infrastructure, for instance Veeam One, or whatever takes your fancy. Here is OP5 (a Nagios derivative) that checks all kinds of jobs:
Protect your backup servers as if your naked pictures were on them
It should go without saying that even non-domain joined servers are still vulnerable. So protect them like nothing else in your data center. They should only allow the bare minimum of inbound connections, and should have firewall rules to prevent anything except management tools in. They should not be pingable, discoverable or any other such thing from anything other than a tiny handful of other devices. A completely separate subnet would be advisable to. Maybe even a hardware firewall between it and everything else. No amount of security around this is too much. Go big or go home.
Additionally, mandate MFA on the OS login (Duo, Okta, etc.) to prevent compromised account access. In short harden this server as you have no other.
Use dedicated log on accounts per backup technician (it's not AD joined remember?) with one-time, not used elsewhere passwords.
Conclusion
While I sincerely hope that you dear reader don't every have to recover from a ransomware incident the odds are not in your favor. This post (and the subsequent 7 other posts) can hopefully help make that no good very bad day just a day or two of downtime and a story to tell at conferences.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
This post is a bit different from the other posts, in that the previous 7 parts were tools and techniques to help prevent the attacks from ever happening (aka the best case scenario). Even if you follow all 7 posts down to the letter, there is still a possibility ransomware will get through your (now) multi-layered defenses. After all you have to be correct every time for everything. Mr and Mrs Hacker only have get it correct once. So plan for the worst and hope for the best. Not the other way round. So this post will cover how to actually put your organization in a place to recover as best as possible were the unthinkable to happen.
While you could pay the ransom, the Sophos - State of Ransomware 2021 report indicates only 8% of paying victims claimed to recover everything. 4% got nothing at all for their payment. On average only 65% of data is restored after an ransomware incident after paying the ransom, so one third of the data is gone, like the snap in Avengers: Infinity Wars, but for data. The average ransom payment was $170,404 USD. But the entire bill for rectifying the attack comes in at a whopping $1,850.000 USD.
The average cost of rectifying a ransomware attack, considering downtime, people time, device cost, network cost, lost opportunity cost, ransom paid. etc. was US$1.85 million.
What I'm about to cover cannot be done with a $100 Microcenter USB external drive and Windows Backup (well, maybe it can, but it shouldn't). Yes, for real backup and recovery build outs can be relatively expensive, but they are far, far less expensive than the average $1,850,000 that it currently costs were you to pay up and all the other things you now have to fix. And once you get hit, YOU WILL BE DOING THIS ANYWAY, so make the argument to do it now. It's not if you will get hit, it's when. And just because you have been hit doesn't mean you won't get hit again. I really wish they'd spend more time on probability in math(s) class.
Alas, sometimes you need a really bad experience to understand the obviously (now with the benefit of hindsight) stupid things you previously did. Exhibit number 1:
So let me start the meat of this post with the most important thing you will ever read in terms of recovering from a ransomware attack.....
Never have any of your backup infrastructure domain joined!
Never have any of your backup infrastructure domain joined!
Never have any of your backup infrastructure domain joined!
No, I'm serious....this includes password and decryption keys as well. So once again, to the chorus.....
Never have any of your backup infrastructure domain joined!
Never. Ever. The stories I have heard....."we had backups but they got encrypted as well"...."we had off-site backups and we even encrypted them for reason x,y,z, however the the private key/password (usually just a text file stored in a "secure" IT Windows file share) was encrypted by the ransomware so our backups are useless". It goes on and on and on. It's extremely common for an organization who gets ransomwared who also has backups that are about as useful as an ashtray on a motorbike. Far more common than you would ever imagine. So plan. And have a plan for when the plan won't work. Print actual copies of any keys you use and put them in a very safe place. Make sure you are not the only one who knows them.
Don't be the guy above that puts temporary hose ramps on a train track. Let's try to save you from that, eh?
For the most part this article will cover Veeam, mainly because of all the systems I've used, it's the easiest and does what it says. You solution du-jour may or may not be able to do the following. If it can't consider changing.
Also this is backup and recovery. Not high availability. Those are two very different things that are != (or <>) to each other at all. While a given product maybe able to do both, I'm not covering both here. HA is a paying gig and track down Lisa if you're interested in that.
Now for the second most important thing to understand about backups.....automate. When humans are involved with backups they fail. All the time. When humans are not involved with backups they fail far less often.
Recognize that not everything needs to be backed up and recoverable
There is some stuff is critical to your organization. Without it you simply cannot function. Back those up. Everything else is optional and is a function of cost vs PIA to rebuild it. For example, SQL servers and AD, sure. But if I had a pretty sizable Tenable install with one or more Nessus Linux scanners feeding it, do I really need to backup *all* the Nessus scanner devices? I would argue no. The value is in the Tenable reports that are harvested from the Nessus scanners. I can rebuild the Nessus scanners at a later date, or just back up one or two of them. Needless to say, the more you back up the more time it takes. Additionally you are taking precious backup resources from other more critical systems.
Frequency and Tagging
Give serious thought to the frequency you need to backup a given device. Break out your backups into these frequencies. Some stuff you want daily, others weekly or even monthly or quarterly. I may backup a given domain controller daily, but others maybe able to be backed up weekly. Also tag the stuff you don't want backed up. Then there is no confusion as to who is to blame when all hell breaks lose and that VM is not in the backup.
Tagging VMs is a way to combat the age old issue of forgetting to add something to the backup. Tagged objects can then be added automatically to backups. Both VMware and HyperV can do this (requiring vCenter and SCVMM respectively). In vCenter create folders for each backup frequency and add a tag to that folder and move VMs to the required folder. Then have Veeam back up that tag. SCVMM is much less user-friendly as you have to tag each VM independently.
Here's a vCenter folder tagged (meaning everything in that folder is also dynamically tagged when Veeam comes looking):
And here is the corresponding Veeam job that adds VMs that match the tag at every execution. Truly dynamic and now you don't need to edit your backup job everytime someone adds a VM. Simply move the VM to the required folder in vCenter and the next time that job runs, the new VM is added to the backup.
SCVMM is a per VM setting, but Veeam is still the same, dynamically adding VMs with associated tag at backup execution time. You cannot set this in HyperV settings, only in SCVMM settings:
Don't forget to backup assets that you will need *during* the recovery. Your PC for example. Also backup and store off-disk the Veeam configuration. You really don't want to have to install a new Veeam server and have it index all the backups across all your different storage tiers. That can add a long time the recovery.
Yes, you do need three tiers of backups
Every knows this already, yet few do it. It's a bit like exercise, we all *know* we should do it and it's not a secret, but doing *it* is a whole different matter. Multi-tier backups are like that. We *know* to do. The majority just don't. And by multi-tier I don't just mean cloud. Cloud for restoring has significant issue which I'll get to later. Just don't go thinking you've avoided all the backup pitfalls by using cloud. Because you haven't.
So a Darren approve system would go something like this....
Backup Location 1: Local disk. Dedicated *only* to the backup system. Not on a shared SAN with everything else. That's simply moronic and your asking for trouble with that approach. Lots and lots of storage. For Veeam your going to want format the storage as ReFS. Local disk has lots of advantages:
- Fast backups. The fastest of backups actually.
- Fast restores. You won't get this with cloud.
- Keep the most recent backups on local disk. This will save time and money when doing normal day-to-day restores of things that users delete. For me recents are 45 to 61 days, depending on your need.
- Disk is cheap to add to. Relatively speaking. Need more? Add disk shelves. Or Veeam servers. Or both.
It does have one pretty big disadvantage:
- It's online, so susceptible to attack. It can be ransomwared. Especially if you are a moron and leave it domain joined. Don't be a moron.
- Relatively OK speed and storage per tape (LTO8 is 12TB uncompressed per tape at 360MB/s....LTO10 and beyond will double the storage of each previous generation). You can have multiple autoloaders off one Veeam server.
- Offline. So extremely low risk of compromise. It's as close to air gapping as you can get and still have a usable backup system.
- Keep the most recent and then some. 90-180 days
- Can be shipped off-site. Try doing that with a disk shelf attached to a Veeam server.
- Great for long term storage.
- Can be made immutable. AWS for example can have Veeam backups made immutable for a period of time, so you can guarantee the backups have not been tampered with.
- Geographically diverse. Not really a ransomware advantage, but still....
- Cloud looks fast when you are backing up to it or moving your backups to it. This is generally because when you backup you are most often backing up incremental changes. These backup files tend to be a tiny fraction of the size that actual full backups would be. Yet when you get hit by ransomware and you have to restore, you are *actually* restoring full backups and not the much smaller incremental backup files. I cannot stress enough how difficult it is to restore a full environment from cloud backups in a timely manner. Basically you can't and it will take a whole lot longer then you ever imagined. It'll take many days to a few weeks. Remember one of the hidden costs of ransomware is the loss of employee productivity, A day is a long time. A week or weeks could put you under.
- It's also expensive to restore from cloud. But it is still way cheaper than paying the ransom.
Build for restore speed
Look, once your hit and you are confident you have good, restorable backups, it's now a time sink, a waiting game if you will. Create restore job, wait, wait, wait. Create restore job wait, wait, wait. The shorter your restore time, the faster you'll be back up and running. So from a restore perspective build the fastest backbone you can. At a minimum I'm talking 10Gb. See 10Gb is literally 10x faster than 1Gb. In real life 10Gb is 5x to 7x faster than 1Gb. That is still a huge factor. See:
10TB restore at 1Gb = ~22 hours
10TB restore at 10Gb = ~4-5 hours
And trust me, when you get hit, 10TB is a tiny amount to restore. If you have 4 VMs hitting 10TB each, on a 1Gb network you'll be up in approx one work week. On a 10Gb network, that is now restored inside of a day.
So this brings me back to the woeful cloud speeds during a restore. Even if your cloud provider were to give you a 10Gb feed back (which I very, very much doubt), can your internet connection back feed that kind of speed through to your virtual hosts? This is why you want recents close at hand and on a very fast backbone.
Restore speed is why the idiot CEO of Colonial Pipeline paid the ransom, thinking that somehow paying for and getting a decryption key would be speedier than restoring the backups they were already restoring. It's CEOs like this one that make ransomware such a lucrative crime.
Did you backup the pre-detonated ransomware? Are you now going to inadvertently restore it?
One of the tricks the ransomware tricksty hobbittes have in their quiver is to let the encryption engine sit dormant for a period of time before detonating, in hopes of contaminating your backups, so when you restore, boom, another no good very bad day for you. While this is a risk for you, it's also a risk for them as the longer they delay their attack the more likely you are to discover it. pre-encryption. That's not to say it's not a real threat, because it is. And the backup vendors are now integrating scanning directly into the restore process to ensure you don't inadvertently reinfect yourself.
In Veeam's case this feature is called Veeam Secure Restore. There could be some setup involved depending on your requirements so make sure you know what they are before you need it. It will add time to the restore as the virtual disk is mounted and scanned prior to full VM restore, but if you need this level of assurance, it is now available.
Configs, keys and the like
This is where I now extol the virtues of the cloud. You want to backup any and all configuration settings that you may need during a restore. I strongly suggest they be kept in secure cloud location. For example. you can have Veeam backup it's own config DB, ship it via SFTP to a SAN, etc. then ship that off to an AWS bucket. There are a multitude of ways of doing this, but again, automate it. Humans are generally useless when it comes to backup tasks.
Monitor
Yes, Veeam will send emails to you when a job succeeds, fails, burps, has a baby or bar mitzvah. etc. but you, as a general rule won't read them. So use something else to monitor your entire backup infrastructure, for instance Veeam One, or whatever takes your fancy. Here is OP5 (a Nagios derivative) that checks all kinds of jobs:
Protect your backup servers as if your naked pictures were on them
It should go without saying that even non-domain joined servers are still vulnerable. So protect them like nothing else in your data center. They should only allow the bare minimum of inbound connections, and should have firewall rules to prevent anything except management tools in. They should not be pingable, discoverable or any other such thing from anything other than a tiny handful of other devices. A completely separate subnet would be advisable to. Maybe even a hardware firewall between it and everything else. No amount of security around this is too much. Go big or go home.
Additionally, mandate MFA on the OS login (Duo, Okta, etc.) to prevent compromised account access. In short harden this server as you have no other.
Use dedicated log on accounts per backup technician (it's not AD joined remember?) with one-time, not used elsewhere passwords.
Conclusion
While I sincerely hope that you dear reader don't every have to recover from a ransomware incident the odds are not in your favor. This post (and the subsequent 7 other posts) can hopefully help make that no good very bad day just a day or two of downtime and a story to tell at conferences.
July 9 2021 Friday
Ransomware Prevention Part 7 - Email Security
Part 7 - Email security
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Most malware enters via email, a March, 2020 report from CSO Online reports that email is the vector for 94% of malware attacks. That same reports the Phishing attacks are involved in 60% of attacks. To say email is the front door for most attacks is a pretty apt metaphor.
Stopping the multitude of malicious emails before they are ever delivered to your users can prevent a whole lot of attacks. Since the dawn of enterprise SMTP email, this had been the great struggle between good and evil. And still it rages on. I'd be shocked if most organizations of any size are not using any type of email spam filter. If you are not, look no further than SpamHero. It's relatively inexpensive and while lacking the sophisticated tooling of some of the products below, it it orders of magnitude better than nothing at all.
So what are your options? A lot of this is available from most tier 1 vendors (Barracuda, Proofpoint, Cisco, etc) but YMMV and it may be extra licensing costs to add a specific feature.
GeoIP/Regional Blocking
This used to be simple but the advent of Office365 and the various acts of government (i.e. the Patriot Act) makes it more complicated and a game of whack-a-mole. For example US based subsidiary of a Japanese corporation may use Office365 that exits from Japan. Some Microsoft Office 365 status emails now originate from Singapore. See, whack-a-mole.
Of course, use GeoIP or regional blocking to filter out the obvious contenders, Russia, Iran, etc, but you really want to limit it as much as possible.
Advanced Threat Protection (ATP)
If there is one add-on that most do not have, but all should, it is advanced threat protection (ATP). This (usually optional add-on) will take attachments embedded in an email and execute it in a cloud sandbox. ATP is a bit like a Number 7 bus, none come along for a long time then all of a sudden several (hundred) turn up at once.
Here's an example from Barracuda Cloud ESS ATP. They also provide a report, although to date I have yet to see any false positives:
Active Content Disarming
Not a common feature (sadly), but this essentially neuters all links within the attachment. So if an entire PDF page is a hidden link that tests if you are using a vulnerable version of Acrobat (hint, you are.....every version of Acrobat is a vulnerability) then this link is removed as it's active content. Thus a user can no longer accidentally click on the link. To date the only product I have seen that can do this is LibraESVA.
URL Protection/SafeLinks
Rewrites URLs in emails so they can be scanned when clicked by the user for malicious intent. Somewhat ironically makes spotting a bad URL with the mark-1 human eyeball an impossible task (and negates some of your cyber-security awareness training your users are doing). I actually really, really dislike Barracuda's implementation and really, really like LibraESVAs as it shows you can actual scan happening. Barracuda, not so much.
Can be used in conjunction with KnowBe4 Second Chance (if you have it) which will unwind the real URL and show it to the user for confirmation.
Reverse DNS
Come on people. Just block anything that doesn't have a reverse DNS pointer. You should have been doing this since 1999.
Sender Protection Framework (SPF)
Now we come to the trifecta of semi-related options. We'll start with SPF. It tells the receiving server if the sending server is authorized to send on behalf of the senders domain. It's does this via DNS. I'll make this easy on you, block anything with a hard SPF fail and quarantine anything with a soft SPF fail. Also you should have SPF set up in your DNS for your outbound email so to let others know. As with all things email security, pass it forward.
If you use them, don't forget to add Salesforce, MailChimp, ConstantContact, et al as allowed SPF senders on your outbound SPF based on their applicable documentation.
Domain Keys Identified Mail (DKIM)
Now it's getting tricky. Where SPF tells you if a server is allowed to send, DKIM takes it a step further and ensures (via PKI and DNS) the received email has not been during tampered with during transmission and that the sender server is authorized to send on behalf of that domain. In a nutshell it adds cryptographic authentication to email (a bit like SSL certificate chains in a web browser, I am who I claim to be).
When done correctly, DKIM can certify that an email is either legitimate or illegitimate. In a perfect world you'd simply discard any illegitimate email. Alas poor reader, a perfect world this is not.....
There is a lot of DKIM out there. A lot of it is configured incorrectly. Which is sad as this could really clean up the world of email. It could literally prevent phishing attacks overnight if everyone enabled it (correctly). You could block or quarantine any that fail, but a LOT will fail, mainly because of mis-configuration on the senders side. It's worth noting that DKIM won't stop malicious email from legitimately signed DKIM servers (sendgrid anyone?).
Again, add DKIM to your outbound flow to pass it forward, the same warnings about 3rd party senders for SPF also apply here, so follow their documentation.
DMARC
DMARC is the odd one out of the three in that it really is an extension of SPF and/or DKIM. Like the other two it is also a DNS record. It tells the recipient how to check SPF, DKIM and the from address in an email. More importantly is tells the receiving server what to do with failures. DMARC also adds reporting to the mix. You can get reports that *can* indicate someone is spoofing your domain. DMARC reporting is pretty complex and you'd usually have a 3rd party go this and collate the results.
Using SPF, DKIM and DMARC correctly really does have the potential to stop most malicious and unwanted email, but alas the world is full of people who don't know what they are doing, or worse, end around IT and start having a 3rd party send email on your behalf which never gets delivered.
Conclusion
Email is still how the majority of attackers get into your networks. This is your Maginot Line from a security perspective and you need to have as many bells and whistles enabled as possible. Add this to cybersecurity awareness training of your users and you can stop 99.8% of attacks at the gates,
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
Most malware enters via email, a March, 2020 report from CSO Online reports that email is the vector for 94% of malware attacks. That same reports the Phishing attacks are involved in 60% of attacks. To say email is the front door for most attacks is a pretty apt metaphor.
Email is the ingress point for 94% of malware attacks.
Stopping the multitude of malicious emails before they are ever delivered to your users can prevent a whole lot of attacks. Since the dawn of enterprise SMTP email, this had been the great struggle between good and evil. And still it rages on. I'd be shocked if most organizations of any size are not using any type of email spam filter. If you are not, look no further than SpamHero. It's relatively inexpensive and while lacking the sophisticated tooling of some of the products below, it it orders of magnitude better than nothing at all.
So what are your options? A lot of this is available from most tier 1 vendors (Barracuda, Proofpoint, Cisco, etc) but YMMV and it may be extra licensing costs to add a specific feature.
GeoIP/Regional Blocking
This used to be simple but the advent of Office365 and the various acts of government (i.e. the Patriot Act) makes it more complicated and a game of whack-a-mole. For example US based subsidiary of a Japanese corporation may use Office365 that exits from Japan. Some Microsoft Office 365 status emails now originate from Singapore. See, whack-a-mole.
Of course, use GeoIP or regional blocking to filter out the obvious contenders, Russia, Iran, etc, but you really want to limit it as much as possible.
Advanced Threat Protection (ATP)
If there is one add-on that most do not have, but all should, it is advanced threat protection (ATP). This (usually optional add-on) will take attachments embedded in an email and execute it in a cloud sandbox. ATP is a bit like a Number 7 bus, none come along for a long time then all of a sudden several (hundred) turn up at once.
Here's an example from Barracuda Cloud ESS ATP. They also provide a report, although to date I have yet to see any false positives:
Active Content Disarming
Not a common feature (sadly), but this essentially neuters all links within the attachment. So if an entire PDF page is a hidden link that tests if you are using a vulnerable version of Acrobat (hint, you are.....every version of Acrobat is a vulnerability) then this link is removed as it's active content. Thus a user can no longer accidentally click on the link. To date the only product I have seen that can do this is LibraESVA.
URL Protection/SafeLinks
Rewrites URLs in emails so they can be scanned when clicked by the user for malicious intent. Somewhat ironically makes spotting a bad URL with the mark-1 human eyeball an impossible task (and negates some of your cyber-security awareness training your users are doing). I actually really, really dislike Barracuda's implementation and really, really like LibraESVAs as it shows you can actual scan happening. Barracuda, not so much.
Can be used in conjunction with KnowBe4 Second Chance (if you have it) which will unwind the real URL and show it to the user for confirmation.
Reverse DNS
Come on people. Just block anything that doesn't have a reverse DNS pointer. You should have been doing this since 1999.
Sender Protection Framework (SPF)
Now we come to the trifecta of semi-related options. We'll start with SPF. It tells the receiving server if the sending server is authorized to send on behalf of the senders domain. It's does this via DNS. I'll make this easy on you, block anything with a hard SPF fail and quarantine anything with a soft SPF fail. Also you should have SPF set up in your DNS for your outbound email so to let others know. As with all things email security, pass it forward.
If you use them, don't forget to add Salesforce, MailChimp, ConstantContact, et al as allowed SPF senders on your outbound SPF based on their applicable documentation.
Domain Keys Identified Mail (DKIM)
Now it's getting tricky. Where SPF tells you if a server is allowed to send, DKIM takes it a step further and ensures (via PKI and DNS) the received email has not been during tampered with during transmission and that the sender server is authorized to send on behalf of that domain. In a nutshell it adds cryptographic authentication to email (a bit like SSL certificate chains in a web browser, I am who I claim to be).
When done correctly, DKIM can certify that an email is either legitimate or illegitimate. In a perfect world you'd simply discard any illegitimate email. Alas poor reader, a perfect world this is not.....
There is a lot of DKIM out there. A lot of it is configured incorrectly. Which is sad as this could really clean up the world of email. It could literally prevent phishing attacks overnight if everyone enabled it (correctly). You could block or quarantine any that fail, but a LOT will fail, mainly because of mis-configuration on the senders side. It's worth noting that DKIM won't stop malicious email from legitimately signed DKIM servers (sendgrid anyone?).
Again, add DKIM to your outbound flow to pass it forward, the same warnings about 3rd party senders for SPF also apply here, so follow their documentation.
DMARC
DMARC is the odd one out of the three in that it really is an extension of SPF and/or DKIM. Like the other two it is also a DNS record. It tells the recipient how to check SPF, DKIM and the from address in an email. More importantly is tells the receiving server what to do with failures. DMARC also adds reporting to the mix. You can get reports that *can* indicate someone is spoofing your domain. DMARC reporting is pretty complex and you'd usually have a 3rd party go this and collate the results.
Using SPF, DKIM and DMARC correctly really does have the potential to stop most malicious and unwanted email, but alas the world is full of people who don't know what they are doing, or worse, end around IT and start having a 3rd party send email on your behalf which never gets delivered.
Conclusion
Email is still how the majority of attackers get into your networks. This is your Maginot Line from a security perspective and you need to have as many bells and whistles enabled as possible. Add this to cybersecurity awareness training of your users and you can stop 99.8% of attacks at the gates,
June 16 2021 Wednesday
Ransomware Prevention Part 6 - GPO tricks and tips
Part 6 - GPO tricks and tips
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
If you only read one of this series, this one should be it. Seriously. And read it all a few times before you start editing the default domain policy!
Most of this series is dedicated from stopping any potential ransomware from getting to the install or execution point. But what happens if all your many Darren-approved, onion skin layers of security fail and the nasty does get through and it does execute? In this worse case scenario GPOs or Local Security Policies (if you are not AD joined) are your friends. I have implemented the techniques in this post to prevent whack-a-mole reoccurrences of a Ryuk ransomware attack. These techniques are that powerful.
The basics - how ransomware works
A rather large caveat. Your users should not be local Windows admins on their machines. If they are you have a somewhat larger issue to fix. The fix being "stop doing that".
If your users are not local Windows admins then how does ransomware execute and install? Simple, it installs and/or runs in the users local profile context. That handy c:\users\\ folder. The one that the likes of WebEx, Zoom, Teams, et al all install and run from. Yeah, there.
So the same useful Windows features that lets you work from home, do video calls and not wear anything below the waist is also the same mechanism ransomware uses to install and execute. Ransomware is usually a series of different malware applications, each with a specific use case. There is some type of "dropper" that is what the unwitting user clicks on, downloads, allows a MS Office macro to run, or otherwise executes. Once the dropper is in place it will attempt to install one or several different programs from the internet to gain a foothold in your network, These "several different things" (that can happen over a series of days, weeks or months so you do see them for what they are) include:
A few years ago step 2 was relatively uncommon. Not anymore as it appears to be pretty good leverage at getting you to pay. Not necessarily to decrypt your data, but to get the hackers to promise they will delete this exfiltrated, sensitive data and they will not publicly release it. A promise. What could possibly go wrong.
In step 1 and 2 the hackers are almost always looking for server based file shares or access into server operating systems (think SQL Server, Exchange, etc) these days. The idea being that the more users I can affect with one attack the more likely you are to be willing to pay. If I encrypt just your files you are unlikely to pay. If I encrypt critical, run your business files that 10, 100, or 1000 users require to work then the pain increases by many orders of magnitude.
Pro-tip, don't pay. Follow this series (this post the the upcoming backup one especially) and you won't have to. I really need to do a "what if you pay" post at some point to so you realize paying for decryption isn't all they promise it will be.
OK, so now we know where and how this stuff works, how do you stop it if none of the other 8+ posts in this series saved me? You prevent it from running.
Prevent it from running in the first place
You prevent it from running by whitelisting. Now just the term whitelisting sends IT professionals off into the woods to remove their clothing, revert to their prehistoric selves never to be seen again. But hear me out before you quit, strip off and go full on paleo in the wilderness.... So long as your users are not local admins and have no rights to install software in to Program Files, etc. then all you need to do is to whitelist applications that you wish to specifically allow to run inside the aforementioned appdata context. This is much, much smaller nut to crack. Why? Because next to nothing *should* be running from the user profile or appdata folder (I say *should* because there are usually way more than you would expect).
Inside your Active Directory Group Policy Object (GPO) and the local security policy is a handy little thing called Software Restriction Policies (SPR). SRPs can be set to not allow anything to run in a specific folder on a Windows device. Additionally the SRP can be expanded to allow only what you want to run:
With a SRP you can easily block exe, Powershell, Zip, 7z, rar, etc. from running is a users appdata context (this is also where the users temp is located to which is another execution hotbed).
Below is an actual SRP. Notice the security level column? Disallowed means you're not running. Using a disallow with a path rule and using Windows environment variables your can simply and effectively block all exe's for all users appdata contexts. Conversely a security level of unrestricted will allow anything that matches to execute. In this example anything signed with the uploaded Adobe Inc signing certificate will be allowed to run, as is AMD, Barracuda, etc.:
SRPs can be set to allow four different ways:
The problems with SRPs
Well, quite simply they stop stuff working by design. When you enable them, programs that previously worked could just stop. This means you need to build out your exception list as fully as possible before enabling the policy. Scour your users appdata folders for exes and you will find (and be able to extract and upload signing certificates) the likes of Adobe, Teams, WebEx, Zoom, Go To Meeting, BlueJeans and all kinds of other web conferencing tools you never heard of. All of these most likely need to be added. Note, most of the web conferencing tools also have a "machine wide" installer that forgoes the need for each and every user to download and these tools. As these machine-wide installers utilize Program Files folders they don't fall foul of SRPs (when you create an SRP the GPO auto-add exceptions for this file path). Start with a small set if users and work out from there.
The 2nd issue is find out what was blocked and why. When a block occurs the user is shown this not very useful error:
Doesn't tell you what was blocked or why. For that you have to look at the local machines event log. If a cunning user or hacker copies a exe to their user profile folder and executes it, not only will they see the message above, but something along the lines of this will be written to the event log:
Obviously managing this for even a small number of PCs can be time consuming when you first enable these policies, so if you have some type of central logging system you can better report on the things that are happening and/or need to be added as exceptions. Here is a SIEM (Eventlog Analyzer) that shows a blocked 7z execution:
With a SIEM (or any other reporting solution that extracts local event logs) it becomes much easier to proactively manage SRPs. For instance you can send a report to your security team listing yesterdays blocks. They can then investigate.
Scheduled Tasks
Another common attack area of ransomware is to install innocuous looking scheduled tasks that will attempt to reinfect or re-detonate the malware tools on reboot or on a scheduled basis. There is little use in a regular, non-admin users being able to create a Windows OS level scheduled task, so simply preventing these users from creating them is simple and effective way to head off this line of attack. This is available in the computer and user policies under Administrative Templates, Windows Components, Task Scheduler. Simply prohibit new task creation:
Conclusion
While one can never guarantee an attack will be prevented (SolarWinds anyone?), whitelisting is about as close to a guarantee as you can get. Added to the onion-skin of protection you build around your devices and (touch wood) you will never have to contemplate paying a ransom or restoring from backups. It is also worth noting that Microsoft has several different options to SPR, AppLocker being the most obvious other choice. Either is fine, I just do a lot more SRP than anything else.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
If you only read one of this series, this one should be it. Seriously. And read it all a few times before you start editing the default domain policy!
Most of this series is dedicated from stopping any potential ransomware from getting to the install or execution point. But what happens if all your many Darren-approved, onion skin layers of security fail and the nasty does get through and it does execute? In this worse case scenario GPOs or Local Security Policies (if you are not AD joined) are your friends. I have implemented the techniques in this post to prevent whack-a-mole reoccurrences of a Ryuk ransomware attack. These techniques are that powerful.
The basics - how ransomware works
A rather large caveat. Your users should not be local Windows admins on their machines. If they are you have a somewhat larger issue to fix. The fix being "stop doing that".
If your users are not local Windows admins then how does ransomware execute and install? Simple, it installs and/or runs in the users local profile context. That handy c:\users\
Ransomware (usually) runs in the user profile folder.
So the same useful Windows features that lets you work from home, do video calls and not wear anything below the waist is also the same mechanism ransomware uses to install and execute. Ransomware is usually a series of different malware applications, each with a specific use case. There is some type of "dropper" that is what the unwitting user clicks on, downloads, allows a MS Office macro to run, or otherwise executes. Once the dropper is in place it will attempt to install one or several different programs from the internet to gain a foothold in your network, These "several different things" (that can happen over a series of days, weeks or months so you do see them for what they are) include:
- Reconnoiter - find what is on the network, what it can get to, find lateral move points and search for systems to compromise (meaning un-patched known, exploitable vulnerabilities).
- Exfiltrate - take your data off-site so if you don't pay the ransom to unlock your files, they can still have leverage over you and threaten to release sensitive information.
- Encryption engine - the program that will download a public key (almost always AES, so to all intents and purposes uncrackable) from a command and control server. It then begins to encrypt items located in 1. Encryption usually begins at the start of a weekend to give the ransomware enough time to do real damage based on the hope that no one is looking at the servers on a weekend. Mondays can be very bad.
- Profit.
This is about as simple as it gets. Find your stuff. Steal your stuff. Encrypt your stuff. Profit.
A few years ago step 2 was relatively uncommon. Not anymore as it appears to be pretty good leverage at getting you to pay. Not necessarily to decrypt your data, but to get the hackers to promise they will delete this exfiltrated, sensitive data and they will not publicly release it. A promise. What could possibly go wrong.
In step 1 and 2 the hackers are almost always looking for server based file shares or access into server operating systems (think SQL Server, Exchange, etc) these days. The idea being that the more users I can affect with one attack the more likely you are to be willing to pay. If I encrypt just your files you are unlikely to pay. If I encrypt critical, run your business files that 10, 100, or 1000 users require to work then the pain increases by many orders of magnitude.
Pro-tip, don't pay. Follow this series (this post the the upcoming backup one especially) and you won't have to. I really need to do a "what if you pay" post at some point to so you realize paying for decryption isn't all they promise it will be.
OK, so now we know where and how this stuff works, how do you stop it if none of the other 8+ posts in this series saved me? You prevent it from running.
Prevent it from running in the first place
You prevent it from running by whitelisting. Now just the term whitelisting sends IT professionals off into the woods to remove their clothing, revert to their prehistoric selves never to be seen again. But hear me out before you quit, strip off and go full on paleo in the wilderness.... So long as your users are not local admins and have no rights to install software in to Program Files, etc. then all you need to do is to whitelist applications that you wish to specifically allow to run inside the aforementioned appdata context. This is much, much smaller nut to crack. Why? Because next to nothing *should* be running from the user profile or appdata folder (I say *should* because there are usually way more than you would expect).
Inside your Active Directory Group Policy Object (GPO) and the local security policy is a handy little thing called Software Restriction Policies (SPR). SRPs can be set to not allow anything to run in a specific folder on a Windows device. Additionally the SRP can be expanded to allow only what you want to run:
SRPs - block everything, except what I specifically allow.
With a SRP you can easily block exe, Powershell, Zip, 7z, rar, etc. from running is a users appdata context (this is also where the users temp is located to which is another execution hotbed).
Below is an actual SRP. Notice the security level column? Disallowed means you're not running. Using a disallow with a path rule and using Windows environment variables your can simply and effectively block all exe's for all users appdata contexts. Conversely a security level of unrestricted will allow anything that matches to execute. In this example anything signed with the uploaded Adobe Inc signing certificate will be allowed to run, as is AMD, Barracuda, etc.:
SRPs can be set to allow four different ways:
- Path - specify an allowed file or folder path (i.e. %appdata%\Temp\Teams\*). This is the most insecure type as *anything* in that folder will be allowed to execute, and hackers know many common folders (a lot of malware adds folders called Google Chrome or Chrome to these paths). It is also the easiest exception to add. Try your hardest to not use this type of exception. Very good for disallow rules. This is, after all, what you are trying to prevent.
- Hash - the file hash of a selected exe. This is also pretty easy to allow, but *ANY* change to the file (so an upgrade to a new Zoom version that replaces zoom.exe) will prevent it from running as those file hashes no longer match. Use this for vendors who refuse to use signing certificates (also find a new vendor).
- Network zone - I'm going to skip this as it's of little use when trying to protect a local machine, and using this could seriously increase your risk to lateral movement of malware in the network.
- Certificate based - the most difficult to do as you need to extract and upload the digital signing certificate from an exe to the SRP (and sometimes more than one). It is also not enabled by default. It is however the most secure (only exe's signed with said digital certificate can run) and it bypasses the issues with hashes as upgraded versions of programs (like zoom.exe) are likely to be signed with the same signing certificate. Certificates do expire or are revoked so this is not quite fire and forget. Indeed just the last few weeks Bitdefender changed signing certs so these had to be updated.
The problems with SRPs
Well, quite simply they stop stuff working by design. When you enable them, programs that previously worked could just stop. This means you need to build out your exception list as fully as possible before enabling the policy. Scour your users appdata folders for exes and you will find (and be able to extract and upload signing certificates) the likes of Adobe, Teams, WebEx, Zoom, Go To Meeting, BlueJeans and all kinds of other web conferencing tools you never heard of. All of these most likely need to be added. Note, most of the web conferencing tools also have a "machine wide" installer that forgoes the need for each and every user to download and these tools. As these machine-wide installers utilize Program Files folders they don't fall foul of SRPs (when you create an SRP the GPO auto-add exceptions for this file path). Start with a small set if users and work out from there.
The 2nd issue is find out what was blocked and why. When a block occurs the user is shown this not very useful error:
Doesn't tell you what was blocked or why. For that you have to look at the local machines event log. If a cunning user or hacker copies a exe to their user profile folder and executes it, not only will they see the message above, but something along the lines of this will be written to the event log:
Obviously managing this for even a small number of PCs can be time consuming when you first enable these policies, so if you have some type of central logging system you can better report on the things that are happening and/or need to be added as exceptions. Here is a SIEM (Eventlog Analyzer) that shows a blocked 7z execution:
With a SIEM (or any other reporting solution that extracts local event logs) it becomes much easier to proactively manage SRPs. For instance you can send a report to your security team listing yesterdays blocks. They can then investigate.
Scheduled Tasks
Another common attack area of ransomware is to install innocuous looking scheduled tasks that will attempt to reinfect or re-detonate the malware tools on reboot or on a scheduled basis. There is little use in a regular, non-admin users being able to create a Windows OS level scheduled task, so simply preventing these users from creating them is simple and effective way to head off this line of attack. This is available in the computer and user policies under Administrative Templates, Windows Components, Task Scheduler. Simply prohibit new task creation:
Conclusion
While one can never guarantee an attack will be prevented (SolarWinds anyone?), whitelisting is about as close to a guarantee as you can get. Added to the onion-skin of protection you build around your devices and (touch wood) you will never have to contemplate paying a ransom or restoring from backups. It is also worth noting that Microsoft has several different options to SPR, AppLocker being the most obvious other choice. Either is fine, I just do a lot more SRP than anything else.
June 3 2021 Thursday
Ransomware Prevention Part 5 - Cybersecurity awareness
Part 5 - Cybersecurity awareness
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
You can add all the security in the world, at the end of the day it is your end-users who either click and download malware or give their credentials to a phishing site. It is our job to help them by either providing education and/or changing their behavior. This makes cybersecurity a team sport. It's a shared responsibility. It was never just a function of IT although many tried (and still do) to make it this way. The more players on your side the better the outcome you will have. You can quite easily increase the size of your team by utilizing cybersecurity awareness.
Cybersecurity awareness is quite possibly the only interaction actual users get from which they could glean a snippet of knowledge that could mean the difference between a ransomware attack and just another deleted email. Awareness training is becoming more common place now (thanks to audits and insurance questionnaires), whereas just as little as three years ago it was next to none existent. As I have mentioned elsewhere in this series, no one solution can or will stop every nasty that tries to get through. Your users could be your last line of defence, and their decision to click on a link or not could be the inflection point that is the difference. That being said, a recent report from Tessian (the psychology of human error) is indicative of the risks posed by employees and the hackers ability to bypass even the most stringent of email security measures.
I'll rephrase the above for you.....90% of breaches are caused by user error. 90%. 10% shy of 100%.
Indeed the report makes for dire reading, with 25% or respondents admitting to clicking on a phishing email, with the younger (under 40), and especially males being much more susceptible than any other group. Probably the most eye-popping statistic in the report is this:
Given a combined one third of your workforce never give cybersecurity a second thought something needs to change. What needs to change is how your user population understands the risks that, for whatever reason, make it past the vast layers of security organizations have. Indeed, employees are often called the weakest link, yet they are often the last line of defence in this on-going battle to prevent the cyber criminals for gaining a foothold in your network. It would appear enterprise IT is doing a woeful job at communication and training. That cybersecurity is a shared responsibility needs to be shouted from the hills, and shouted often.
To make matters worse, a report from KnowBe4 (Security culture report 2021) states that:
The above statement is an absurd notion (it's a least an order of magnitude too high, if not two), but to make matters worse only 20% of respondents reported to needing more training. Essentially if the aforementioned results hold true, then is it any surprise that organization after organization falls foul of the cyber criminals?
So how do we overcome this apparent gap in what employees believe they know and what they actually know? Cybersecurity awareness training. Spoiler alert, you simply can't do this alone. You need assistance from one of the above mentioned (or the many not mentioned) to help close the gap. Don't get me wrong, cybersecurity awareness training is no panacea, it is however a good starting point and just moving the knowledge needle 5% is still moving it. So while organizations are embracing it, I see massive room for improvement.
Episode I Episode IV - A new hope
You may already have a program in place, but even if you do how effective is it if your employees only see it once per year? Not very. So the first step to overcoming these hurdles is to define what you are doing. A once annual 5 minute video is not going to cut it. I know KnowBe4 pretty well so that is what I will cover here, but most providers such as Barracuda Phishline also provide some of these features. So here's a series of suggestions to add to, replace, or when creating a cybersecurity awareness program:
With your first simulated phishing campaign (hint, never offer free money in your campaigns, it could make you famous for all the wrong reasons) you should now have a series of hard facts that you can work on:
With this in hand you can now target remediation (do some users need to retake the 45 minute course? Do I need to add extra content?) or add in other tools to assist the users. Tools? Yes tools.
A lot of organizations have filters in between the users and their email. Happily rewriting links in email so as to be confusing to a human as possible but hopefully preventing the user from navigating to a malicious web site. Indeed one of the most common ways to spot a phishing email is to look at the target URL. Our additional layers of security have just negated some of the video training your users will do. Fantastic!!! The good news is that there are tools starting to percolate out that help decipher these seemingly incomprehensible URLs. KnowBe4 have add-in named Second Chance that for certain desktop email clients that will show the user the actual link they are clicking on. It turns this jibberish behind an email button:
Into this warning that decodes the link:
Now if someone could make this a universal plugin that also works with web based email, we'd have a winner. Still, it's a start and if you have KnowBe4 there's a good chance you don't know about Second Chance.
Another tool to empower users is VirusTotal. There are plugins for most browsers that will allow users to self-check worrisome URLs and/or files. IT may not always be available or accessible, the internet however is. Finally telling user about HaveIBeenPwned is seeing them use it is quite the sight to behold.
Password reuse
Beyond end-user training is end-user education. What they don't know because you didn't tell them can, and often will, hurt you. As I mentioned earlier, the online video how-to's are no panacea. Some don't even touch on password hygiene or reuse. From some truly shocking (not shocking) statistics on passwords, look no further than the Compaitech Password Statistics page. Some highlights (or more correctly low lights):
Google found that :
Also present in this page is maybe the most disheartening statistic (again, surprised, not surprised):
Yet again the IT professionals unerring belief that they are superhuman and immune from the perils that only mere mortals fall for strikes again. How the use of enterprise password managers such as ManageEngine's Password Manager Pro or Keeper Enterprise is not mandated in every IT department on the planet is beyond me. I'm often stunned by an organization's desire to keep passwords less than or equal to 8 characters (the Windows GPO default). Simply making them longer and requiring a special character can do wonders for password security. An oldie but goodie is this LifeHacker article on passwords. I'll sum it up with this table which outlines the estimated time to brute force a password based on adding on an upper-case and special character vs lower-case only:
Yeah, as as IT professional you'll want at least 12 characters for your own passwords, and at least 10 for your end-users. So how does on overcome the perils of password reuse, woeful complexity and overall crappy password hygiene? Multi-factor authentication or MFA. Or 2FA.
MFA is incredibly effective at prevent credential theft. A 2019 Microsoft study has it as high as 99.9% effective. Given that success rate you would expect almost every organization to have implemented it right? In Wrong. While I admit it can be complex and relatively expensive (much less so that being ransomwared FWIW) just over half of organizations in 2019 have implemented MFA (57%). In fact a 2021 report from the Fido Alliance indicates that 91% of MFA projects are to prevent credential theft.
So *where* to you do MFA? Well, everywhere, or not. The possible exception is when you are in a trusted location (read on-network, on-LAN). There is little use having MFA enabled in your corporate LAN when accessing Office365 and you already have 12 character strong passwords and SSO is enabled. All you do is piss your users off with little effect to your overall security posture. However when accessing *anything* from outside the LAN you'd want MFA. MFA to VPN. MFA for Office365. MFA for Azure App Proxy. If I'm coming from the outside to the inside (and even if inside is an externally hosted cloud service) you need to require MFA.
Now there are some select users who should be forced to use MFA even when inside the corporate LAN. You. The IT admin. The Domain Admin. The people with the keys to the kingdom. At every logon. At every screen lock. Every time. And your critical servers too. DMZ servers. Proxy servers. Domain controller. Every. Single. Time. How you'd do this is a little complex now that Microsoft foisted Windows Hello on the world (don't use Windows Hello). but would probably involve Cisco Duo, Okta or the like. Why?
Because you are part of the problem. Now you can be part of the solution.
I often hear MFA is expensive and difficult (I'll give you the latter point), but every Office365 license has the ability to do MFA. Everyone license. Now you'll need something like Azure P1 or P2 (or Duo, or Okta, or any of the other providers of enterprise SSO) to get some of the more useful features such as trusted locations (not requiring MFA for Office365 on the LAN), but it does have it and you can implement it. And you should because a 2019 article from TechRepublic citing a report from Cyren and Osterman Research states that a staggering 40% of enterprises experienced Office 365 credential theft. And if those stolen credentials happen to be the ones you use for AD (because SSO and DirSync) then a users AD credentials have just been compromised. And if said user is a domain admin level of user....yeah, now you can see how these attacks you read about happen. MFA FTW!
Conclusion
Your end-user population can the difference between a ransomware meltdown and none event. Engage them, train them, educate them. After all cybersecurity is a team sport. Build a program, create an internal blog. Because even an incremental increase in knowledge is an increase. And you need all the help you can get.
Finally, roll out MFA. Yes it's difficult. Yes it can be somewhat costly, The the results in decreasing credential theft are simply astounding. Oh, and change your password policies to at least 10 characters with a requirement for a special character.
See here for the entire series of posts, if you are just stumbling onto these posts.
As I said in part one, these post are supposed to be helpful in giving you meaningful useful advice to prevent ransomware.
You can add all the security in the world, at the end of the day it is your end-users who either click and download malware or give their credentials to a phishing site. It is our job to help them by either providing education and/or changing their behavior. This makes cybersecurity a team sport. It's a shared responsibility. It was never just a function of IT although many tried (and still do) to make it this way. The more players on your side the better the outcome you will have. You can quite easily increase the size of your team by utilizing cybersecurity awareness.
Cybersecurity awareness is quite possibly the only interaction actual users get from which they could glean a snippet of knowledge that could mean the difference between a ransomware attack and just another deleted email. Awareness training is becoming more common place now (thanks to audits and insurance questionnaires), whereas just as little as three years ago it was next to none existent. As I have mentioned elsewhere in this series, no one solution can or will stop every nasty that tries to get through. Your users could be your last line of defence, and their decision to click on a link or not could be the inflection point that is the difference. That being said, a recent report from Tessian (the psychology of human error) is indicative of the risks posed by employees and the hackers ability to bypass even the most stringent of email security measures.
9 out of 10 breaches are caused by end user mistakes.
I'll rephrase the above for you.....90% of breaches are caused by user error. 90%. 10% shy of 100%.
Indeed the report makes for dire reading, with 25% or respondents admitting to clicking on a phishing email, with the younger (under 40), and especially males being much more susceptible than any other group. Probably the most eye-popping statistic in the report is this:
11% never think about cybersecurity at work and a further 22% rarely.
Given a combined one third of your workforce never give cybersecurity a second thought something needs to change. What needs to change is how your user population understands the risks that, for whatever reason, make it past the vast layers of security organizations have. Indeed, employees are often called the weakest link, yet they are often the last line of defence in this on-going battle to prevent the cyber criminals for gaining a foothold in your network. It would appear enterprise IT is doing a woeful job at communication and training. That cybersecurity is a shared responsibility needs to be shouted from the hills, and shouted often.
To make matters worse, a report from KnowBe4 (Security culture report 2021) states that:
An astounding 57% of employees believe they would recognize if their device got hacked.
The above statement is an absurd notion (it's a least an order of magnitude too high, if not two), but to make matters worse only 20% of respondents reported to needing more training. Essentially if the aforementioned results hold true, then is it any surprise that organization after organization falls foul of the cyber criminals?
So how do we overcome this apparent gap in what employees believe they know and what they actually know? Cybersecurity awareness training. Spoiler alert, you simply can't do this alone. You need assistance from one of the above mentioned (or the many not mentioned) to help close the gap. Don't get me wrong, cybersecurity awareness training is no panacea, it is however a good starting point and just moving the knowledge needle 5% is still moving it. So while organizations are embracing it, I see massive room for improvement.
Episode I Episode IV - A new hope
You may already have a program in place, but even if you do how effective is it if your employees only see it once per year? Not very. So the first step to overcoming these hurdles is to define what you are doing. A once annual 5 minute video is not going to cut it. I know KnowBe4 pretty well so that is what I will cover here, but most providers such as Barracuda Phishline also provide some of these features. So here's a series of suggestions to add to, replace, or when creating a cybersecurity awareness program:
- Make sure everyone understands cybersecurity is a team sport. Users can't do it without help from you, and you can't do it without help from the users.
- Start with education in mind, never blame. If a user thinks that they may have done something to compromise security you want to them to notify you as soon as possible. Using blame is a sure fire way to ensure you will never be notified and this could be the difference between a successful defence and a successful attack.
- Don't start with a phishing attack simulation. That just leads to huge amounts of animosity. Again. start with education in mind.
- For new hires, you have to baseline them. You have no earthly idea what they do or do not know. Start every new hire with at least a 45 minute online class. If possible have this tied into your AD new user creation process and on-boarding process. KnowBe4 can do this, simply add a user to a specific AD group and they get added to the correct new employee training on KnowBe4. If you are just starting a program, I strongly suggest *every* employee do a baseline 45 minute class.
- Every existing employee implement an every 6 month 15 minute refresher. If each time we run the 15 minute class we gain an additional 15% if employee knowledge that's a least a starting point. Build, build, build. Repeat, repeat, repeat. A year between training is simply too long a gap. Cybersecurity is a shared responsibility, and this is the employee's share.
- Once you've done a 6 month cycle or two, you can do a simulated phishing attack. Again, no blame, no publicizing the results (yes, I've seen this, yes it's really, really bad).
- Remember, it's no longer just phishing. Your education program needs to include vishing, smishing and all the other cool names for being attacked.
- Ensure your employee policies and handbooks cover what to do in the event they suspect that have been compromised. And that these are easy to locate. Time is off the essence when a possible compromise is happening. And that these policies align with what you are trying to achieve.
With your first simulated phishing campaign (hint, never offer free money in your campaigns, it could make you famous for all the wrong reasons) you should now have a series of hard facts that you can work on:
- How many users opened the email?
- How many users clicked the link?
- How may users reported that they think this is bad/a test/your all trying to trick them?
- How many users entered credentials?
- What your score is relative to others in your industry.
With this in hand you can now target remediation (do some users need to retake the 45 minute course? Do I need to add extra content?) or add in other tools to assist the users. Tools? Yes tools.
A lot of organizations have filters in between the users and their email. Happily rewriting links in email so as to be confusing to a human as possible but hopefully preventing the user from navigating to a malicious web site. Indeed one of the most common ways to spot a phishing email is to look at the target URL. Our additional layers of security have just negated some of the video training your users will do. Fantastic!!! The good news is that there are tools starting to percolate out that help decipher these seemingly incomprehensible URLs. KnowBe4 have add-in named Second Chance that for certain desktop email clients that will show the user the actual link they are clicking on. It turns this jibberish behind an email button:
Into this warning that decodes the link:
Now if someone could make this a universal plugin that also works with web based email, we'd have a winner. Still, it's a start and if you have KnowBe4 there's a good chance you don't know about Second Chance.
Another tool to empower users is VirusTotal. There are plugins for most browsers that will allow users to self-check worrisome URLs and/or files. IT may not always be available or accessible, the internet however is. Finally telling user about HaveIBeenPwned is seeing them use it is quite the sight to behold.
Password reuse
Beyond end-user training is end-user education. What they don't know because you didn't tell them can, and often will, hurt you. As I mentioned earlier, the online video how-to's are no panacea. Some don't even touch on password hygiene or reuse. From some truly shocking (not shocking) statistics on passwords, look no further than the Compaitech Password Statistics page. Some highlights (or more correctly low lights):
Google found that :
- 52% of users reuse a password some of the time.
- 13% use the *same* password for *all accounts*
- Only 35% use a different password for all accounts.
Also present in this page is maybe the most disheartening statistic (again, surprised, not surprised):
IT professional reuse password more than average users (50% vs 39%).
Yet again the IT professionals unerring belief that they are superhuman and immune from the perils that only mere mortals fall for strikes again. How the use of enterprise password managers such as ManageEngine's Password Manager Pro or Keeper Enterprise is not mandated in every IT department on the planet is beyond me. I'm often stunned by an organization's desire to keep passwords less than or equal to 8 characters (the Windows GPO default). Simply making them longer and requiring a special character can do wonders for password security. An oldie but goodie is this LifeHacker article on passwords. I'll sum it up with this table which outlines the estimated time to brute force a password based on adding on an upper-case and special character vs lower-case only:
Yeah, as as IT professional you'll want at least 12 characters for your own passwords, and at least 10 for your end-users. So how does on overcome the perils of password reuse, woeful complexity and overall crappy password hygiene? Multi-factor authentication or MFA. Or 2FA.
MFA is incredibly effective at prevent credential theft. A 2019 Microsoft study has it as high as 99.9% effective. Given that success rate you would expect almost every organization to have implemented it right? In Wrong. While I admit it can be complex and relatively expensive (much less so that being ransomwared FWIW) just over half of organizations in 2019 have implemented MFA (57%). In fact a 2021 report from the Fido Alliance indicates that 91% of MFA projects are to prevent credential theft.
MFA is reported to be as high as 99.9% effective in reducing credential theft.
So *where* to you do MFA? Well, everywhere, or not. The possible exception is when you are in a trusted location (read on-network, on-LAN). There is little use having MFA enabled in your corporate LAN when accessing Office365 and you already have 12 character strong passwords and SSO is enabled. All you do is piss your users off with little effect to your overall security posture. However when accessing *anything* from outside the LAN you'd want MFA. MFA to VPN. MFA for Office365. MFA for Azure App Proxy. If I'm coming from the outside to the inside (and even if inside is an externally hosted cloud service) you need to require MFA.
Now there are some select users who should be forced to use MFA even when inside the corporate LAN. You. The IT admin. The Domain Admin. The people with the keys to the kingdom. At every logon. At every screen lock. Every time. And your critical servers too. DMZ servers. Proxy servers. Domain controller. Every. Single. Time. How you'd do this is a little complex now that Microsoft foisted Windows Hello on the world (don't use Windows Hello). but would probably involve Cisco Duo, Okta or the like. Why?
IT professional reuse password more than average users (50% vs 39%).
Because you are part of the problem. Now you can be part of the solution.
I often hear MFA is expensive and difficult (I'll give you the latter point), but every Office365 license has the ability to do MFA. Everyone license. Now you'll need something like Azure P1 or P2 (or Duo, or Okta, or any of the other providers of enterprise SSO) to get some of the more useful features such as trusted locations (not requiring MFA for Office365 on the LAN), but it does have it and you can implement it. And you should because a 2019 article from TechRepublic citing a report from Cyren and Osterman Research states that a staggering 40% of enterprises experienced Office 365 credential theft. And if those stolen credentials happen to be the ones you use for AD (because SSO and DirSync) then a users AD credentials have just been compromised. And if said user is a domain admin level of user....yeah, now you can see how these attacks you read about happen. MFA FTW!
40% of enterprises have experienced Office365 credential theft.
Conclusion
Your end-user population can the difference between a ransomware meltdown and none event. Engage them, train them, educate them. After all cybersecurity is a team sport. Build a program, create an internal blog. Because even an incremental increase in knowledge is an increase. And you need all the help you can get.
Finally, roll out MFA. Yes it's difficult. Yes it can be somewhat costly, The the results in decreasing credential theft are simply astounding. Oh, and change your password policies to at least 10 characters with a requirement for a special character.
