February 25 2009 Wednesday
DAOS my 2 cents and a possible future - and how to back it up
As part of my day job at STS we give weekly live demos of Domino 8.5. We've been doing these for over 2 years now, first with 8.0 and now with 8.5. On yesterdays demo I was asked how to backup DAOS enabled servers. I had posted about his a while back but figured I'd give it another shot and see if we can't stop the FUD some more. Additionally there are some possible future titbits that may or may not happen.
First things first. Transaction Logging should be enabled on all production servers. Period. The fact that DAOS requires it is a moot point. Also, before you roll out DAOS, test it. There is a practical limit of 40,000,000 NLO files per DAOS enabled server.
As Andrew Pollack points out, the easiest way to backup a DAOS enabled server is to replicate (or cluster) to another Domino server that is not DAOS enabled and then run backups from that server. Simple right? OK, so lets say that is not an option. Follow these simple guidelines and you should be good to go (in order!):
1. Use a supported backup system that has the ability to access the Domino backup and restore C-API. TSM, NetBackup and BackupExec with the appropriate agents are examples. If you know of others, feel free to comment. These will backup the transaction logs automatically and allow complete backup of NSF files while the server is running. If you don't have an agent then you can shut the Domino server down and ignore the order outlined here.
2. In the server doc, set the "Defer object deletion for" to 1 (one) day more than your full backup schedule. For example if you do full weekly backups, ie. every 7 (seven) days, set the defer to 7 + 1 = 8 days. This has some bearing on restores, which we will cover later but as a matter of practicality I want "free space on the server as soon as reasonably practicable" over "ease of restore". Feel free to pick your own poison and increase the defer if you need.
3. Backup the NSF files first before you get to the DAOS file system folder and NLO files. This way you can ensure every single attachment that is pointed to in the NSF files are backed up. If you do it the other way round you could get into a scenario where you restore a NSF file, but all the required NLO files didn't exist at the time of backup (ie, a new mail arrived in an NSF while the NLO repository is being backed up).
4. Backup the DAOSCAT.NSF and DAOS.CFG files. These files are located in the data directory. These files basically manage the NLO files and what exists.
5. Backup the DAOS file system folder. Be sure to maintain the folder hierarchy.
6. Make sure you backup the server ID file. NLO files are encrypted with this.
OK, so now you have a backup. As I like to say, "you are only as good as your last restore", so what do you do about that? Well, it is simple yet complicated! This is where that 50% in free disk space costs you, at least until the backup software vendors build it into their products (like point in time restores, yes Domino can do that).
1. Looking back at (2 above) you will see the prune frequency has some bearing on the restore. Restore the actual NSF file in question and run the command "tell daosmgr listnlo -o filename.txt". This text file will list the NLO files that are needed by the NSF files but appear to have been pruned (read deleted) from the DAOS file folder. These will need to be restored.
2. Once the NLO files in the text file have been restored you must run the dass resync command as soon as is practicable to rebuild DAOSCAT.
3. Open restored NSF file and test.
Based on some conversations at LS09 it appears IBM Lotus are really into this "savings" stuff. They are looking at adding DAOS to the replicator and cluster tasks so that if an attachment already exists on server B when server A replicates then the attachment is NOT sent over the wire. Imagine getting a 50-80% reduction in replication and/or clustering bandwidth!
Additionally, there was talk about the "NLO life cycle". What this means is that an attachment goes through two phases. Phase one is when it is first added to a database or sent to a mail file user or users. During this phase the attachment has a fair number of opens, but after about 7-10 days (phase 2) the number of opens of said attachment is dramatically reduced. What this could mean is that there are two NLO storage areas, a fast and local one for < 7-10 days, and the option for a slow one (SAN, NAS, etc) for any older.
My 8.5 beta 2 DAOS testing
My 8.5 gold results in the STS production environment
Domino Wiki - DAOS Backup and Restore
Paul Mooney vigorously defends DAOS
IBM DAOS Estimator Tool
Ed Brill's list of DAOS results
First things first. Transaction Logging should be enabled on all production servers. Period. The fact that DAOS requires it is a moot point. Also, before you roll out DAOS, test it. There is a practical limit of 40,000,000 NLO files per DAOS enabled server.
Backups
As Andrew Pollack points out, the easiest way to backup a DAOS enabled server is to replicate (or cluster) to another Domino server that is not DAOS enabled and then run backups from that server. Simple right? OK, so lets say that is not an option. Follow these simple guidelines and you should be good to go (in order!):
1. Use a supported backup system that has the ability to access the Domino backup and restore C-API. TSM, NetBackup and BackupExec with the appropriate agents are examples. If you know of others, feel free to comment. These will backup the transaction logs automatically and allow complete backup of NSF files while the server is running. If you don't have an agent then you can shut the Domino server down and ignore the order outlined here.
2. In the server doc, set the "Defer object deletion for" to 1 (one) day more than your full backup schedule. For example if you do full weekly backups, ie. every 7 (seven) days, set the defer to 7 + 1 = 8 days. This has some bearing on restores, which we will cover later but as a matter of practicality I want "free space on the server as soon as reasonably practicable" over "ease of restore". Feel free to pick your own poison and increase the defer if you need.
3. Backup the NSF files first before you get to the DAOS file system folder and NLO files. This way you can ensure every single attachment that is pointed to in the NSF files are backed up. If you do it the other way round you could get into a scenario where you restore a NSF file, but all the required NLO files didn't exist at the time of backup (ie, a new mail arrived in an NSF while the NLO repository is being backed up).
4. Backup the DAOSCAT.NSF and DAOS.CFG files. These files are located in the data directory. These files basically manage the NLO files and what exists.
5. Backup the DAOS file system folder. Be sure to maintain the folder hierarchy.
6. Make sure you backup the server ID file. NLO files are encrypted with this.
Restores
OK, so now you have a backup. As I like to say, "you are only as good as your last restore", so what do you do about that? Well, it is simple yet complicated! This is where that 50% in free disk space costs you, at least until the backup software vendors build it into their products (like point in time restores, yes Domino can do that).
1. Looking back at (2 above) you will see the prune frequency has some bearing on the restore. Restore the actual NSF file in question and run the command "tell daosmgr listnlo -o filename.txt". This text file will list the NLO files that are needed by the NSF files but appear to have been pruned (read deleted) from the DAOS file folder. These will need to be restored.
2. Once the NLO files in the text file have been restored you must run the dass resync command as soon as is practicable to rebuild DAOSCAT.
3. Open restored NSF file and test.
One Possible Future
Based on some conversations at LS09 it appears IBM Lotus are really into this "savings" stuff. They are looking at adding DAOS to the replicator and cluster tasks so that if an attachment already exists on server B when server A replicates then the attachment is NOT sent over the wire. Imagine getting a 50-80% reduction in replication and/or clustering bandwidth!
Additionally, there was talk about the "NLO life cycle". What this means is that an attachment goes through two phases. Phase one is when it is first added to a database or sent to a mail file user or users. During this phase the attachment has a fair number of opens, but after about 7-10 days (phase 2) the number of opens of said attachment is dramatically reduced. What this could mean is that there are two NLO storage areas, a fast and local one for < 7-10 days, and the option for a slow one (SAN, NAS, etc) for any older.
Other Links
My 8.5 beta 2 DAOS testing
My 8.5 gold results in the STS production environment
Domino Wiki - DAOS Backup and Restore
Paul Mooney vigorously defends DAOS
IBM DAOS Estimator Tool
Ed Brill's list of DAOS results
Ahh you beat me to the punch line. I was going to blog about my recommendations on backing up DAOS on Thursday.
I just wanted to clarify that to backup Domino with transactions logs you need Tivoli Data Protection for Domino (TDP) and TSM. TDP backups the Domino data and TSM backups the file (NLO,etc) data. TDP and TSM is not the cheapest product on the market but you get what you pay for.
I am not sure about other backup systems and I assume they would have some similar capability but TSM is a no brainer since when you run "tell daosmgr listnlo -o filename.txt" you can take the text file and in one command tell TSM to restore all of the files needed for the NSF you just restored.