Initial DAOS with Beta 2 testing looked like a winner from IBM. Well now that IBM have released 8.5 as gold I thought I'd scare the crap out of the STS worker bees and upgrade the production mail server and turn on DAOS for mail and mail archive files. For those just back from Mars, see the official IBM DAOS (Domino Attachment and Object Services) blurb. (they should have named it "Store" not "Services")

To quote another great tester (that is me for those wondering).....HOLY FRICKEN COW!!!

40% reduction in file size at the OS level!


Read the line again......

OK, so you got that. And that was with the 8.0.1 compression already enabled on most of the files. You may see north of 45% if you jump straight from R6, R7 or 8.0.0 Domino servers.

So what is the catch? Well there is one. Transaction logging is needed for this baby to work. So all of you that were (incorrectly) told by your previous services provider (you'll be using STS now right?) that you didn't need transaction logging turned on will need to revaluate that strategy and find separate volumes to store those on.

But other than that, you should be good to go. I have tried this on an 8.5 Client and 8.0.2CH1 Client and it was completely transparent to Notes. My guess is that R6 and R7 clients will work too as this is all server side magic but you will have to verify that for yourselves. Take a look at the html file below for the full breakdown. Oh, and do you mail.boxes too!
Darren Duke   |   January 6 2009 07:26:20 PM   |    8.5  daos  domino    |  
  |   Next Document   |   Previous Document

Discussion for this entry is now closed.

Comments (11)

Gravatar Image
1 - Sean Cull    http://www.seancull.co.uk    01/07/2009 7:50:56 AM

Hi Darren, how are you backing up your data with DAOS ?

tks, sean

Gravatar Image
2 - Darren Duke    http://blog.darrenduke.net    01/08/2009 6:48:16 AM

Per the IBM link above:

< snip >

1. If you are using deferred deletion with DAOS, set the interval to longer than the interval between your backups. For example, if you back up weekly, specify eight days for the setting "Defer deletion of DAOS objects n days" in the server document.

2. Back up NSF files on the server using a backup utility that is compatible with NSF files. The utility must be able to use the backup and recovery methods of the Lotus Domino C API Toolkit.

3. Back up the DAOSCAT.NSF and DAOS.CFG files. These files are located in the data directory.

4. Back up all NLO files in your DAOS repository. You can use any flat file backup utility of your choice (such as Tivoli® Storage Manager). If DAOS has created subdirectories, maintain the directory hierarchy in your backup.

5. After the first backup of the DAOS repository, perform incremental backups as desired of both NSF and NLO files.

6. (Optional) It is highly recommended that you archive any transaction logs so that changes that occurred after the last backup can be replayed for the most complete restoration of data.

< / snip >

We have a cluster server that is still 8.0.2 and that is what we backup until I verify that Backup Exec 12.5 can indeed backup and restore DAOS.

Note, NLO files are encrypted by the server they are stored on via the server ID file. This means you can only restore the NLO files to a server running the same ID file.

Gravatar Image
3 - vlad       01/08/2009 10:50:40 AM

Hi Darren

Does DAOS work in a clustering environment? Since the files are encrypted with the server id do you need to back up all servers in a cluster? If I want to copy over a database on a server that does not have DAOS, does it work? If I would like to change back a DAOS database to a non DAOS how easy is it?

thanks

vlad

Gravatar Image
4 - Darren Duke    http://blog.darrenduke.net    01/08/2009 11:07:04 AM

vlad - each member of the cluster will have DAOS enabled (or not) independently of other cluster members.

As each server had it's own DAOS storage folder (which is file level, and therefore not clustered) and as each server uses it's own ID file for encryption it can be said to be a per server setting. The non-DAOS cluster members don't know or care about this as to Domino, Notes, C-API and anything else it "looks" like the attachments are still in the Notes DBs.

As for turning it off, from the help it looks like it cannot really be turned off, but can be "disabled":

< snip >

Read Only - DAOS is currently disabled on this server. Because DAOS was previously enabled, file attachments exist in the DAOS repository, and are read when documents containing them are retrieved. New copies of any attachments, whether or not the originals are already stored in DAOS, are stored in documents instead of in the DAOS repository.

< /snip >

So it looks like when it is disabled, attachments are now left in the document but all documents prior to disablement are left in DAOS.

You could "revert" by creating a new server (with DAOS disabled) and replicating all the dbs across.

Note, I HAVE NOT TESTED ANY OF THIS. PROCEED AT YOUR OWN RISK.

Gravatar Image
5 - Darren Duke    http://blog.darrenduke.net    01/08/2009 11:09:27 AM

Sometimes when creating new servers (and this is not the "preferred" way) you can copy the NFS files across via a file copy. With DAOS enabled you CAN NO LONGER DO THIS.

You will have to use Domino replication to pull the NSF files to the new server (and this is the "preferred" way anyway).

Gravatar Image
6 - vlad       01/09/2009 10:31:20 AM

Hi Darren

It makes sense what you says... After re-reading how DAOS works it seems that:

-it is mostly designed for emails and occurs when an email is broadcasted... What would happen with the existing attachments (before you enable the DAOS)? Are they saved once or multiple times?

-would a regular application benefit from DAOS?

-personally I do not see any advantage from encrypting the file with the server id. Is it possible to choose to not use the encryption?

Any idea?

Thanks

vlad

Gravatar Image
7 - Darren Duke    http://blog.darrenduke.net    01/09/2009 11:48:45 AM

Lets see...

1) Prior to enabling DAOS the attachment would be stored once per mail file. After DAOS the attachment is saved only once. Note, that this also looks to be true for replies to the original email that have the exact same attachment but I can't vouch for this yet. Do the math and this is powerful stuff. 1.5MB attachment to 10 people = 15MB of attachment data without DAOS. Post DAOS the server only stores 1.5MB attachment. Now, (and if replies work like I think they do) someone does a "reply to all with history and attachments".......

2) Probably not. Unless you have the same attachment in multiple documents.

3) I don't think encryption can be turned off. As to the reason why, I guess it is so that admins (or users) can't go snooping for personal, private or secret data in the NLO files. I'd rather it be encrypted than not.

Gravatar Image
8 - vlad       01/09/2009 12:35:14 PM

browsing through notes ini settings at { http://www.ibm.com/developerworks/lotus/documentation/notes-ini/ I have noticed "DAOS_Encrypt_NLO" setting which "Specifies whether encryption is enabled or disabled for subsequently created .NLO files in the Domino Attachment and Object Service (DAOS) repository"... } I have noticed "DAOS_Encrypt_NLO" setting which "Specifies whether encryption is enabled or disabled for subsequently created .NLO files in the Domino Attachment and Object Service (DAOS) repository"...

It is really helpful for a clustered environment and not only (imagine that you would like to change the server's name). I would not worry about the admins. If a user email is encrypted, the attachment would be encrypted and protected when saved... still admins already have access to the databases and to the users' id as well.

vlad

Gravatar Image
9 - Darren Duke    http://blog.darrenduke.net    01/09/2009 12:46:37 PM

Obviously as the default is enabled you'll want to add this ini setting before you convert any databases.

Gravatar Image
10 - Darren Duke    http://blog.darrenduke.net    01/09/2009 12:50:09 PM

If the OS is compromised.....

At least with Domino even if the OS is compromised Domino and the email file "should" still be safe depending on the setup and OS. But unencrypted NLO files are at risk.

The big advantage I see from NOT enabling encryption is that you can now do (5) again.

Gravatar Image
11 - Eric Radloff       12/22/2011 2:15:06 PM

I recently observed that using replication as the "preferred" method of transfer .nsf's to a new server has some risks... if any documents are in a truncated status, they won't be copied to the new server and will be effectively lost (the same documents work fine on the old server, though, so even though they say they're "truncated", they're still useful and need to be kept). This caused a number of end user complaints during one of my server upgrades. To remedy it, I found there were 2 possible solutions:

(1) write an agent to compare the document counts of each nsf between the old server and the new one, and if the document counts don't match, compare all UNID's between the two and fire a "document.CopyAllItems" at each one that's missing, and sync the UNID value on the new doc in the destination db.

(2) perform a compact -c -daos off <nsfpath> and then perform an operating system copy, and then afterwards perform a compact -c -daos on <nsfpath> on the new server.

I use solution (1) more often because it's easily mass automated in a notes agent (Set it and forget it), however sometimes I have to use solution (2) because the # of UNID's in the nsf is too large for an agent to handle (I have the agent email me when that happens). I don't like solution 2 though because it requires manual interaction with the server console and the compact takes foooooooreeeeeeeverrrrrr.