View Darren Duke's profile on LinkedIn
Admin

The very un-scientific look at transaction logging on Domino - part 2 - a real world server

Darren Duke  February 17 2010 11:20:24 AM
In my earlier post, A very un-scientific look at transaction logging in Domino I'd speculated about the Database.RM.Logger.IO Domino statistics.

As I'd mentioned right at the bottom of that post (and Richard also did in a comment) IO.Avg.Write.Time is probably more likely the statistic you need for production and not necessarily IO.Max.Write.Time. We'll look at both statistics for a server that has few users (<100) but huge amounts of email traffic. This is a production mail and apps server where we will move the transaction logs from the c: drive to a discrete disk. Let's see what we get....

The server in question is Domino 8.5.1 running on Windows 2003 32 bit server. It has a lot of traffic. For this exercise we will move the transaction logs from the c: drive which houses apps, OS and page file drive and the TX logs (the way the original server was configured) and move them to a discrete SATA disk. This new disk is a straight 500GB SATA2 7200 RPM drive . The original OS, apps and page file drive was a 3 disk RAID 5 array using SATA drives on a Dell PERC 5 controller. Domino data is situated on a separate 5 disk RAID 5 array also connected to the PERC.

Before the TX Log move, we have our stats. The server has been running about 6 days at this point.

Image:The very un-scientific look at transaction logging on Domino - part 2 - a real world server

Post TX Log move, after 6 days we have the following:
Image:The very un-scientific look at transaction logging on Domino - part 2 - a real world server

As you can see the difference is nothing short of staggering. The IO.Max.Write.Time is reduced by 99% and the IO.Avg.Write.Time went down by 77 orders of magnitude!

I can only imagine what a transaction log write that takes 8.2 seconds will do to Domino performance.If you have any doubt that the recommendation to move Domino transactions logs to discrete drive(s) is hog wash, then I beg to differ. If you have a very busy server and/or lots of users get the fastest drives possible (currently solid state drives or SSDs). There was some talk a while ago on Stephan Wissel's blog where SSD's we accused of being the latest panacea for poor Domino performance and that they are no faster in sequential writes than traditional (mechanical) drives. While they maybe a panacea (but you want a fast server, right?) they are hands down faster than traditional drives. If you are seeing high IO.Avg.Write.Time values then it could be well worth the investment. Regardless of where you sit on the SSD debate, put transaction logs on a discrete drive or array, period!

I''ll take a second to point out why I believe the IO.Number.Writes is so low on the second screen grab. This is not due to moving the TX Logs, but because this customer moved from Symantec Mail Security for Domino Anti-Spam to Lotus Protector for Mail Security. This removed some 4,000 messages per day from the Domino system (mail.box, etc) as Protector blocked them. I'll keep track of this over the next few weeks and when/if we get to a comparable IO.Number.Writes I will post again (quick math says 10 weeks....on a Windows server, hum). I have a feeling the IO.Avg.Write.Time and the IO.Max.Write.Time will not be adversely affected.
Comments

Gravatar Image  1norm  2/24/2010 10:01:59 AM  agreed

Hi Darren;

Interesting findings. We finally moved our T-logs from SAN to internal RAID-1 disk on a dedicated controller - at the same time we moved from Domino 7.0.3 to 8.5. I don't have the before/after stats to back it up like you do, but I can say that the server is significantly faster. As a simple example, running a re-build of clubusy.nsf for our 3000 mail users used to take 6 hours. It now takes under one hour. We are now taking steps to upgrade the ODS to the latest, where I hear I/O is even faster.

Do you have any comment on the notes.ini parm that drives the translog block size: CREATE_R85_LOG ??

Gravatar Image  2Lars Berntrop-Bos  2/24/2010 10:18:02 AM  transaction log wisdom for iSeries

Anyone data on how this transaltes to iSeries? I'm getting conflicting answers whether this is or isn't a good idea on iSeries.

Gravatar Image  3Darren Duke  2/24/2010 10:26:54 AM  The very un-scientific look at transaction logging on Domino - part 2 - a real world server

@1, I haven't done any testing on this new setting CREATE_R85_LOG. However, any server that doesn't have TX logging enabled when we take it to 8.5 is getting this setting. To change it in production "could" be a challenge. The setting is not retroactive so you have to remove all your logs and have Domino reibuild them. If you do that, you'll force a consistency check on all TX databases. Needless to say, if you have a lot of mail files......

@2, For the i5, it is indeed confusing. Most of the "official" documentation points to the i5 not needing a seperate ASP for the logs as it "just works". For DAOS on the i5 though, the recommendation is for the minimum DAOS attachment size to be 1MB.

Gravatar Image  4Sean Cull  2/24/2010 2:04:48 PM  Thanks

Darren, nice effective demonstration, tks, Sean

Gravatar Image  5Juergen  2/24/2010 3:46:53 PM  The very un-scientific look at transaction logging on Domino - part 2 - a real world server

Thanks for this interesting article.

If I unterstand this correctly you/your customer moved the TX logs to a single SATA drive.

Is this HD also connected to the PERC controller?

How critical is it, if this single HD fails?

Gravatar Image  6Darren Duke  2/24/2010 4:15:46 PM  @5, it is NOT on the PERC

For this client it is a single SATA drive that is not plugged into the PERC, and subsequently nor it is fault tolerant. It simply uses the PowerEdge's mobo SATA connector.

TX logging for this client was enabled for DAOS and is circular. Should the HD fail, we'll simply replace it and have Domino rebuild the TX logs. It is a price vs MTBF decision for this install. Price won ;)

If it was archived TX logs then that would be a totally different decision.

Gravatar Image  7John O’Laughlin  3/12/2010 1:14:28 PM  The very un-scientific look at transaction logging on Domino - part 2 - a real world server

Alright I'll play, the hightest IO.Max.Write.Time I have is 29948, if I read this correctly that's about 30 seconds. Ouch, vmware server with all disk io together.

Is this stat only available in 8.5 servers? I don't see it in lower versions of Domino.

Gravatar Image  8Darren Duke  3/12/2010 7:09:46 PM  The very un-scientific look at transaction logging on Domino - part 2 - a real world server

Not sure what version it came in, but yes, your max write time for the TX Log was almost 30 seconds! Having said that, the more pertinent value is the average. With VMware Server (as opposed to ESX(i) ) I would expect some very high values due to the muliple layers of abstraction needed by the guest OS.

Gravatar Image  9Adam Egressy  3/22/2010 9:05:15 AM  Moving TX log to a different drive

Hi Darren,

This article is very cool, I really liked it and decided to move my TX log to a different drive.

However, I can't find what's necessary to do?

1) Change TX file path in the server document

2) Shutdown the server

3) Move the files

4) Start the server

Is there anything else needed to to?

Thanks for your help!

Adam

Gravatar Image  10Darren Duke  3/22/2010 9:28:25 AM  Moving TX log to a different drive

You have it close:

1) Change TX file path in the server document

2) Shutdown Domino

3) COPY (not move) the files. This will stop a full consistency check from happening if something goes wrong. DO NOT RESTART DOMINO as this will, on occasion, use the old path.

4) Completely reboot the server at the OS level

5) At reboot Domino should restart

6) Once the Domino server is running, check the to see if the file modification date of the moved TX log has been updated to a time closer to now. You can also check the nlogctrl.lfh file in the logdir to see. Provided the mod date is now, the TX logs are moved. The old logs mod time should be the time you shut the server down.

7) Delete the old logs.

If the move doesn't work it is usually due to not rebooting the server (on Windows at least).

Gravatar Image  11Adam Egressy  3/22/2010 10:27:26 AM  Moving TX log to a different drive

Thanks for the quick answer, I'll try it tonight!

Cheers,

Adam

Gravatar Image  12Karen Hooper  5/25/2011 8:25:53 AM  Great article

Did you change the transaction style, is it still circular and 4GB in size?

Cheers,

Karen

    NOTE: - Email, name and subject are required, otherwise I'll have to approve the comment.....