In a comment on Ed Brill's blog post about Google's dubious marketing I asked (comment 11) to allow full text indexes to be moved to different, non-Domino data storage. See, Google are saying you can't really search your mail file, or to quote Ed:
....they [Google] seem to have a total blind spot to the full-text search engine in Notes, asserting that sort by sender and browsing folders are the only ways to "search" in Notes


Anyone reading this blog will know that Google's assertion is as blatant a lie are there is. But, alas, most companies do not have full text indexing turned on for all, if any, of  their mail users.

Why not? Is it IBM that have the blindspot here?

After all, full text indexing a Notes database application immeasurably increased the usability, value and experience of Lotus Notes so why not turn it on? Disk space is the why. Storage consistently rates as one of the major concerns for IT in any survey you can find, so why double the storage needed  on my shiny new SAN just to enable searching? It is a good question and one that has never been addressed.

We can move attachments out to different storage.

We can move view rebuilds out to different storage.

Why can't we move full text indexes to different storage? Why do they have to reside in the same OS directory/folder as the mail file?

With Lotus Notes 8, IBM tackled head on the user interface issues with Lotus Notes. Now they need to tackle the usability of full text indexes and their location on the server. Yes you can use a local replica, indeed on this very blog I show you how to set that up, but IBM, you must tackle the issue at the lowest common denominator, the server-based mail file user. With the plethora of thin client technologies on the market (Citrix, Terminal Services and VMware View) more and more organizations have to use server based mail files due to "thin clients" having no storage. Add to that the brilliant experience that is iNotes, yet for most it is crippled beyond simple actions like sending and reading emails due to, you guessed it, being a server based mail file with no full text index.

Tripp Black elaborates my feelings on Ed's blog (comment 30):

The reason that Google can get away with the Full Text comment is, that as implemented, most companies cram as many users on a box until the experience is pretty dismal - No FT Indexes allowed. At a couple RTP companies (which one converted to MS live but still has $$$ going out to reinvent the apps), I told friends how to setup local replicas, turn on replication and then have FT Indexes. Suddenly, Notes was fast, you could find something and it didn't SUCK!. What a concept...... It took years of crippling Notes to get Notes to be evil to escape at any cost.


User satisfaction is king. And right now, due in some large part to this restriction, the majority of user are anything but satisfied when it comes to searching. And the shocker here is not that lack of functionality but being able to enable said functionality. IBM Lotus need to focus on the removal of every single roadblock that prevents full text indexing from being enabled.

As indicated above, there is already an IdeaJam for this, and it already has over 70 votes. 70 votes? That's a pretty unanimous cry for help. As Hynek Kobelka comments on the idea:
Very good idea ! And it should be so easy to accomplish :-)


To reduce storage on the server  IBM Lotus gave us DAOS and I would have to say this has been a leading factor in the massive and fast adoption of Domino 8.5.x in the last 18 months. If IBM wants to see another fast uptake on a Notes/Domino release why not add this? It is a win-win. The user wins with increased functionality and IT wins by being able to use cheap storage for FTI's.

What am I missing here?
Darren Duke   |   July 18 2010 05:28:04 AM   |    lotus notes    |  
  |   Next Document   |   Previous Document

Discussion for this entry is now closed.

Comments (18)

Gravatar Image
1 - Erik Brooks       07/18/2010 8:35:53 AM

You're absolutely right. Don't forget that IBM added a setting (ND7 I think) that allowed the admin to prohibit the slow-on-the-fly FT search.

Normally if you don't have an FT index and a user performs a search you get the "unoptimized search." Which is slow, thrases disk i/o, and makes the server get flogged pretty heavy. But at least it works.

Now admins can turn this off, meaning there's zero ways to search mail.

I also don't believe there's been much in the way of improvements to the FT indexer in the past several years, probably due to the entire Workplace fiasco (i.e. "Why bother? It'll all be in DB2 soon.")

Getting FT indexes and view indexes out to separate volumes from the NSF data itself would do a world of good for many reasons.

Gravatar Image
2 - Nathan T. Freeman    http://nathan.lotus911.com    07/18/2010 10:48:11 AM

With the new managed cache mode in 8.5.2, I'm skeptical as to whether IBM will be motivated to make this change. Perhaps if we can find a reason why it's needed for the sake of LotusLive...

Just like the Widget Catalog, DCT, ID Vaults, Policies, SmartUpdate, Presence Awareness, Server-side Archiving and Clustering, IBM remains content to let the user experience be dictated by the implementer instead of the vendor. As a result, Notes shops not large enough to employ full-time professional admins and not progressive enough to find a high-quality services from a business partner are under constant siege. You're absolutely correct, Darren, that people will believe this statement about search from Google simply because it's been their own experience. In the same vein, they're likely to believe the lies about not being able to access Notes mail over the internet, because they have admins who don't understand NRPC encryption or the iNotes redirector.

As far as I can tell, IBM operates on the constant assumption that all their customers have business partner relationships to address implementation issues. One VP recently learned that this is not at all true, even in organizations where you'd expect it (like mid-sized pharmaceuticals.) Until the reality sets in that on-premises admins in organizations of less than 1000 are not motivated to make Notes a world-class experience themselves, we're not going to see these kinds of simple, basic changes that would make the vast majority of customers much happier. It's just not part of their reality.

I believe some robots pointed out that the consequence of leaving things in the hands of IT administrators is the continued attrition of the platform in mid-market shops.

Gravatar Image
3 - Darren Duke    http://blog.darrenduke.net    07/18/2010 11:00:16 AM

@1, I agree, why not store view indexes outside the NSF....I think this maybe a bit more difficult to achieve though than just place a xxx.ft folder in a different location ;)

@2, as I was writing this post, I thought the exact same thing, that the new local replica controls coming in 8.5.2 would be "their" answer to this. I hope it is not because (a) local replica management has always been crap and 8.5.2 fixes that, and (b) this only fixes 1/3 of the issue. The other 2/3 being iNotes access and think client access.

As Nathan brought this up in @2, I agree with Notes access outside the firewall, simply open 1352, turn on port encryption and have at it. However, can anyone point me to an undisputed source in the support side that stipulates exactly the type and strength that Notes uses. Last I looked it was "believed" to be 64 bit RC4 { Link}.

Gravatar Image
4 - Darren Duke    http://blog.darrenduke.net    07/18/2010 11:12:56 AM

It looks like since R6 we've had 128 bit.....time to test me thinks {Link }.

It would be nice if this info was in the trace command too.

Gravatar Image
5 - Darren Duke    http://blog.darrenduke.net    07/18/2010 11:45:07 AM
Gravatar Image
6 - Nathan T. Freeman    http://nathan.lotus911.com    07/18/2010 2:28:04 PM

...DAOS. You have to elect to implement DAOS. Think about that. It's just insane. If you have 8.5+ Domino server, it should just send you daily emails if you have Mail users and DAOS isn't enabled.

Gravatar Image
7 - Erik Brooks       07/18/2010 6:05:39 PM

@6 - Along those same lines: why do you have to add an INI parameter to allow creation of ODS51 databases?

Now that I'm thinking about it, for new installs both features frankly SHOULD be on. For DAOS, simply default to use the Notes data directory -- the overhead is virtually the same as with DAOS off.

For upgrades I'm actually OK with any "dot-oh" features requiring enablement. 8.5.0's DAOS implementation, while overall awesome, did have some fairly show-stopping corruption/missing-attachment issues. It wasn't until 8.5.1FP-something-or-other-CF-something-else that they had the last of those worked out.

Maybe that's the real answer: Due to the "newness" of these features IBM decides to let us battle-test them. Maybe they'll turn them on by default in future versions? They've done this sort of thing for years with memory management INI settings (include as optional in one version, commit as default in the next).

Gravatar Image
8 - Nathan T. Freeman    http://nathan.lotus911.com    07/18/2010 10:30:59 PM

...any of the features dating back to the R6 era were NOW enabled by default. If transaction logs were automatic (and refused to install on non-dedicated platters); if server-side archiving just asked you every day "what's your aging cutoff for old emails?"; if policies automatically set up for all the default settings and drew the admins attention for controls; if clusters were smart enough to auto-mirror NSFs; if Widget Catalogs were automatically generated and replicated against Greenhouse, devWorks or OpenNTF -- if ANY of those things happened, then yeah, I'd accept the "dot-oh" argument. But they don't.

So no, I don't buy the battle-testing angle. It's just a rationalization.

Gravatar Image
9 - Erik Brooks       07/19/2010 6:34:42 AM

Yeah, I agree. I was just grasping at straws hoping there was some sort of rationale behind it all.

Transaction logs in particular and SS archiving are great examples.

Gravatar Image
10 - Nathan T. Freeman    http://nathan.lotus911.com    07/19/2010 7:54:13 AM

Rather than making a few small adjustments in storage location and adding better automation to Domino's full-text search, IBM will instead pursue search as a separate add-on. In fact, they probably already HAVE, but typical of their marketing, we haven't heard of the product because it's IBM Websphere Enterprise Search Aggregator or some other such nonsense.

At any rate, whatever it is, it probably has a 5-figure minimum purchase price, and requires 3 weeks of IBMGS consulting to set up. Once you set it up, there's probably a nightly batch process to run the indexes. And when you want to search your own email, you have to go to a web page, type in your search criteria, and submit it. Then the Aggregator server emails you a digest list with url links back to it's own presentation of your searched content (ie: not links to your own mail file in Notes)

The justification for doing this is so you can search the ECM archives of your email at the same time, which of course are no longer stored in Domino.

IBM may not ship this product yet, but I promise that there's some team working diligently on providing the worst possible user experience for searching your email, all in the name of "content management." And because it's a big ticket item, any attempts to improve native FT searching will be waved off as "non-strategic."

Gravatar Image
11 - Paul MOoney    http://www.pmooney.net    07/19/2010 9:41:45 AM

I asked for this at "ask the developers" at Lotusphere 2008. Lots of applause. No progress.

Gravatar Image
12 - Peter Smith    http://www.ionetsoftware.com/ftsearch    07/19/2010 3:53:14 PM

As I just posted on vowe.net ({ http://vowe.net/archives/011584.html), I've just done a quick test, but found no problem with using a directory link file to access the FT index, e.g. database.ft pointing to a directory containing the index files somewhere outside the server data directory. } I've just done a quick test, but found no problem with using a directory link file to access the FT index, e.g. database.ft pointing to a directory containing the index files somewhere outside the server data directory.

This allowed me to update the index successfully either manually or via a server command, and search multiple databases.

You can also create the dir link file, then build the index, and it gets built in the location specified.

Still more testing required, but we'll try to offer this as an option in our FT Search Manager product, along with DWA mail & archive searching (another shameless plug, I'm outdoing myself)

Gravatar Image
13 - Darren Duke    http://blog.darrenduke.net    07/19/2010 6:00:39 PM

I had actually tried this too, and it works just like the old stub file hack to move a DB out of the data folder. The problem with this approach is that you need access to the file system. As I see it, the customers who have FTI "disabled" are the larger ones (1,000's or 10,000's of users). They will never give you file system access and they should not have to.

Why must my life be full of hacks?

Gravatar Image
14 - Peter Smith    http://www.ionetsoftware.com/ftsearch    07/19/2010 6:37:25 PM

That's true, so maybe not great for the standard user, but a very nice option for our product - & thanks for the pointer.

I guess what might be possible is providing a server-based agent 'run on behalf of' for immediate manual FTI moving, or a server based agent for scheduled FTI moving, such that a script agent could delete the db index, add a dir link file to act as the stub, then rebuild the index.

If this works, we'll make a utility available for people to use as well.

Gravatar Image
15 - Peter Smith    http://www.ionetsoftware.com    07/22/2010 1:25:05 PM

This method is not reliable enough to be used, sometimes it works fine, but at other times you get a 'index is in use' type error, so it's not worth pursuing.

I wonder if there is an OS way to do the same thing?

Gravatar Image
16 - Peter Smith    http://www.ionetsoftware.com    08/09/2010 12:06:18 AM

If the OS is Vista, Windows 7 or a version of unix, it seems to be possible to use symbolic links to move the Full Text Index. So we've written a utility that lets you do this, which if it proves acceptable enough, we'll give away and also include in our FT Search Manager product.

Basically you define your dbs, and for each db, the new index location (or the same for all), and if the db should be automatically 'processed' or not. The utility will move the index files in the FTGI folder to the location\server\filepath, then set up a symbolic link to the moved files. We move the ftgi folder because Notes seems to insist the dir.ft directory exists in the normal location.

If an index is rebuilt, it'll recognise this and move the files again. Updates occur as normal.

So right now we're looking for testers, is anyone keen?

Gravatar Image
17 - Peter Smith    http://www.ionetsoftware.com    08/09/2010 3:33:30 AM

If so, plse email me using peter at ionetsoftware.com

Gravatar Image
18 - Ed Reep    http://www.avabiz.com    06/06/2011 10:00:50 AM

I wonder if third-party solutions like from www.avabiz.com can add those features. IBM, in the inevitable Notes 9, should implement this.