1Erik Brooks  7/18/2010 9:35:53 AM  Yup

You're absolutely right. Don't forget that IBM added a setting (ND7 I think) that allowed the admin to prohibit the slow-on-the-fly FT search.

Normally if you don't have an FT index and a user performs a search you get the "unoptimized search." Which is slow, thrases disk i/o, and makes the server get flogged pretty heavy. But at least it works.

Now admins can turn this off, meaning there's zero ways to search mail.

I also don't believe there's been much in the way of improvements to the FT indexer in the past several years, probably due to the entire Workplace fiasco (i.e. "Why bother? It'll all be in DB2 soon.")

Getting FT indexes and view indexes out to separate volumes from the NSF data itself would do a world of good for many reasons.

  2Nathan T. Freeman  7/18/2010 11:48:11 AM  Skeptical

With the new managed cache mode in 8.5.2, I'm skeptical as to whether IBM will be motivated to make this change. Perhaps if we can find a reason why it's needed for the sake of LotusLive...

Just like the Widget Catalog, DCT, ID Vaults, Policies, SmartUpdate, Presence Awareness, Server-side Archiving and Clustering, IBM remains content to let the user experience be dictated by the implementer instead of the vendor. As a result, Notes shops not large enough to employ full-time professional admins and not progressive enough to find a high-quality services from a business partner are under constant siege. You're absolutely correct, Darren, that people will believe this statement about search from Google simply because it's been their own experience. In the same vein, they're likely to believe the lies about not being able to access Notes mail over the internet, because they have admins who don't understand NRPC encryption or the iNotes redirector.

As far as I can tell, IBM operates on the constant assumption that all their customers have business partner relationships to address implementation issues. One VP recently learned that this is not at all true, even in organizations where you'd expect it (like mid-sized pharmaceuticals.) Until the reality sets in that on-premises admins in organizations of less than 1000 are not motivated to make Notes a world-class experience themselves, we're not going to see these kinds of simple, basic changes that would make the vast majority of customers much happier. It's just not part of their reality.

I believe some robots pointed out that the consequence of leaving things in the hands of IT administrators is the continued attrition of the platform in mid-market shops.

  3Darren Duke  7/18/2010 12:00:16 PM  The most useful feature of Lotus Notes that most users can’t use - Full Text Indexing

@1, I agree, why not store view indexes outside the NSF....I think this maybe a bit more difficult to achieve though than just place a xxx.ft folder in a different location ;)

@2, as I was writing this post, I thought the exact same thing, that the new local replica controls coming in 8.5.2 would be "their" answer to this. I hope it is not because (a) local replica management has always been crap and 8.5.2 fixes that, and (b) this only fixes 1/3 of the issue. The other 2/3 being iNotes access and think client access.

As Nathan brought this up in @2, I agree with Notes access outside the firewall, simply open 1352, turn on port encryption and have at it. However, can anyone point me to an undisputed source in the support side that stipulates exactly the type and strength that Notes uses. Last I looked it was "believed" to be 64 bit RC4 { Link}.

  4Darren Duke  7/18/2010 12:12:56 PM  Now this is useful

It looks like since R6 we've had 128 bit.....time to test me thinks {Link }.

It would be nice if this info was in the trace command too.

  5Darren Duke  7/18/2010 12:45:07 PM  The most useful feature of Lotus Notes that most users can’t use - Full Text Indexing

See this new post for port encryption testing.

  6Nathan T. Freeman  7/18/2010 3:28:04 PM  Left out...

...DAOS. You have to elect to implement DAOS. Think about that. It's just insane. If you have 8.5+ Domino server, it should just send you daily emails if you have Mail users and DAOS isn't enabled.

  7Erik Brooks  7/18/2010 7:05:39 PM  Well....

@6 - Along those same lines: why do you have to add an INI parameter to allow creation of ODS51 databases?

Now that I'm thinking about it, for new installs both features frankly SHOULD be on. For DAOS, simply default to use the Notes data directory -- the overhead is virtually the same as with DAOS off.

For upgrades I'm actually OK with any "dot-oh" features requiring enablement. 8.5.0's DAOS implementation, while overall awesome, did have some fairly show-stopping corruption/missing-attachment issues. It wasn't until 8.5.1FP-something-or-other-CF-something-else that they had the last of those worked out.

Maybe that's the real answer: Due to the "newness" of these features IBM decides to let us battle-test them. Maybe they'll turn them on by default in future versions? They've done this sort of thing for years with memory management INI settings (include as optional in one version, commit as default in the next).

  8Nathan T. Freeman  7/18/2010 11:30:59 PM  I would think about buying that, if...

...any of the features dating back to the R6 era were NOW enabled by default. If transaction logs were automatic (and refused to install on non-dedicated platters); if server-side archiving just asked you every day "what's your aging cutoff for old emails?"; if policies automatically set up for all the default settings and drew the admins attention for controls; if clusters were smart enough to auto-mirror NSFs; if Widget Catalogs were automatically generated and replicated against Greenhouse, devWorks or OpenNTF -- if ANY of those things happened, then yeah, I'd accept the "dot-oh" argument. But they don't.

So no, I don't buy the battle-testing angle. It's just a rationalization.

  9Erik Brooks  7/19/2010 7:34:42 AM  Yeah...

Yeah, I agree. I was just grasping at straws hoping there was some sort of rationale behind it all.

Transaction logs in particular and SS archiving are great examples.

  10Nathan T. Freeman  7/19/2010 8:54:13 AM  The real reason this will never get fixed...

Rather than making a few small adjustments in storage location and adding better automation to Domino's full-text search, IBM will instead pursue search as a separate add-on. In fact, they probably already HAVE, but typical of their marketing, we haven't heard of the product because it's IBM Websphere Enterprise Search Aggregator or some other such nonsense.

At any rate, whatever it is, it probably has a 5-figure minimum purchase price, and requires 3 weeks of IBMGS consulting to set up. Once you set it up, there's probably a nightly batch process to run the indexes. And when you want to search your own email, you have to go to a web page, type in your search criteria, and submit it. Then the Aggregator server emails you a digest list with url links back to it's own presentation of your searched content (ie: not links to your own mail file in Notes)

The justification for doing this is so you can search the ECM archives of your email at the same time, which of course are no longer stored in Domino.

IBM may not ship this product yet, but I promise that there's some team working diligently on providing the worst possible user experience for searching your email, all in the name of "content management." And because it's a big ticket item, any attempts to improve native FT searching will be waved off as "non-strategic."

  11Paul MOoney  7/19/2010 10:41:45 AM  Thinking back a few years

I asked for this at "ask the developers" at Lotusphere 2008. Lots of applause. No progress.

  12Peter Smith  7/19/2010 4:53:14 PM  I think this is possible?

As I just posted on vowe.net ({ http://vowe.net/archives/011584.html), I've just done a quick test, but found no problem with using a directory link file to access the FT index, e.g. database.ft pointing to a directory containing the index files somewhere outside the server data directory. } I've just done a quick test, but found no problem with using a directory link file to access the FT index, e.g. database.ft pointing to a directory containing the index files somewhere outside the server data directory.

This allowed me to update the index successfully either manually or via a server command, and search multiple databases.

You can also create the dir link file, then build the index, and it gets built in the location specified.

Still more testing required, but we'll try to offer this as an option in our FT Search Manager product, along with DWA mail & archive searching (another shameless plug, I'm outdoing myself)

  13Darren Duke  7/19/2010 7:00:39 PM  I think this is possible?

I had actually tried this too, and it works just like the old stub file hack to move a DB out of the data folder. The problem with this approach is that you need access to the file system. As I see it, the customers who have FTI "disabled" are the larger ones (1,000's or 10,000's of users). They will never give you file system access and they should not have to.

Why must my life be full of hacks?

  14Peter Smith  7/19/2010 7:37:25 PM  Yes that’s true....

That's true, so maybe not great for the standard user, but a very nice option for our product - & thanks for the pointer.

I guess what might be possible is providing a server-based agent 'run on behalf of' for immediate manual FTI moving, or a server based agent for scheduled FTI moving, such that a script agent could delete the db index, add a dir link file to act as the stub, then rebuild the index.

If this works, we'll make a utility available for people to use as well.

  15Peter Smith  7/22/2010 2:25:05 PM  Not ideal

This method is not reliable enough to be used, sometimes it works fine, but at other times you get a 'index is in use' type error, so it's not worth pursuing.

I wonder if there is an OS way to do the same thing?

  16Peter Smith  8/9/2010 1:06:18 AM  Looking for testers

If the OS is Vista, Windows 7 or a version of unix, it seems to be possible to use symbolic links to move the Full Text Index. So we've written a utility that lets you do this, which if it proves acceptable enough, we'll give away and also include in our FT Search Manager product.

Basically you define your dbs, and for each db, the new index location (or the same for all), and if the db should be automatically 'processed' or not. The utility will move the index files in the FTGI folder to the location\server\filepath, then set up a symbolic link to the moved files. We move the ftgi folder because Notes seems to insist the dir.ft directory exists in the normal location.

If an index is rebuilt, it'll recognise this and move the files again. Updates occur as normal.

So right now we're looking for testers, is anyone keen?

  17Peter Smith  8/9/2010 4:33:30 AM  Looking for testers (2)

If so, plse email me using peter at ionetsoftware.com

  18Ed Reep  6/6/2011 11:00:50 AM  Can third-party solutions add this feature?

I wonder if third-party solutions like from www.avabiz.com can add those features. IBM, in the inevitable Notes 9, should implement this.