Indexing Performance in Apache Lucene

Our current use case seems somewhat similar to yours: 1.6 billion rows, most fields are exact matches, periodic addition of files/rows, regular searching. Our initial indexing is not distributed or parallelized in any way, currently, and takes around 9 hours. I only offer that number to give you a very vague sense of what your indexing experience may be.

To try and answer your questions:

1. Our indexing time does not grow exponentially with the number of rows already indexed, though it does slow down very gradually. For us, perhaps 20% slower by the end, though it could also be specific to our data.

   If you are experiencing significant slow-down, I support femtoRgon's suggestion that you profile to see what's eating the time. Lucene has never been the slowest/weakest component in our system.

2. Yes, you can write to your index in parallel, and you may see improved throughput. Whether it helps or not depends on where your bottlenecks are, of course. Consider using Solr - it may ease your efforts here.

3. We use a mixture of StringField, LongField, and TextField. It seems unlikely that the type of field is causing your slowdown on its own.

These answers are all anecdotal, but perhaps they'll be of some use to you.

This page is now quite dated, but if you exhaust all your other options, may provide hints of which levers you can pull that might tweak performance: How to make indexing faster

Have you profiled to see what is actually causing your performance issues? You could find something unexpected is eating up all that time. When I profiled a similar performance issue I thought was caused by lucene, turned out the problem was mostly string concatenations.

As to whether you should use StringField or IntField (or TextField, or whatever), you should determine that based on what is in the field on how you are going to search it. If you might want to search the field as a range of numeric values, it should be an IntField, not a StringField. By the way, StringField indexes the entire value as a single term, and skips analysis, so this is also the wrong field for full text, for which you should use a TextField. Basically, using a StringField for everything would seem very much like a bad code smell to me, and could cause performance issues at index time, but I would definitely expect the much larger problems would appear when you start trying to search.

As far as "how will the search performance be with 5 billion values", that's far too vague a question to even attempt to answer. No idea. Try it and see.

Lucene has a hard limit of 2.14 billion documents per shard. So you have to divide your index into shards or have multiple indexes so that each index/shard does not contain documents more than 2.14 billion. I have indexes which are reaching that hard limit (I am using lucene directly..not elastic search/Solr) and haven't seen any significant slowdown in indexing speed. I am using Numeric fields (Int, Float and Long) as well as StringField and TextField types. Also check your RAM Buffer Size set for Index Writer. Increasing that might help slightly with the speed.