I have a 1M items store in List<Person> Which I'm serializing in order to insert to Redis. (2.8)
I divide work among 10 Tasks<> where each takes its own section ( List<> is thread safe for readonly ( It is safe to perform multiple read operations on a List)
Simplification :
example:
For ITEMS=100, THREADS=10 , each Task will capture its own PAGE and deal with the relevant range.
For exaple :
void Main()
{
var ITEMS=100;
var THREADS=10;
var PAGE=4;
List<int> lst = Enumerable.Range(0,ITEMS).ToList();
for (int i=0;i< ITEMS/THREADS ;i++)
{
lst[PAGE*(ITEMS/THREADS)+i].Dump();
}
}
PAGE=0will deal with :0,1,2,3,4,5,6,7,8,9PAGE=4will deal with :40,41,42,43,44,45,46,47,48,49
All ok.
Now back to SE.redis.
I wanted to implement this pattern and so I did : (with ITEMS=1,000,000)

My testing :
(Here is dbsize checking each second) :

As you can see , 1M records were added via 10 threads.
Now , I don't know if it's fast but , when I change ITEMS from 1M to 10M -- things get really slow and I get exception :
The exception is on the for loop.
Unhandled Exception: System.AggregateException: One or more errors occurred. ---
System.TimeoutException: Timeout performing SET urn:user>288257, inst: 1, queu e: 11, qu=0, qs=11, qc=0, wr=0/0, in=0/0 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message messa ge, ResultProcessor
1 processor, ServerEndPoint server) in c:\TeamCity\buildAgen t\work\58bc9a6df18a3782\StackExchange.Redis\StackExchange\Redis\ConnectionMultip lexer.cs:line 1722 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProces sor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\58bc9a6df 18a3782\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 79 ... ... Press any key to continue . . .
Question:
- Is my way of dividing work is the RIGHT way (fastest)
- How can I get things faster ( a sample code would be much appreciated)
- How can I resolve this exception?
Related info :
<gcAllowVeryLargeObjects enabled="true" /> Is present in App.config ( otherwise i'm getting outOfmemoryException ) , also - build for x64bit, I have 16GB , , ssd drive , i7 cpu).
ParallelEnumerable)? or the bit inside theforeloop? It really wouldn't amaze me if the top bit failed, but that is nothing to do with SE.Redis.IDatabaseinstance is fully thread-safe and is intended for heavily concurrent accessforloop ( after a while)