3

Here's my problem, I have one sqlite memory database by using QSql. I have several threads each handling one different table of this common database. And I use Win API to make sure these threads working on different processor, like this:

SetThreadAffinityMask (hThread, processorMask);

When there's only thread handling one table, it takes 10 seconds and uses 25% of total CPU. But when there're 4 threads handling 4 different tables, it takes nearly 40 seconds and uses only 35% of total CPU. I think the reason is there's some kind of thread-safe sync in one database. But due to different thread reading or writing different table, thread-safe slows my program. How can I optimize it.

Update: The most possible reason is some kinds of lock inside of Qt or/and Sqlite 3 slows my program, so is it possible to shutdown or bypass these locks by pre-setting.

Update2: Here's an example. (Maybe a little long, sorry)

class MultiProcessorThread
{
public:
    virtual void run();
    bool start()
    {
        m_hThread = CreateThread (NULL, 0, MultiProcessorThread::ThreadFunc, this, CREATE_SUSPENDED, NULL);

        if (m_hThread != INVALID_HANDLE_VALUE)
        {
            RunningThreadCount++;
            m_ProcessorMask = 1 << ( (RunningThreadCount - 1) % ProcessorCount);
            SetThreadAffinityMask (m_hThread, m_ProcessorMask); // Make thread working on different processor
            ResumeThread (m_hThread);
            return true;
        }
        else
            return false;
    }
protected:
    static DWORD WINAPI ThreadFunc (LPVOID in);
    HANDLE m_hThread;
    DWORD_PTR m_ProcessorMask;
    static DWORD_PTR ProcessorCount;
    static DWORD_PTR RunningThreadCount;
    static DWORD_PTR GetNumCPUs();
};

DWORD_PTR MultiProcessorThread::ProcessorCount = GetNumCPUs();
DWORD_PTR MultiProcessorThread::RunningThreadCount = 0;
DWORD_PTR MultiProcessorThread::GetNumCPUs() // Get how many processors on this PC
{
    SYSTEM_INFO m_si = {0};
    GetSystemInfo (&m_si);
    return (DWORD_PTR) m_si.dwNumberOfProcessors;
}
DWORD WINAPI MultiProcessorThread::ThreadFunc (LPVOID in)
{
    static_cast<MultiProcessorThread*> (in)->run();
    return 0;
}

class Run : public MultiProcessorThread
{
public:
    void run()
    {
        int i = 0;
        QString add = "insert into %1 values(1)";
        add = add.arg (table);
        QString sel = "select a from %1 ";
        sel = sel.arg (table);
        QString del = "delete from %1 where a=1";
        del = del.arg (table);

        while (++i) // read and write database
        {
            query.exec (add);
            query.exec (sel);
            query.exec (del);
        }
    }
    QSqlQuery query;
    QString table;
};  

int main (int argc, char *argv[])
{
    QCoreApplication a (argc, argv);
    QSqlDatabase db = QSqlDatabase::addDatabase ("QSQLITE", "test"); 
    db.setDatabaseName (":memory:"); // All threads working on the same memory database.
    db.open();
    QSqlQuery q (db), q1 (db), q2 (db);
    q.exec ("create table A (a)");
    q1.exec ("create table B (a)");
    q2.exec ("create table C (a)"); // All threads working on different table.
    Run b[3];
    b[0].query = QSqlQuery (q);
    b[0].table = "A";
    b[1].query = QSqlQuery (q1);
    b[1].table = "B";
    b[2].query = QSqlQuery (q2);
    b[2].table = "C";
    b[0].start();
    b[1].start();
    b[2].start();
    return a.exec();
}
6
  • This really depends on how you're using sqlite3. Paste some examples of what you're doing with the database. Commented Jan 24, 2012 at 11:30
  • Before you proceed with optimizations take a look at this question: stackoverflow.com/questions/1680249/… Commented Jan 24, 2012 at 12:18
  • @Neox it do some help but my situation has some differences. first, in my program I use QSql not sqlite directly, so I cannot choose how to compile sqlite3 lib. second I don't need to sync between threads because they're running on different table. third I use memory database so I have to share the same connection. Commented Jan 24, 2012 at 12:30
  • @michael911009 SQLite supports multithreading, but write access to the same db file is still serialized. sqlite.org/faq.html#q5 In my opinion multithreading with SQLite is not worth it. Commented Jan 24, 2012 at 13:15
  • @Neox How about this? I create several memory temp databases, each thread use one db, after all kinds of insert action(it takes very long time due to calculating), I attach all the temp db's data to one single db. Commented Jan 24, 2012 at 13:44

3 Answers 3

1
+50

First of all, don't set affinity mask explicitly, windows will automatically allocate threads on the most idle cores. It's better to rely on OS to do thread distribution than your code in this case.

As far as I know, sqlite locks whole database while writing, that's why you don't get a performance boost you expected. Take a look at sqlite locking documentation http://www.sqlite.org/lockingv3.html

Sign up to request clarification or add additional context in comments.

Comments

0

Have you measured how much time the threads are spending on the CPU as compared to, say, disk I/O throughput?

This could have nothing to do with threading and locks. It could have everything to do with Amdahl's law.

2 Comments

there's not disk I/O at all, it's a pure calculation in memory database.
Ok, so it's not disks, but that's not the point. It could be any blocking resource which leads to an Amdahl effect.
0

The Qt docs are unambiguous about this. From http://doc.qt.nokia.com/4.7/threads-modules.html#threads-and-the-sql-module:

Threads and the SQL Module

A connection can only be used from within the thread that created it. Moving connections between threads or creating queries from a different thread is not supported.

In addition, the third party libraries used by the QSqlDrivers can impose further restrictions on using the SQL Module in a multithreaded program. Consult the manual of your database client for more information

There is no way to do what you want through the Qt API.

2 Comments

However, I can do it(use QSql btwn threads) since my code is runnable. I just don't know how to optimize it.
The point is, this is not supported. Your program might crash, your data might get corrupted, anything else could happen. In any case, conceptually speaking, trying to write simultaneously to different tables in the same database without locking doesn't make sense. Your tables are either related, so the locking is needed, or they're not and they don't need to be in the same database at all. Which is it?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.