0

My applikations runs in several threads. Up to four threads need to access my sqlite3 database. To implement this, I decided to run the database in a seperate thread. In this thread I have a queue (simply a vector of strings). This queue is filled through a pipe from the other four worker threads. After a timelimit or a max. number of elements in my queue I execute it.

Is this an acceptable solution? Or does anyone have a better one?

6
  • 1
    Having a message queue to communicate between threads is common, but why have a timeout or any kind of waiting? What if the requesting thread wants an answer as soon as possible? Also, how do you keep track of which message is from which thread, so you can send back the result? Commented Jul 10, 2013 at 7:32
  • 1
    If all requests to the database are inserts|deletes|updates - that's okay(if not to mention data loss on shutdown). But if worker threads will wait for result - I agree with @Joachim Pileborg, timeout is unneeded. Commented Jul 10, 2013 at 7:42
  • @JoachimPileborg I get by the worker threads every 2 or 3 seconds a value to insert. if i write those values immediately its a lot of I/O on the disk. If i loose a few values (1 min is my timeout) it is not so bad. But the other thing, how to identify the threads for a possible result, i did not think at this :/ Right now I cant think of a solution for this. Can cou recommend something or is there a common sulution for this? Commented Jul 10, 2013 at 7:53
  • 1
    Read this sqlite.org/threadsafe.html You can have 1 thread for batching writes to database as you already do, and 1 connection to the database per worker for selects only. Commented Jul 10, 2013 at 8:01
  • 1
    @GreenScape Thanks for the link. If I understand that correctly, i can use the SQLITE_OPEN_FULLMUTEX flag like this: sqlite3_open_v2(Path.c_str(), &DatabaseConnection, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_FULLMUTEX, NULL); and I dont need to check if I already insert something? For the other threads I would use the SQLITE_OPEN_READONLY and FULLÖMUTEX flag? Commented Jul 10, 2013 at 8:15

2 Answers 2

4

SQLite fully supports multithreading. By implementing your own management system you may improve performance and responsiveness of your application, but it is also probable that... you will not.

All you need to know about sqlite multithreading is in sqlite threadsafe documentation. I suggest you consider information in that document and make a prudent decision.

And if you decide to make your own solution, you can then use the information in the document above to disable unnecesary sqlite internal synchronization.

Side note: there are some useful information in this question/answers: How to use SQLite in a multi-threaded application?

Sign up to request clarification or add additional context in comments.

1 Comment

This answer is correct. I'll note though that many programmers I have worked with get confused by the names "multithreaded" mode and "serialized" mode. OP almost certainly wants serialized mode (which essentially turns the database connection into a monitor.)
1

A common way to handle message queues is to actually have two queues, one where the sender adds its message, and one for the reply. Then instead of using simple strings in the queue, you have a structure which contains the data (the string in your case) and the queue where to add the replies and maybe some kind of identifier to know what message the reply is for.

It's some extra memory overhead as the number of queues increases, but considering this scheme was used on computers in the mid 1980's with only 512 kb RAM (or even less) one can conclude that the overhead is not very big.


As for the implementation, you could create a simple class wrapping most of the functionality, even making it a templated class if you want different queues to be able to receive different data.

Maybe something similar to this:

template<typename Td>
class message_queue
{
public:
    static const message_queue<Td> none{};

    // Post a message to the queue, with an optional reply queue
    void post(const Td& data, message_queue<Td>& reply_queue = none);

    // Get the first message from the queue
    // The second member of the pair is an optional message queue
    // where the reply is posted, it's `none` if no reply needed
    std::pair<Td, message_queue<Td>> get();

    // Wait for a message to arrive in the queue
    void wait() const;

private:
    // The actual message queue
    std::queue<std::pair<Td, message_queue<Td>> queue;
}

The class above can be used something like this:

message_queue<std::string> sql_queue;

// ...

message_queue<std::string> reply_queue;
sql_queue.post("SELECT * FROM some_table", reply_queue);
reply_queue.wait();
std::string data = reply_queue.get().first;

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.