writing multithread program for variable number of cores

You could use std::hardware_concurrency:

#include <iostream>
#include <thread>

int main() {

  std::cout << "Number of available cores on this system (hint): " 
            << std::thread::hardware_concurrency() << "\n";

}

Then launch the appropriate number of std::threads.

For the moment, let's ignore the GUI thread, since's it's always going to be the same.

For the worker threads, you're basically creating the same number of threads as cores, up to three, so you want something like:

HANDLE worker_threads[3];
int num_threads = min(num_cores, 3);

for (int i=0; i<num_threads; i++)
    worker_threads[i] = CreateThread(...);

Now, as to how to use those threads: the short answer is that I would not do it as you've described in your question. It makes little sense to take three steps that need to be executed in sequence, and break them up across threads. In fact, I'd avoid dedicating a thread to a particular type of task at all.

Instead, I'd keep the threads generic, so each thread can execute any of the three tasks. Use a queue of tasks to be carried out. The threads simply retrieve tasks from the queue and execute them. If that task results in another task to be carried out, that task gets put back in the queue with the rest.

Since you're doing this in C++, one obvious way to implement that is to create a base class for all tasks:

class Task {
public:
    virtual int operator()() = 0;
};

And then have your three different task types derive from that. Then you'll probably want to create a small wrapper class that covers up the fact that you're dealing with pointers to keep the code simple and straightforward.

As the question is not very specific, it is hard to guess what you need.

In general you could create workers which do a certain job and notify you (via callback or something else) about the result.

You then just start a new worker each time you need something to be done. There could be a maximum number of workers depending on the number of cores. It could be possible to somehow store the tasks you want to do in a queue and give them to the next free worker.

If you could somehow describe what you want to do (perhaps as a simplified example) it could be possible to help you better. Perhaps it is possible to show you how you could fit the worker-design to your purpose.