boost python threading segmentation fault

In short, there is a mutex around the CPython interpreter known as the Global Interpreter Lock (GIL). This mutex prevents parallel operations to be performed on Python objects. Thus, at any point in time, a max of one thread, the one that has acquired the GIL, is allowed to perform operations on Python objects. When multiple threads are present, invoking Python code whilst not holding the GIL results in undefined behavior.

C or C++ threads are sometimes referred to as alien threads in the Python documentation. The Python interpreter has no ability to control the alien thread. Therefore, alien threads are responsible for managing the GIL to permit concurrent or parallel execution with Python threads. With this in mind, lets examine the original code:

while (running) {
  std::cout << py::len(messages) << std::endl;           // Python
  messages.append(py::len(messages));                    // Python
  std::this_thread::sleep_for(std::chrono::seconds(1));  // No Python
}

As noted above, only two of the three lines in the thread body need to run whilst the thread owns the GIL. One common way to handle this is to use an RAII classes to help manage the GIL. For example, with the following gil_lock class, when a gil_lock object is created, the calling thread will acquire the GIL. When the gil_lock object is destructed, it releases the GIL.

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

The thread body can then use explicit scope to control the lifetime of the lock.

while (running) {
  // Acquire GIL while invoking Python code.
  {
    gil_lock lock;
    std::cout << py::len(messages) << std::endl;
    messages.append(py::len(messages));
  }
  // Release GIL, allowing other threads to run Python code while
  // this thread sleeps.
  std::this_thread::sleep_for(std::chrono::seconds(1));
}

Here is a complete example based on the original code that demonstrates the program working properly once the GIL is explicitly managed:

#include <thread>
#include <atomic>
#include <iostream>
#include <boost/python.hpp>

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

struct foo
{
  foo() : running(false) {}
  ~foo() { stop(); }

  void start()
  {
    namespace python = boost::python;
    running = true;
    thread = std::thread([this]
      {
        while (running)
        {
          {
            gil_lock lock; // Acquire GIL.
            std::cout << python::len(messages) << std::endl;
            messages.append(python::len(messages));
          } // Release GIL.
          std::this_thread::sleep_for(std::chrono::seconds(1));
        }
      });
  }

  void stop()
  {
    if (running)
    {
      running = false;
      thread.join();
    }
  }

  std::thread thread;
  boost::python::list messages;
  std::atomic<bool> running;
};

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::class_<foo, boost::noncopyable>("Foo", python::init<>())
    .def("start", &foo::start)
    .def("stop", &foo::stop)
    ;
}

Interactive usage:

>>> import example
>>> import time
>>> foo = example.Foo()
>>> foo.start()
>>> time.sleep(3)
0
1
2
>>> foo.stop()
>>>