C++ Boost binary serialization of std::map containing pointers constructed from Boost object_pool

Question

My application has a class "MyClass". It's objects are being constructed from the Boost Object_pool.

I need to serialized/de serialize a std::map containing these objects as value via Boost Binary Serialization.

For Serialization -

I take a pointer from the pool, do some operations, insert it in the std::map and serialize it via Boost binary serialization.

For De-serialization -

I fetch that serialized buffer and de-serialize it with Boost binary serialization. De-serialization happens successfully, but during the process new memory is being allocated for the pointers by Boost Serialization mechanism which is not constructed from the Object Pool.

Hence, I won't be able to reuse this memory allocated by the boost serialization mechanism as it cannot be given back to the object pool because it was not constructed from the pool.

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/binary_object.hpp>
#include <boost/pool/object_pool.hpp>
#include <boost/serialization/map.hpp>
#include <map>
#include <iostream>
#include <sstream>
#include <string>
#include <functional>
#include <stdint.h>

class MyClass
{
  public :
   friend class boost::serialization::access;
   MyClass():data(0)
   {
     std::cout << std::endl << "MyClass()" << std::endl ;
   }

   MyClass( uint32_t val):data(val)
   {
     std::cout << std::endl << "Parameterized MyClass()" << std::endl ;
   }

    template<class Archive>
    void serialize(Archive & ar, const unsigned int version)
    {
      ar & data;
    }

   ~MyClass(){}

    friend std::ostream &operator<<( std::ostream &output, const MyClass &D )
    {
      output << "Data : " << D.print() ;
      return output;
    }

   void print()
   {
      std::cout << std::endl << "Data : " << data << std::endl ;
   }


   private :
   uint32_t data ;
};

int main()
{ 
  try
  {
    typedef std::map<int, MyClass *> ObjectMap ;

    ObjectMap map;

     boost::object_pool<MyClass> pool ;

     map[1] = pool.construct(6) ;
     map[2] = pool.construct(7) ;
     map[3] = pool.construct(8) ;
     map[4] = pool.construct(9) ;

     // Serialization
     std::stringbuf strbuf;
      boost::archive::binary_oarchive oa( strbuf ) ;
      oa << map;


     // Deserialzation
      ObjectMap mapRoundTrip;

      boost::archive::binary_iarchive ia( strbuf ) ;
      ia >> mapRoundTrip ;
    }
    catch ( boost::archive::archive_exception &e )
    {
      std::cout << std::endl << e.what() << std::endl ;
    }
 }

My requirement is to populate the map, during deserialzation, with pointers fetched from the object_pool.

Yup, any advice on the question? Trying - load_construct_data. No progress so far. — Jerry
– Jerry, Commented May 24, 2018 at 10:14
yes, it was a good exercise. I was unfamiliar with boost::pool, boost::serialization, std::map, the use of parameter packing, etc., so I've learned a lot. — JHBonarius
– JHBonarius, Commented May 24, 2018 at 12:28
By the way, why are you using std::map? You're keys are sequential integers. Could you not just use a std::vector? — JHBonarius
– JHBonarius, Commented May 24, 2018 at 12:41
It's a sample program that is why the keys are simple sequential integers. Actual code/logic is far more complex and keys are not sequential. — Jerry
– Jerry, Commented May 24, 2018 at 12:57

Community · Accepted Answer · 2020-06-20 09:12:55Z

This is a limitation of the way Boost serialization is implemented. Instead of just copying the pointer address, it de-references pointers and copies the whole object. This is done for all STL containers. When de-serializing, a new object is created, using standard allocator.

There are two ways to circumvent this: by building custom map class or by using the pool_allocator.

Using a wrapper for `std::map`

You can circumvent this by not using the STL containers. Write your own map (wrapper). E.g.

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/binary_object.hpp>
#include <boost/pool/object_pool.hpp>
#include <map>
#include <iostream>
#include <sstream>

class MyClass
{
public:
    friend class boost::serialization::access;
    MyClass() { std::cout << "MyClass()\n"; }
    MyClass(int val) :data(val) { std::cout << "MyClass(" << val << ")\n";  }

    template<class Archive>
    void serialize(Archive & ar, const unsigned int version) { ar & data; }

    void print() { std::cout << "Data : " << data << "\n"; }
private:
    int data;
};

template<class Key, class T>
class MyMap
{
public:
    MyMap(boost::object_pool<T> &pool) : mr_pool(pool) {}

    ~MyMap()
    {
        for (auto& kv : m_map)
        {
            if (kv.second != nullptr) mr_pool.destroy(kv.second);
            kv.second = nullptr;
        }
    }

    typename std::map<Key, T*>::iterator begin() { return m_map.begin(); }
    typename std::map<Key, T*>::iterator end() { return m_map.end(); }

    template<class ... Types>
    void construct(const Key& key, Types ... args)
    {
        m_map[key] = mr_pool.construct(args...);
    }

    template<class Archive>
    void save(Archive & ar, const unsigned int version) const
    {
        ar << m_map.size();
        for (auto& kv : m_map)
        {
            ar << kv.first;
            ar << boost::serialization::binary_object(kv.second, sizeof(T));
        }
    }

    template<class Archive>
    void load(Archive & ar, const unsigned int version)
    {
        size_t size;
        ar >> size;
        for (size_t i = 0; i<size; i++)
        {
            Key key;
            ar >> key;
            T* prt = mr_pool.construct();
            ar >> boost::serialization::make_binary_object(prt, sizeof(T));
            m_map[key] = prt;
        }
    }

    template<class Archive>
    void serialize(Archive & ar, const unsigned int file_version)
    {
        boost::serialization::split_member(ar, *this, file_version);
    }
private:
    boost::object_pool<T>& mr_pool;
    std::map<Key, T*> m_map;
};

int main()
{
    try
    {
        using ObjectMap = MyMap<int, MyClass>;

        boost::object_pool<MyClass> pool;
        ObjectMap map(pool);

        map.construct(1, 6);
        map.construct(2, 7);
        map.construct(3, 8);
        map.construct(4, 9);

        // Serialization
        std::stringbuf strbuf;
        boost::archive::binary_oarchive oa(strbuf);
        oa << map;

        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", ";
            kv.second->print();
        }
        std::cout << "pre destory\n";
        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", data addr: " << kv.second << "\n";
        }

        for (auto& kv : map) {
            if (kv.second != nullptr) pool.destroy(kv.second);
            kv.second = nullptr;
        }

        std::cout << "post destroy\n";
        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", data addr: "  << kv.second << "\n";
        }

        MyClass* temp = pool.construct(10); // to create memory offset
        // Deserialzation
        ObjectMap mapRoundTrip(pool);

        boost::archive::binary_iarchive ia(strbuf);
        ia >> mapRoundTrip;

        for (auto& kv : mapRoundTrip) {
            std::cout << "mapRoundTrip: " << kv.first << ", data addr: " << kv.second << "\n";
        }
        for (auto& kv : mapRoundTrip) {
            std::cout << "mapRoundTrip: " << kv.first << ", ";
            kv.second->print();
        }
        
        pool.destroy(temp);
        temp = nullptr;
    }
    catch (boost::archive::archive_exception &e)
    {
        std::cout << std::endl << e.what() << std::endl;
    }
    return 0;
}

output:

MyClass(6)
MyClass(7)
MyClass(8)
MyClass(9)
map: 1, Data : 6
map: 2, Data : 7
map: 3, Data : 8
map: 4, Data : 9
pre destory
map: 1, data addr: 0x24ad720
map: 2, data addr: 0x24ad728
map: 3, data addr: 0x24ad730
map: 4, data addr: 0x24ad738
post destroy
map: 1, data addr: 0
map: 2, data addr: 0
map: 3, data addr: 0
map: 4, data addr: 0
MyClass(10)
MyClass()
MyClass()
MyClass()
MyClass()
mapRoundTrip: 1, data addr: 0x24ad728
mapRoundTrip: 2, data addr: 0x24ad730
mapRoundTrip: 3, data addr: 0x24ad738
mapRoundTrip: 4, data addr: 0x24ad740
mapRoundTrip: 1, Data : 6
mapRoundTrip: 2, Data : 7
mapRoundTrip: 3, Data : 8
mapRoundTrip: 4, Data : 9

DEMO

Using `pool_allocator`

Alternatively you can circumvent this by not using the standard allocator, but instead use the pool allocator. E.g.

#include <boost/archive/binary_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/binary_object.hpp>
#include <boost/pool/object_pool.hpp>
#include <boost/pool/pool_alloc.hpp>
#include <boost/serialization/map.hpp>
#include <map>
#include <iostream>
#include <sstream>

class MyClass
{
public:
    friend class boost::serialization::access;
    MyClass() { std::cout << "MyClass empty construct\n"; }
    MyClass(MyClass const& src) :data(src.data) { std::cout << "MyClass copy construct\n"; }
    void swap(MyClass& src) noexcept { std::swap(data, src.data); }
    MyClass(MyClass&& src) :MyClass() { src.swap(*this); std::cout << "MyClass move construct\n"; }
    
    MyClass(int val) :data(val) { std::cout << "MyClass data construct (" << val << ")\n"; }

    MyClass& operator=(MyClass src) { src.swap(*this); return *this; }

    template<class Archive>
    void serialize(Archive & ar, const unsigned int version) { ar & data; }

    void print() { std::cout << "Data : " << data << "\n"; }
private:
    int data;
};

int main()
{
    using ObjectMap = std::map<int, MyClass, std::less<int>, boost::pool_allocator<MyClass>>;
    using Pool = boost::singleton_pool<boost::pool_allocator_tag, sizeof(ObjectMap::value_type)>;

    try
    {
        ObjectMap map;

        map[1] = MyClass(6);
        map[2] = MyClass(7);
        map[3] = MyClass(8);
        map[4] = MyClass(9);

        // Serialization
        std::stringbuf strbuf;
        boost::archive::binary_oarchive oa(strbuf);
        oa << map;

        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", ";
            kv.second.print();
        }
        std::cout << "pre destory\n";
        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", data addr: " << &kv.second << "\n";
        }
        
        map.clear();
        Pool::purge_memory();

        map[5] = MyClass(10);

        std::cout << "post destroy and reassign\n";
        for (auto& kv : map) {
            std::cout << "map: " << kv.first << ", data addr: " << &kv.second << "\n";
        }

        // Deserialzation
        ObjectMap mapRoundTrip;

        boost::archive::binary_iarchive ia(strbuf);
        ia >> mapRoundTrip;

        for (auto& kv : mapRoundTrip) {
            std::cout << "mapRoundTrip: " << kv.first << ", data addr: " << &kv.second << "\n";
        }
        for (auto& kv : mapRoundTrip) {
            std::cout << "mapRoundTrip: " << kv.first << ", ";
            kv.second.print();
        }

        mapRoundTrip.clear();
    }
    catch (boost::archive::archive_exception &e)
    {
        std::cout << std::endl << e.what() << std::endl;
    }
    Pool::purge_memory();

    return 0;
}

output:

MyClass data construct (6)
MyClass empty construct
MyClass data construct (7)
MyClass empty construct
MyClass data construct (8)
MyClass empty construct
MyClass data construct (9)
MyClass empty construct
map: 1, Data : 6
map: 2, Data : 7
map: 3, Data : 8
map: 4, Data : 9
pre destory
map: 1, data addr: 0x118e604
map: 2, data addr: 0x118e62c
map: 3, data addr: 0x118e654
map: 4, data addr: 0x118e67c
MyClass data construct (10)
MyClass empty construct
post destroy and reassign
map: 5, data addr: 0x118e604
MyClass empty construct
MyClass empty construct
MyClass move construct
MyClass empty construct
MyClass empty construct
MyClass move construct
MyClass empty construct
MyClass empty construct
MyClass move construct
MyClass empty construct
MyClass empty construct
MyClass move construct
mapRoundTrip: 1, data addr: 0x118e62c
mapRoundTrip: 2, data addr: 0x118e654
mapRoundTrip: 3, data addr: 0x118e67c
mapRoundTrip: 4, data addr: 0x118e6a4
mapRoundTrip: 1, Data : 6
mapRoundTrip: 2, Data : 7
mapRoundTrip: 3, Data : 8
mapRoundTrip: 4, Data : 9

You can see how the memory pool is being reused by checking the addresses.

DEMO

p.s. this can probably be cleaned up. I'm not the best C++ programmer.
@sehe because I did not know that was a possibility. Thank you for showing the way. I now know something new :)
Had a thought of using pool allocator, but cannot use it. For simple cases, it works perfectly as it is shown in the solution. In my situation, memory of entire application needs to be in control of the logic/algorithm and not the map. The pointer to object needs to execute tons of instruction and if it passes some specific criteria then only it will go into the map other wise it will go back to the pool. And there is more than one map in which this pointer needs to be inserted, and multiple maps won't be able to share same memory pool if we use pool_allocators for map.
@Jerry doe you want me to roll back to a previous version, where I have the std::map-wrapper? I that worth something to you?

Collectives™ on Stack Overflow

C++ Boost binary serialization of std::map containing pointers constructed from Boost object_pool

1 Answer 1

Using a wrapper for `std::map`

Using `pool_allocator`

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Using a wrapper for std::map

Using pool_allocator

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

Using a wrapper for `std::map`

Using `pool_allocator`