I'm no C++-Expert, but in the last days I did comesome research around "smart" pointers, so maybe I can help you:
Shared pointers may help you in this situation, but they may be just a too easy way out of a situation where it is not too clear in the first place who the data really belongs to. Accessing a shared pointer is cheap, but copying it has a small penalty since there has to be some reference counting to be done in the background.
If you are programming in c++11, try redesign your classes (if possible) using unique_ptr with move semantics which come at no overhead.