Calling Python function with parameters from C++ project (Visual Studio)

Question

I need to call a pipeline realized as a Python (3.6) function from my C++ project under the Windows platform. Function “function_name” from file "experiment_test.py" takes text string as input parameter and return another text string as the result. I try the code below but it doesn’t work properly – python functions from libraries shutil, codecs, makedirs, etc doesn’t work.

C++ code (reduced):

std::string Text,Result;
PyObject *pName, *pModule, *pDict, *pFunc, *pArgs, *pValue;
Py_Initialize();

pName = PyUnicode_FromString("experiment_test");
pModule = PyImport_Import(pName);    
pDict = PyModule_GetDict(pModule);

pFunc = PyDict_GetItemString(pDict, "function_name");

pArgs = PyTuple_New(1);
pValue = PyUnicode_FromString(Text.c_str());
PyTuple_SetItem(pArgs, 0, pValue);

if (PyCallable_Check(pFunc))
{
    pValue = PyObject_CallObject(pFunc, pArgs);
    if (pValue != NULL)
    {
        Result = PyUnicode_AsUTF8(pValue);    
        Py_DECREF(pValue);
    }
    else return false;    
}
// ...

Py_Finalize();

Python code (reduced):

#!/usr/local/bin/python3
import shutil
import codecs
from os import makedirs
from os import path
from os import unlink
from subprocess import call

def function_name():

    name = 'working_files/current_text'

    if not path.exists('working_files'):
        makedirs('working_files')
    if path.exists('result.txt'):
        unlink('result.txt')
    with codecs.open(name + '.txt', 'w', encoding='utf-8') as f:
        f.write(text)
    # ...
    return result

So no new files will be generated by Python. I tried to import Python modules in C++ by calling PyRun_SimpleString("import shutil"); etc after Py_Initialize(); but it doesn’t help.

What do I do wrong?

What exactly doesn't work (error, traceback, etc)? Did you try running the Python module directly (without adding the C++ layer)? Note that your code is syntactically incorrect (indentation-wise). — CristiFati
– CristiFati, Commented Dec 22, 2017 at 14:19
Python functions from shutil, codecs, makedirs doesn't work when “function_name” is called from C++. If I call Python module from the command line, it works correctly. No errors occur - C++ receive the result which is equal to input. Python function should write input string into the text file, process this file and then read the result and return it back. Indentation in my code is correct, unfortunately I did a mistake when I wrote it here, thank you. — Pavel
– Pavel, Commented Dec 23, 2017 at 15:35

CristiFati · Accepted Answer · 2023-05-05 17:51:11Z

7

+200

I tried replicating the problem with the given intel, but it was impossible, so I created a small example (as close as possible to what's described in the question) - Also referred to as [SO]: How to create a Minimal, Reproducible Example (reprex (mcve)) (that should be included in the question BTW)

So, the problem that I'm illustrating here, is:

C++
- Load the Python engine
- Load a Python module
- From that module, load a function which:
  - Receives a (string) argument representing a file name
  - Reads the file contents (text) and returns it
  - In case of error, simply returns the file name
- Call that function
- Get the function call result

I am using (on Win 10 x64 (10.0.16299.125)):

Python 3.5.4 x64
VStudio 2015 Community Edition

The structure consists of:

VStudio project / solution
- Source file (main00.cpp (renamed it from main.cpp, but didn't feel like doing all the screenshots (containing it) all over again))
Python module (experiment_test.py)
Test file (test_file.txt)

main00.cpp:

#include <iostream>
#include <string>

#if defined(_DEBUG)
#  undef _DEBUG
#  define _DEBUG_UNDEFINED
#endif
#include <Python.h>
#if defined(_DEBUG_UNDEFINED)
#  define _DEBUG
#  undef _DEBUG_UNDEFINED
#endif

#define MOD_NAME "experiment_test"
#define FUNC_NAME "function_name"
#define TEST_FILE_NAME "..\\test_dir\\test_file.txt"

using std::cout;
using std::cin;
using std::endl;
using std::string;


int cleanup(const string &text = string(), int exitCode = 1) {
    Py_Finalize();
    if (!text.empty())
        cout << text << endl;
    cout << "Press ENTER to return...\n";
    cin.get();
    return exitCode;
}


int main() {
    string fName = TEST_FILE_NAME, result;
    PyObject *pName = NULL, *pModule = NULL, *pDict = NULL, *pFunc = NULL, *pArgs = NULL, *pValue = NULL, *pResult = NULL;
    Py_Initialize();
    pName = PyUnicode_FromString(MOD_NAME);
    if (pName == NULL) {
        return cleanup("PyUnicode_FromString returned NULL");
    }
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (pModule == NULL) {
        return cleanup(string("NULL module: '") + MOD_NAME + "'");
    }
    pDict = PyModule_GetDict(pModule);
    if (pDict == NULL) {
        return cleanup("NULL module dict");
    }
    pFunc = PyDict_GetItemString(pDict, FUNC_NAME);
    if (pFunc == NULL) {
        return cleanup(string("module '") + MOD_NAME + "' doesn't export func '" + FUNC_NAME + "'");
    }
    pArgs = PyTuple_New(1);
    if (pArgs == NULL) {
        return cleanup("NULL tuple returned");
    }
    pValue = PyUnicode_FromString(fName.c_str());
    if (pValue == NULL) {
        Py_DECREF(pArgs);
        return cleanup("PyUnicode_FromString(2) returned NULL");
    }
    int setItemResult = PyTuple_SetItem(pArgs, 0, pValue);
    if (setItemResult) {
        Py_DECREF(pValue);
        Py_DECREF(pArgs);
        return cleanup("PyTuple_SetItem returned " + setItemResult);
    }
    pResult = PyObject_CallObject(pFunc, pArgs);
    Py_DECREF(pArgs);
    Py_DECREF(pValue);
    if (pResult == NULL) {
        return cleanup("PyObject_CallObject returned NULL");
    } else {
        int len = ((PyASCIIObject *)(pResult))->length;
        char *res = PyUnicode_AsUTF8(pResult);
        Py_DECREF(pResult);
        if (res == NULL) {
            return cleanup("PyUnicode_AsUTF8 returned NULL");
        } else {
            cout << string("C(++) - Python call: ") << MOD_NAME << "." << FUNC_NAME << "('" << fName << "') returned '" << res << "' (len: " << len << ")" << endl;
        }
    }
    return cleanup("OK", 0);
}

Notes:

The _DEBUG / _DEBUG_UNDEFINED stuff at the beginning - a (lame) workaround (gainarie) to link against Release Python lib (python35.lib) when building in Debug mode (as opposed to python35_d.lib) - read below
As I said, tried to simplify the code (got rid of the PyCallable_Check test)
It's easily noticeable that the code is written in C style, although it uses the C++ compiler
Since Python API ([Python.Docs]: Embedding Python in Another Application) (both extending / embedding) uses pointers, make sure to test for NULLs, otherwise there's a high chance getting SegFault (Access Violation)
Added the [Python.Docs]: Reference Counting - void Py_DECREF(PyObject *o) statements to avoid memory leaks
Build (compile / link) / Run options (obviously, you got past these, since you were able to run your program, but I'm going to list them anyway - for sure there are some shortcuts here, when dealing with more than one such project):
- Check [SO]: LNK2005 Error in CLR Windows Form (@CristiFati's answer) for details building a Win PE
- To speed-up process when building projects that depend on Python, I created a VStudio User Macro (called e.g. Python35Dir - like in the image below, pointing to my Python 3.5 installation dir)
Notes:
- The path ("c:\Install\x64\Python\Python\3.5") points to the installation downloaded from the official site
- Obviously, for 032bit, the path must be set accordingly (to 32bit Python)
- This path contains (as expected) a Release version, and this is fine as long as I don't need to get into Python code (and as long as I don't mess around with memory - as (when building my app in Debug mode) I have 2 C runtimes in my .exe - check the links below to see what happens when tampering with MSVC runtimes (UCRTs)):
  - [SO]: When using fstream in a library I get linker errors in the executable (@CristiFati's answer)
  - [SO]: Errors when linking to protobuf 3 on MS Visual C (@CristiFati's answer)
  - For exceptional cases, I have built Python in Debug mode and got the binaries, but that's not my 1^st choice, since it requires settings (paths) changes
- Compile:
  
  Let VStudio know about the Python include files location:
- Link:
  
  Let VStudio know about the Python lib files location (if only pythonxx*.lib (PYTHONCORE) is required, nothing extra needed, since PYTHONCORE is included by default by Python code; otherwise, all the rest should be specified in the [MS.Learn]: .Lib Files as Linker Input:
- Run / Debug - let:
  - VStudio know where Python runtime python35.dll (PYTHONCORE) is located (%PATH%)
  - Loaded Python runtime know where additional modules are located (%PYTHONPATH%)

experiment_test.py:

import codecs
import os
import shutil


def function_name(file_name):
    print("Py - arg: '{:s}'".format(file_name))
    if not os.path.isfile(file_name):
        return file_name
    with open(file_name, mode="rb") as f:
        content = f.read().decode()
        print("Py - Content len: {:d}, Content (can spread across multiple lines): '{:s}'".format(len(content), content))
        return content

Notes:

An almost dummy module, as specified at the beginning
Works only with text files (decode will fail for binary files)
Imports modules that aren't used, to see that they are OK (that's obvious, if one such import statement succeeds, all should)
Prints some data on stdout (to be matched with what's on the C++ side)
Located in a path known by Python (%PYTHONPATH% from previous step)
Has 1 argument (file_name) - crucial difference compared to the one in the question which doesn't have any (don't know whether that's a logical mistake or a typo like one)

test_dir\test_file.txt:

line 0 - dummy
line 1 - gainarie

Output (VStudio console):

Py - arg: 'test_dir\test_file.txt'
Py - Content len: 33, Content (can spread across multiple lines): 'line 0 - dummy
line 1 - gainarie'
C(++) - Python call: experiment_test.function_name('test_dir\test_file.txt') returned 'line 0 - dummy
line 1 - gainarie' (len: 33)
OK
Press ENTER to return...

Final note:

In this case (the simple one), PYTHONCORE from installation path was used. But in other ones (e.g. shipped in another product), Python's standard library path must be set before Py_Initialize, via Py_SetPythonHome.
Although it's old, [SO]: What files are required for Py_Initialize to run? (@CristiFati's answer) might contain some useful info

edited May 5, 2023 at 17:51

answered Dec 25, 2017 at 23:44

CristiFati

41.5k9 gold badges68 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Pavel Over a year ago

Thank you for such a detailed answer! Im' trying to use it with my code, but the python module fails to load today. I think, I will try to use your code on my machine if I'll not find my error.

CristiFati Over a year ago

Did you have a chance to test (it's not just the code but also the build process)? If you still have issues executing the Python code, wrap all the code from experiment_test.py in a try/except clause and print the exception.

Pavel Over a year ago

Sorry for keep you waiting - my code doesn't work at the time.

Pavel Over a year ago

It's really interesting now with build process differences. Today I started to rebuild your solution (with my software versions) - but if I write "$(PythonDir)\libs" in Additional dependencies, I get link error. At the time I stopped with "$(PythonDir)\libs\python36.lib" and will continue a little later. I will write about the result.

CristiFati Over a year ago

But you set $(PythonDir) under User Macros for the correct architecture(s) pointing to the right path(s), right?

|

Collectives™ on Stack Overflow

Calling Python function with parameters from C++ project (Visual Studio)

1 Answer 1

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related