Chapel-Python interopability - export records (or interface with Python dataclasses)

Question

I have the following use case. My main program is written in Python, and I want to accelerate some parts of it. The Python program makes use of dataclasses (~structures) to store parameters, arrays, etc. Consider the following main Python program ; which uses the Point structure.

# 0/main.py

from dataclasses import dataclass

@dataclass
class Point:
    x: float


def addOne(p: Point) -> Point:
    return Point(p.x + 1.0)


p1 = Point(1.0)
print(p1)
p2 = addOne(p1)
print(p2)

(0) $ python main.py
Point(x=1.0)
Point(x=2.0)

Now I want to accelerate the addOne function. My starting point is to have Chapel define the Point structure, and have Python import it.

# 1/main.py
from libexample import Point, makePoint, printPoint, addOne

p1 = Point(1.0)   # or makePoint(1.0)
printPoint(p1)
p2 = addOne(p1)
printPoint(p2)

// 1/libexample.chpl
// -- types used by exported functions need to be exported
// -- cannot export records, so we use extern
// export record Point {
//   var x: real(64);
// }


extern {
  typedef struct {
    double x;
  } Point;
}

// -- altenatively
// require "point.h";
// -- 
// $ cat point.h
// #ifndef POINT_H
// #define POINT_H
// typedef struct {
//   double x;
// } Point;
// #endif


extern record Point {
  var x: real(64);
}


export proc makePoint(x: real): Point {
  var p: Point = new Point(x);
  return p;
}

export proc printPoint(in p: Point) {
  writeln("Point(", p.x, ")");
}

export proc addOne(in p: Point): Point {
  return new Point(p.x + 1.0);
}

A Chapel test program works fine:

// 1/main.chpl
use libexample;


var p1 = makePoint(1.0);
printPoint(p1);
var p2 = addOne(p1);
printPoint(p2);

(1) $ chpl main.chpl
(1) $ ./main
Point(1.0)
Point(2.0)

But compiling as a shared library does not:

(1) $ chpl --library --library-python libexample.chpl

Error compiling Cython file:
------------------------------------------------------------
...
from libc.stdint cimport *
from chplrt cimport *

cdef extern from "libexample.h":
    void chpl__init_libexample(int64_t _ln, int32_t _fn);
    Point makePoint(double x);
 ^
------------------------------------------------------------
./chpl_libexample.pxd:6:1: 'Point' is not a type identifier

# ... and more of such errors

As I understand, exporting records and classes is not yet implemented: https://github.com/chapel-lang/chapel/issues/9326.

A working (but tedious and unmaintainable) workaround would be to write Python wrappers to unpack everything into basic types to call into Chapel, and repack the returned value(s)

# 2/main.py
from dataclasses import dataclass


@dataclass
class Point:
    x: float


def addOne(p: Point):
    from libexample import addOne as addOne_chpl
    return Point(addOne_chpl(p.x))


p1 = Point(1.0)
print(p1)
p2 = addOne(p1)
print(p2)

// 2/libexample.chpl
export proc addOne(p_x: real(64)): real(64) {
  return p_x + 1.0;
}

Then compiling and running:

(2) $ chpl --library --library-python libexample.chpl
(2) $ python main.py 
Point(x=1.0)
Point(x=2.0)

Obviously doing this is undesirable and error-prone, especially when dealing with nested structures.

I have also looked into the Python module and saw it allows defining custom types (https://github.com/chapel-lang/chapel/blob/main/test/library/packages/Python/correctness/customType.chpl).
As far as I understand, you can call Python code from Chapel (with possibly custom structures), but I want to do the opposite: call Chapel code from Python with custom structures. The structures may be defined first either in Python or in Chapel (though defining them in Chapel first seems to be more reasonable)

Is there currently a "good" way to do this ? Ideally, the structure would be defined only once, without having to rely on custom-written TypeConverters and such (which in practice duplicate the code). Thank you for your help !

Answer 1 · 2025-11-21 16:05:59Z

I think you're on the right track with this by looking at the Python module. What I want to try here is something along the lines of defining the type originally in Python and relying on the Python module's Class wrapper type, then potentially defining methods in Chapel as much as possible and calling over into the Python type for only the more basic functionality. But I'm not sure if that'd allow you to call those methods from Python, or even what the performance would look like from doing so, as it might still require a custom TypeConverter to accomplish that or even some feature requests for things we haven't implemented yet.

I'm hoping Jade will chime in on this thread, they may have some better ideas than I do. In the meanwhile, I'll muse on the structure of this and play with some things locally.

Answer 2 · 2025-11-21 20:06:43Z

I would avoid trying to mix the Python module and a Chapel shared module loaded by Python. I suspect that will cause weirdness and likely crash.

That said, I think its possible to do this with some careful wrapping of types. As you said, the ideal would be to support export record. That's definitely worthwhile to implement to improve our interop in general, so I encourage you to comment on https://github.com/chapel-lang/chapel/issues/9326 and cite your usecase.

In any case, I've created a working example that hopefully will work for you with minimal tedious coding.

// libexample.chpl
use CTypes;
class Point {
  var x: real(64);
}
type handle = c_intptr;

proc unwrap(pv: handle): unmanaged Point? {
  return pv:c_ptr(void):unmanaged Point?;
}

export proc getX(in pv: handle): real(64) {
  return unwrap(pv)!.x;
}

export proc makePoint(x: real): handle {
  var p: Point = new unmanaged Point(x);
  var ret = c_ptrTo(p):handle;
  return ret;
}

export proc printPoint(in pv: handle) {
  writeln("Point(", unwrap(pv)!.x, ")");
}

export proc addOne(in pv: handle): handle {
  return makePoint(getX(pv) + 1.0);
}

# main.py
from libexample import makePoint, printPoint, addOne, getX, chpl_setup, chpl_cleanup

class Point:
    def __init__(self, x):
        self.instance = makePoint(x)
    def print(self):
        printPoint(self.instance)
    def addOne(self):
        new_instance = addOne(self.instance)
        return Point.from_instance(new_instance)
    @property
    def x(self):
        return getX(self.instance)
    @classmethod
    def from_instance(cls, instance):
        obj = cls.__new__(cls)
        obj.instance = instance
        return obj

chpl_setup()
p = Point(2.5)
p.print()
p2 = p.addOne()
p2.print()
print("p.x =", p.x)
print("p2.x =", p2.x)
chpl_cleanup()

Here is a quick summary of what I am doing to achieve this

Do all operations on Point in Chapel, avoiding needing an extern/export record.
Use class Point, so we can pass an opaque handle back and forth to Python. WARNING: this must be an unmanaged class, or memory errors may occur. With more care, you could also implement a __del__ for Point to avoid the current memory leak.
Create a thin Python shim that forwards all operations to the opaque Chapel handle. This gives you natural Python syntax while doing all the real work in Chapel.

Collectives™ on Stack Overflow

Chapel-Python interopability - export records (or interface with Python dataclasses)

2 Replies 2

Your Reply

Collectives™ on Stack Overflow

Chapel-Python interopability - export records (or interface with Python dataclasses)

2 Replies 2

Your Reply

Sign up or log in

Post as a guest