Distributed Dask CPP workers

Question

DASK has a very powerful distributed api. As far as I can understand it can only support though native python code and modules.

Does anyone know if distributed DASK can support c++ workers? I could not find anything in the docs.

Would there be any other approach apart from adding python bindings to cpp code to use that functionality?

mdurant · Accepted Answer · 2019-04-09 12:56:14Z

2

You are correct, if you wanted to call into C++ code using Dask, you would do it by calling from python, which usually means writing some form of binding layer to make the calling convenient. If there is also a C API, you could use ctypes or cffi.

In theory, the scheduler is agnostic of the language of the client and workers, so long as they agree with each other, but no one has implemented a C++ client/worker. This has been done, at leats a POC, for Julia.

answered Apr 9, 2019 at 12:56

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Yannis Over a year ago

Thanks for the fast reply. I was just exploring the option of having a c++ worker for computational efficiency and speed. What do you mean the scheduler is agnostic? I read already the open issue in your git but there aren't much details there. Do you mean the add_plugin functionality of the scheduler?

mdurant Over a year ago

No, I mean that a python (or cython, if someone does that) scheduler could in theory interact with client/workers of other languages.

Collectives™ on Stack Overflow

Distributed Dask CPP workers

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related