Trigger Dataflow(Python SDK) Job using cloud-functions

Question

I have a requirement to process incoming csv files in gcs.

I am not able to trigger execution by shell commands in cloud-functions like:

subprocess.run([
    "python", "-W", "ignore", "dataflow_ingestion_engine.py",
    "gs://logs-check/csv_input/input2.csv", "--runner", "DataflowRunner"]
    )

or

command = """
python tmp/dataflow_ingestion_engine.py gs://logs-check/csv_input/input2.csv --runner DataflowRunner
"""
os.system(command)

Cloud-function is not execution the shell execution part(Not seeing anything in the log). How to achieve the requirement of triggering dataflow(Python-SDK) using cloud-function?

Jayadeep Jayaraman · Accepted Answer · 2019-10-23 09:18:34Z

2

You cannot invoke dataflow jobs from Cloud Functions using a sub-process command. You will have to make REST or gRPC calls from your cloud function code.

A sample job to run templates can be found here [1].

You can refer to this and submit your own job.

1 - https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/dataflow/run_template

answered Oct 23, 2019 at 9:18

Jayadeep Jayaraman

2,8453 gold badges17 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Arnab Mukherjee Over a year ago

Can you share some tutorial on creating template?

Jayadeep Jayaraman Over a year ago

You can take a look at cloud.google.com/dataflow/docs/guides/templates/…. Also, in the dataflow job page there are multiple templates already available. All those templates can be found here - github.com/GoogleCloudPlatform/DataflowTemplates. If this answer helped please do accept the answer.

Arnab Mukherjee Over a year ago

Thanks for the answer but the google tutorial for creating templates is not that great.

Jayadeep Jayaraman Over a year ago

Thanks for the feedback, in the tutorial at the end there is a link to provide feedback. It would be great if you can provide your feedback so that we can improve upon your suggestion.

Collectives™ on Stack Overflow

Trigger Dataflow(Python SDK) Job using cloud-functions

or

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

or

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related