How To call a rest API inside an Apache Flink program

Question

I m wondering if there is a solution to call a

REST API (or multiple REST APIs) inside a flink program directly or not ? if such solution is exist.
Do you think it is better to push my processed data from flink to a message broker like kafka or something at first and then from kafka call REST APIs?
or I can Call REST APIs directly from my flink program as well?

"from Kafka call REST api" - not really clear how you'd make that work, but if you are in control of that server's code, you could embed a Kafka consumer rather than only accept REST — OneCricketeer
– OneCricketeer, Commented Feb 3, 2020 at 14:58
Thanks ,Very nice point. So you believe, Using Kafka consumer instead of rest api as endpoints which consumes processed data is more efficient in a streaming environment? — Salman Meraji
– Salman Meraji, Commented Feb 3, 2020 at 15:55
If you want to have a non-blocking, eventually consistent, reactive architecture, then yes — OneCricketeer
– OneCricketeer, Commented Feb 3, 2020 at 15:58

David Anderson · Accepted Answer · 2020-02-03 07:15:42Z

6

The code in your user functions (e.g. a RichFlatMapFunction or a KeyedProcessFunction) can do anything you want, including making REST calls to external services. However, you should avoid doing blocking i/o in your user functions, because checkpoint barriers can't progress through an operator while it is blocked in the user function.

A good way to approach this then is to use Flink's Async I/O API in combination with an HTTP library that offers an asynchronous client interface.

answered Feb 3, 2020 at 7:15

David Anderson

44.3k4 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Salman Meraji Over a year ago

Thanks alot. Would you please kindly write a simple code which demonestrate how can I call a rest api with using Async I/O?Thanks again

David Anderson Over a year ago

If you try rest + async i/o and get stuck, feel free to ask another question. But I do prefer the more event-driven architecture recommended by @cricket_007 anyway, if that's an option.

Collectives™ on Stack Overflow

How To call a rest API inside an Apache Flink program

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related