0

I am working on an analytic API for small ML project. I have created an endpoint, which uses Flask's stream_with_context function like in the example below:

def post():
       # some logic
       [...]

        try:
            res = get_data_from_elastic()

            def generate():
                for hit in res:
                    resp_dict = {
                        "timestamp": hit.timestamp,
                        "user_id": hit.user_id,
                        "node_id": hit.node_id,
                        "loc": hit.loc,
                        "is_target": hit.is_target
                    }
                    yield json.dumps(resp_dict) + '\n'

            return Response(stream_with_context(generate()), status=200, mimetype="application/json")

        # except:
           # some exception handling

It's not an exact extract from my code, but the generator works the same. I use Python requests to connect to the API, with the following code:

response = requests.post(analytic_api_url, 
                         headers={'Authorization': token}, 
                         data={'since': since,'till': till})

When I connect to the API and download small amounts of data at once, everything works fine. Unfortunately, when I try to download bigger amount of data at once, I am getting the following error:

ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

As the error is about chunked encoding, I've checked things like setting Transfer-Encoding: chunked header in the server's response, I've also tried using stream=True parameter from the requests library - none of these solutions worked. How should I deal with this problem? Should I explicitly set some other Transfer-Encoding header, or create another generator for my API?

Thank you for your help!

3
  • Do you have any stacktrace from your flask app? Commented Mar 12, 2021 at 1:12
  • Yup. The answer came as soon as I took a look at this stacktrace once more :) I think I'll left this comment for the people that stumble upon this problem in the future: "node_id": node_details.id AttributeError: 'NoneType' object has no attribute 'id' - some of the documents I've tried to iterate over was not complete (due to the way I collected this data). So, it seems like the point is to always take care of the integrity of the data. Commented Mar 12, 2021 at 22:33
  • You can edit your question and add your solution Commented Mar 15, 2021 at 1:41

1 Answer 1

0

I'll left this comment for the people that stumble upon this problem in the future: "node_id": node_details.id AttributeError: 'NoneType' object has no attribute 'id' some of the documents I've tried to iterate over were not complete (due to the way I collected this data). So, it seems like the point is to always take care of the integrity of the data.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.