4

I have a go server that is reading & returning large datafiles bundled into a tar. I have tested that this works & writes the tar in pieces and when all the data loads, it's all good.

The problem is that there can be unexpected errors that break the download. I currently write an HTTP error code & an error message, but the error message just gets put at the end of the stream/file. Is there a good way to communicate that the export failed partway through? Is it possible to do this with HTTP status codes & also providing error messages?

I am using the following curl command:

curl --insecure https://127.0.0.1/api/export/030e28f3-4ab6-446a-852e-fda0a497ffe2 -o "test.tar"

Do I have to change the curl command to detect errors (too)?

Improve this question

2 Answers 2

Reset to default
6

If the download started already, then all the HTTP headers are already sent to the HTTP client. You cannot rewrite the status code anymore (it was on the first line).

The only thing you could do is cut the TCP/IP connection. If you used a Content-Length header the client will see that the transfer was incomplete. If you used Transfer-Encoding: chunked the client will see that the end-of-chunks marker was not received. In all cases this will simply invalidate the whole transfer.

You could try to play with Range requests and Partial Content responses, and send the content in several HTTP request-response dialogs. But if you manage the big file transmission over a single HTTP dialog the only thing you can do is breaking the transmission, and starting it back completely.

Usually big file transmissions in chunks are managed on the application layer, on top of HTTP. Like in case of uploads a JavaScript thing will explode the file in chunks, and send it back to a dedicated application server, rebuilding the file, with a dedicated protocol on top of HTTP to ask for missing pieces. This is because in real-life wild HTTP environments, long transmissions are hard, and range/partial content transmissions not very well managed by all potential proxies. On the other hand, using simple medium size HTTP requests works almost everywhere. So if you control both the client and server side you can build your own dialog on top of HTTP and make your own chunked transmission protocol, with good error management.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Is that still true if you use HTTP/2 or HTTP/3?
I'm not an expert on HTTP/2 and HTTP/3, but it seems the concept of sending headers forst is still present. Maybe there's something new there for huge file transfer like an automatic md5 control. But I would still feel safer using an application layer algorithm, and not expecting the HTTP layer to push you an error in the middle of a huge data transfer. Like for example using a second channel to communicate informations about the current transmissions.
1

If you control the client side, you can add an HTTP response trailer header to indicate the error, then handle it on the client side.

For more info about response trailer headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer

Improve this answer

1 Comment

That's new to me! Interesting that someone added that feature to HTTP/1.1.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.