Uploading Images using multipart/form-data with node.js using the in-build http module

Question

So I need to send two images and an api key as part of a multipart/form-data http request to an api. I'm receiving the images from an aws s3 bucket and that's working perfectly, but whenever I try to send the image as part of the form-data I get an EPIPE http error. Somehow, the request gets canceled before all the data was received by the api. I tried the same using postman and everything works well, it's just my node program that doesn't seem able to achieve this. Please find below the code snippet:

const http = require('http')
const https = require('https')
const AWS = require('aws-sdk')
const s3 = new AWS.S3({apiVersion: '2006-03-01'});

//simple http post request, there doesn't seem to be anything wrong with it 
const httpPromise = (protocol, params, postData) => {
return new Promise((resolve, reject) => {
    const requestModule = protocol === 'http' ? http : https;
    const req = requestModule.request(params, res => {
        // grab request status
        const statusCode = res.statusCode;
        if(statusCode < 200 || statusCode > 299) {
            throw new Error(`Request Failed with Status Code: ${status}`);
        }

        let body = '';
        // continuosly update data with incoming data
        res.setEncoding('utf8');
        res.on('data', data => body += data);

        // once all data was received
        res.on('end', () => {
            console.log(body)
            resolve(body)
        });
    })

    // write data to a post request
    if(typeof(params.method) === 'string' && params.method === 'POST' && postData) {
        req.write(postData)
    }

    // bind to the error event
    req.on('error', err => reject(err));

    // end the request
    req.end();
})
}

const handler = async (event) => {

// requestOption parameters
const apiKey = '000000';
const protocol = 'http';
const path = '/verify';
// set to the defined port, if the port is not defined set to default for either http or https
const port = Port ? Port : protocol === 'http' ? 80 : 443;
const hostname ='www.example.com';
const method = "POST";
const boundary = '__X_PAW_BOUNDARY__';

// get correct keys for the relevant images
const image1Key = 'image1Key';
const image2Key = 'image2Key';
const imageKeys = [image1, image2];



try {
    // get the images, this works all as intended
    const s3GetObjectPromises = [];
    imageKeys.forEach(key => s3GetObjectPromises.push(
        s3.getObject({Bucket: BucketName, Key: key})
        .promise()
        .then(res => res.Body)
    ))
    const [image1, image2] = await Promise.all(s3GetObjectPromises);
    //========== ALL GOOD TILL HERE ============



    // THIS IS WHERE IT GETS PROBLEMATIC:
    // create the postData formData string
    const postData = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"key\"\r\n\r\n" + apiKey + "\r\n--" + boundary + "Content-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\nContent-Type: image/jpeg \r\n\r\n" + image1 + "\r\n--" + boundary + "\r\nContent-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\nContent-Type: image/jpeg\r\n\r\n" + image2 + "\r\n--" + boundary + "--";

    // the formData headers
    const headers = {
        "Content-Type":`multipart/form-data; charset=utf-8; boundary=${boundary}`,
        "Content-Length": `${postData.length}`,
        "User-Agent": "Paw/3.1.7 (Macintosh; OS X/10.14.0) GCDHTTPRequest"
    }
    // the options object
    const options = {hostname, port, path, method, headers};

    let result = await httpPromise(protocol, options, postData)
    console.log(result)
    return result;
} catch(err) {
    console.log(err)
    //this either throws an EPIPE error or it simply states that no key was available
    throw err;
}

//execute the handler
handler()
.then(res => console.log(res))
.catch(err => console.log(err))

Wassily · Accepted Answer · 2018-10-07 09:55:26Z

Alright, so after lots of trying around and experimentation I figured out why the above code wasn't working. First of all, the content-type in the postdata string should be set to image/ but that's a small one wasn't really a reason why this didn't work.

The EPIPE or Network error was due to the fact that I set the Content-Length header to the wrong length. Instead of simply setting it to the length of the string it has to be set to the ByteLength of the string. So simply replace 'Content-Length': postData.length.toString() with 'Content-Length': Buffer.byteLength(postData).toString(). That should solve the EPIPE error.

But there is an additional issue: I'm essentially converting the whole data to one big data string (postData) and sending the whole string in one req.write(postData) operation. So apparently this is not how one should do it (again after much experimentation), instead one should send an array containing the single lines of the data and then write each item of the array to the http request. So essentially:

// instead of this string: 
const postData = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"key\"\r\n\r\n" + apiKey + "\r\n--" + boundary + "Content-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\nContent-Type: image/jpeg \r\n\r\n" + image1 + "\r\n--" + boundary + "\r\nContent-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\nContent-Type: image/jpeg\r\n\r\n" + image2 + "\r\n--" + boundary + "--";

// use this array: 
const postData = [`--${boundary}`, `\r\nContent-Disposition: form-data; name=\"key\"\r\n\r\n`, apiKey, `\r\n--${boundary}\r\n`, `Content-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\n`, `Content-Type: image/jpeg \r\n\r\n`, image1, `\r\n--${boundary}\r\n`, `Content-Disposition: form-data; name=\"image1\"; filename=\"IMG_7264.JPG\"\r\n`, `Content-Type: image/jpeg\r\n\r\n`, image2, `\r\n--${boundary}--`];

and then in the actual request you have to write this array to the http request item by item:

// instead of simply
req.write(postData)
// do:
for(let data of postData) {
    req.write(data);
}

also, make sure to add some functionality for the content-length header calculation that takes into account that the body is now stored in an array, something like this should do the job:

const postDataLength = postData.reduce((acc, curr) => acc += Buffer.byteLength(curr), 0)

And then simply set the Content-Length header attribute equal to postDataLength.

Hope this helps anyone trying to build a form-data post request from scratch instead of using a third-party library like request which would also sort this out for you.

Collectives™ on Stack Overflow

Uploading Images using multipart/form-data with node.js using the in-build http module

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related