0

I have been facing an issue on node.js express framework encoding/decoding style. Brief background, I store pdf file in mysql database with longblob data-type with latin1 charset. From server side, i need to send the binary data with UTF8 Encoding format as my client knows utf8 decoding format only. I tried all the possible solutions available on google.

For ex: new Buffer(mySqlData).toString('utf8'); Already tried module "UTF8" with given functionality utf8.encode(mySqlData); But it is not working.

Also i already tried "base64" encoding and retrieve data at client with base64 decoding. It is working just fine but i need to have utf8 encoding set. Also you know base64 certainly increase the size.

Please help guys.

1
  • How do you send binary data as utf-8? Commented Oct 6, 2015 at 11:36

1 Answer 1

1

Ok, your problem is the conversion of latin to utf-8. If you just call your buffer.toString('utf-8'), the latin encoded characters were wrong.

To convert other charset to utf-8, the simple wai is to use iconv and icu-charset-detector. With that, you can switch to utf-8 from all possibles charset (except certains charset).

This is an example of conversion using stream. The result stream is encoded with utf-8 :

var charsetDetector     = require("node-icu-charset-detector"),
    Iconv               = require('iconv').Iconv,
    Stream              = require('stream'),

function convertToUtf8(source, callback) {
    var iconv,
        charsetTestStream = new Stream.PassThrough(),
        newResStream      = new Stream.PassThrough();

    source.pipe(charsetTestStream);
    source.pipe(newResStream);

    charsetDetector.detectCharsetStream(charsetTestStream, function (charset) {
        if (!iconv && charset && !/utf-*8/i.test(charset.toString())) {
            try {
                iconv = new Iconv(charset, 'utf-8');
                console.log('Converting from charset %s to utf-8', charset);
                iconv.on('error', function (err) {
                    callback(err);
                });

                var convertStream = newResStream.pipe(iconv);
                callback(null, convertStream);
            } catch(err) {
                callback(err);
            }
            return;
        }
        callback(null, newResStream);
    });
}
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.