Sending large javascript array to Web API

Question

I am struggling with this issue for 2 days...

I have a JavaScript array (20,000K rows and 41 columns). It was originally received in javaScript through an ajax call as shown below,

var dataArray = [];
var dataRequest = {};
            dataRequest.SearchCondition = 'some value'; 
            $.ajax({
                type: "POST",
                url: "api/GetData/ProcessRequest",              
                dataType: 'json',
                cache: false,
                async: true,
                crossDomain: false,
                data: dataRequest ,
                success: function (response) {
                  dataArray = response; 
                },
                error: function (xhr, textStatus, errorThrown) {
                   dataArray = null;
                }
            });

In the application, the user will verify the data and send it back to Web API method.

I am trying to send the same data back (dataArray) to web api method but, it fails. Please see the code below,

Option 1: (failed - the request did not hit web api method)

 var dataArrayJsonStr = JSON.stringify(dataArray);                     

    $.ajax({
        type: "POST",
        url: "api/SendData/ProcessRequest",
        dataType: 'json',
        data: {'dataValue':dataArrayJsonStr },
        success: function (response) {
            alert('success');
        },
        error: function (xhr, textStatus, errorThrown) {
            alert(errorThrown)
        }
    });

In IE 8, I am getting 'out of memory' exception popup. (most of our application users still have IE 8)

In Chrome, it crashes.

Option 2 tried: (don't know how to read the value)

I tried to send the same value to web api through XmllHttpRequest

var dataArrayJsonStr = JSON.stringify(dataArr);
            var xmlRequest;
            if (window.XMLHttpRequest) {
                xmlRequest = new XMLHttpRequest();                
            }
            xmlRequest.open("POST", "api/SendData/ProcessRequest", false);
            xmlRequest.setRequestHeader('Content-Type', 'application/text');
            xmlRequest.send("dataValue=" + dataArrayJsonStr);

Using Chrome, I am able to post the data successfully to Web API, I am seeing the content-length as '128180309'. But, I don't see the values. How do i get the values in Web API?

Please suggest me how to send large data back to web api from javascript.

Thanks, Vim

vasilyk · Accepted Answer · 2015-02-05 19:36:37Z

1

I think you create overhead, maybe I wrong, you can edit me. Did you really need send back all datas back or you just need send modified data? Because in real life hard to imagine that user will review 20.000 of rows.

Good example is ExtJS stores, you can see example here Key thing of stores that they send back to the server only modified or deleted data, it save browser, network and server resources.

answered Feb 5, 2015 at 19:36

vasilyk

1621 silver badge11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Vim Over a year ago

I have to send all the data back.

hfloresv · Accepted Answer · 2015-02-05 18:57:42Z

0

Try to add more memory for API or more time excecution, also you can try return data in more small parts. Defining the number of parts to send.

answered Feb 5, 2015 at 18:57

hfloresv

542 bronze badges

5 Comments

Vim Over a year ago

Can you provide me code sample or refer to an article to return data in small parts?

hfloresv Over a year ago

I will search an article, but is more of logic, you obtain length from array then var numbers_object = object / 1000( registres); and execute the response with each or for I search a example or create for give a idea

Vim Over a year ago

In this case, how do I receive the data in Web API and make them to a single object?

hfloresv Over a year ago

I think that is something like this: global array= [][] function get_data(){ part = request.part; part_data = request.data; end = part_data.length * part; count = end - part_data.length; for(count; count <= end; count++){ global array[count][part_data[count]]; } } Anyway you need check this logic with your code, but I think that this can run

hfloresv Over a year ago

If the user only check the info but don't update nothing, in your api save the same data before sending in a global variable, and with the response of the user, use this data save. This only if user don't update data.

Kirill Rogovoy · Accepted Answer · 2015-02-06 08:24:50Z

0

Did you try to send the data by chunks? I mean, you need to split it in small pieces and perform multiple number of requests.

For example, it can be like:

--HELLO SERVER. STARTING TRANSMITION FOR DATA SET #177151--  
PIECE 1/13  
PIECE 2/13  
...  
PIECE 13/13  
--BUE SERVER--

So, it will take some time, but you can send any amounts of data without memory problems. If you're struggling with it for 2 days, I think you got some time to code it :)

UPD1: Client code example.

Here's an example of client code. This is a simple chunking algorithm. Have to say I didn't test it, because it would take a lot of time to represent your situation.

So, you should read it and get the point. You have a simple function, that takes you whole data set and callbacks for each response (to update your progress bar, e.g.), for successful finish and for error.

Hope, it will help you to make some problems. Also, I can help you to build architecture on the server-side, but I need to know what technologies do you use.

function sendData(data, onEach, onFinish, onError) {
    var CHUNK_SIZE = 1000;
    var isFailed = false;
    var chunkNum = 0;
    var chunk, chunkStart, chunkEnd;

    while(data.length + CHUNK_SIZE > chunkNum * CHUNK_SIZE) {
        if(isFailed) {
            return;
        }
        chunkStart = chunkNum * CHUNK_SIZE;
        chunkEnd   = chunkStart + CHUNK_SIZE + 1;

        chunk = {
            num: chunkNum,
            data: data.slice(chunkStart, chunkEnd)
        };

        ajaxCall(chunk);
        chunkNum++;
    }

    function ajaxCall(data) {
        $.ajax({
            type: "POST",
            url: "api/GetData/ProcessRequest",              
            dataType: 'json',
            async: true,
            data: dataRequest ,
            success: function (response) {
                onEach(data, response);
            },
            error: function (xhr, textStatus, errorThrown) {
                isFailed = true;
                onError(arguments);
            }
        });
    }


}

edited Feb 6, 2015 at 8:24

answered Feb 5, 2015 at 19:01

Kirill Rogovoy

6014 silver badges12 bronze badges

6 Comments

Kirill Rogovoy Over a year ago

Also, you can send the number of current chunk with whole request and send all requests asynchronously. Then server will able to construct right set of data in right order when last piece will be recieved

Vim Over a year ago

Thanks for the suggestion. I did not try sending the data by chunks. Can you provide me code sample?

Kirill Rogovoy Over a year ago

You also can modify the code to send "hello message" and tell the server, how many chunks does he should to receive and give each separate transmission a uniq id that generates on the server-side as response on the "hello message"

Vim Over a year ago

Thanks for the code sample. I will try it today. Earlier, I tried using Blob in HTML5 and it worked perfectly in Chrome but not in IE8 as it is not supported. One question though... in the ProcessRequest method, how do we combine the chunks in to a complete object?

Kirill Rogovoy Over a year ago

And why can't you send only the difference between data that user made with "client version" of data and server version? I mean, user takes 20k rows and he changes just a few of them. Why can't you send back only changed rows and let server know that other rows wasn't changed? Like Version Control Systems send only the difference of files instead of whole file

|

Collectives™ on Stack Overflow

Sending large javascript array to Web API

3 Answers 3

1 Comment

5 Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

5 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related