Merging arrays that hold CSV data into one object array

Question

I've taken influence from this post to store CSV files in individual arrays, and it works. I'm looking to add about 8 more arrays that each hold the contents of a CSV file.

But how can I merge these arrays into one array of objects? Having an array of objects is critical for my subsequent application of these arrays. What I have so far is below, but I'm not sure if it is correct or not?

Any help would be appreciated.

var dataCSV;
var dataCSV2;

var collection = {dataCSV, dataCSV2};

d3.csv("data/csv1.csv", function(data) {
    data.forEach(function(d) {
        d.word = d.word;
        d.frequency = +d.frequency;
})
    dataCSV = data;
    console.log(dataCSV);
});

d3.csv("data/csv2.csv", function(data) {
    data.forEach(function(d) {
        d.word = d.word;
        d.frequency = +d.frequency;
    })
    dataCSV2 = data;
    console.log(dataCSV2);
});

function merge(data1, data2) {
    collection = {data1, data2};
    console.log(collection);
}

merge(dataCSV, dataCSV2);

Ideally, I'd like the output of collection to be:

collection {
    dataCSV {
        word: ..., frequency: ...,
        word: ..., frequency: ...
    },
    dataCSV2 {
        word: ..., frequency: ...,
        word: ..., frequency: ...
    }
 }

Are you bound to using an old version of d3/d3-queue? You could make your life easier if you upgraded to v5. — altocumulus
– altocumulus, Commented Sep 13, 2018 at 15:25
I had no idea V5 was available... I'll look into it. What a solution for what i need look like in V5? — dumbquestionsbydumbpeople
– dumbquestionsbydumbpeople, Commented Sep 13, 2018 at 15:27
Have a look at my answer to "Load multiple files using the d3-fetch module". d3-fetch module replaces the now deprecated module d3-queue in v5. — altocumulus
– altocumulus, Commented Sep 13, 2018 at 16:12
@altocumulus This is confusing, i may need to post another question — dumbquestionsbydumbpeople
– dumbquestionsbydumbpeople, Commented Sep 13, 2018 at 19:19
The solution from my answer I linked to is basically the same as the one proposed by @OriDrori's answer. However, v5 makes your life easier as there is no need to create new Promises yourself as d3 makes use of the Fetch API which will return Promises on its own, i.e. d3.csv returns a Promise. This is what you can see in my answer. Same technique, just shorter. — altocumulus
– altocumulus, Commented Sep 13, 2018 at 19:40

Ori Drori · Accepted Answer · 2018-09-13 15:11:37Z

1

Instead of repeating yourself, create a single method that gets the data, formats it, and returns a promise. Then collect the results of several promises using Promise.all(). Then you can collect both arrays in an object.

Example (not tested):

const getCsv = (url) => new Promise((resolve) => {
  d3.csv(url, function(data) {
    data.forEach(function(d) { // you can use a Array.map() instead for a cleaner solution
      d.word = d.word;
      d.frequency = +d.frequency;
    })

    resolve(data);
  });
});

Promise.all([ // load both csvs, and wait for the results of both
  getCsv('data/csv1.csv'),
  getCsv('data/csv1.csv')
])
.then(([dataCSV, dataCSV]) => { 
  const collection = { dataCSV, dataCSV2 };
  console.log(collection );
});

edited Sep 13, 2018 at 15:11

answered Sep 13, 2018 at 13:44

Ori Drori

196k32 gold badges243 silver badges233 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dumbquestionsbydumbpeople Over a year ago

Hey man, just want to say this works great. But is it possible to have collection in the global scope when using promises?

Ori Drori Over a year ago

You can assign the data inside the function to a global variable (like you do in your original code), but it's not sure that it'll we'll there when you'll want it. In your original code, add a console.log(dataCSV1) after the d3.csv();. The result will always be undefined. Because of the async call, code after the call is executed before the result arrives, and the only way to know when it's done is using a callback, promise, generator or async/await (which uses promises). My advice to you is to learn how javascript works.

Collectives™ on Stack Overflow

Merging arrays that hold CSV data into one object array

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related