NodeJS: Extracting duplicates from Array

Question

I'd need your help... I'm sorry if the question has already been asked but I can't seem to find a suitable answer for my problem: I'm trying to extract (not remove) the list of the duplicates from my array.

ultimately, the main objective is to keep only one of the duplicated objects (in the array) with the higher profit...

Here's a simple example of my array:

var arr = [
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 10 },
    availability: true,
    option: 1
  },
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 15 },
    availability: false,
    option: 2
  },
  {
    mkBase: "test1",
    mkComp: "test",
    invest: { profit: 8 },
    availability: true,
    option: 3
  },
  {
    mkBase: "test2",
    mkComp: "test",
    invest: { profit: 6 },
    availability: true,
    option: 4
  },
  {
    mkBase: "test",
    mkComp: "test2",
    invest: { profit: 6 },
    availability: true,
    option: 5
  },
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 7 },
    availability: true,
    option: 6
  },
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 10 },
    availability: true,
    option: 7
  },
  {
    mkBase: "test3",
    mkComp: "test4",
    invest: { profit: 10 },
    availability: true,
    option: 8
  }
];

And I managed to extract a list of almost all duplicates using:

for (var i = 0; i < arr.length; i++) {
  if (_.uniqBy(arr, "mkBase").indexOf(arr[i]) == -1) {
    console.log("[SAME BASE]: " + JSON.stringify(arr[i], null, 2));
  } else if (_.uniqBy(arr, "mkComp").indexOf(arr[i]) == -1) {
    console.log("[SAME COMP]: " + JSON.stringify(arr[i], null, 2));
  }
}

And here's the result:

[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test1",
  "invest": {
    "profit": 15
  },
  "availability": false,
  "option": 2
}
[SAME COMP]: {
  "mkBase": "test2",
  "mkComp": "test",
  "invest": {
    "profit": 6
  },
  "availability": true,
  "option": 4
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test2",
  "invest": {
    "profit": 6
  },
  "availability": true,
  "option": 5
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test3",
  "invest": {
    "profit": 7
  },
  "availability": true,
  "option": 6
}
[SAME BASE]: {
  "mkBase": "test",
  "mkComp": "test3",
  "invest": {
    "profit": 10
  },
  "availability": true,
  "option": 7
}

The Lodash method (_.uniqBy) is keeping one of the duplicates in the main Array, and, in order to ultimately get the best (_.maxBy(arr, 'profit')) of the duplicates, I'd need it with the other duplicates.

I'm not sure I'm very clear, but if you need any clarification please let me know!

Thanks in advance to you all!

********** EDIT ************* As suggested by stasovlas you'll find below the expected result and why the other objects in the array were removed:

var result = [
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 15 },
    availability: false,
    option: 2
  },
  {
    mkBase: "test1",
    mkComp: "test",
    invest: { profit: 8 },
    availability: true,
    option: 3
  },
  {
    mkBase: "test3",
    mkComp: "test4",
    invest: { profit: 10 },
    availability: true,
    option: 8
  }
];

var removed = [
  //Reason: Same Base **and** Comp mk as option 2 && Profit is too low versus option 2
  {
    mkBase: "test",
    mkComp: "test1",
    invest: { profit: 10 },
    availability: true,
    option: 1
  },
  //Reason: Same Comp mk as option 3 && Profit is too low versus option 3
  {
    mkBase: "test2",
    mkComp: "test",
    invest: { profit: 6 },
    availability: true,
    option: 4
    //Reason: Same Base mk as option 2 && Profit is too low versus option 2
  },
  {
    mkBase: "test",
    mkComp: "test2",
    invest: { profit: 6 },
    availability: true,
    option: 5
  },
  //Reason: Same Base mk as option 2 && Profit is too low versus option 2 
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 7 },
    availability: true,
    option: 6
  },
  //Reason: Same Base mk as option 2 && Profit is too low versus option 2 
  {
    mkBase: "test",
    mkComp: "test3",
    invest: { profit: 10 },
    availability: true,
    option: 7
  }
];

Do you need the duplicate items anytime in your app ? You might want to take a look at this thread about data structure: "Don't use an array when you actually need a set" : stackoverflow.com/questions/40055764/… I know the context is Firebase but you might find an answer there — t3__rry
– t3__rry, Commented Jan 18, 2018 at 16:02
Thanks I'll be looking into it, but I'm not familiar at all with Firebase. And to answer your question, no I won't need it afterwards... Once I've selected the objects (from the duplicates) that has the highest profit value, I basically throw the others away. — Martin Carre
– Martin Carre, Commented Jan 18, 2018 at 16:46
Glad if it helps. Firebase is a different topic but the answer it provides here might be a good reflexion to have on the data structure (I mean, it made me think differently as well on data structure and habits we all have) Take a look at this answer which is what you're looking for btw: stackoverflow.com/questions/16747798/… — t3__rry
– t3__rry, Commented Jan 18, 2018 at 16:49

stasovlas · Accepted Answer · 2018-01-19 08:56:50Z

1

i'm not sure in my question understanding, but here my solution:

const res = _.reduce(arr, (result, item) => {
    const same = _.find(result, r => _.some([
        r.mkBase === item.mkBase,
        r.mkComp === item.mkComp
    ])); // find same already added item

    if (same === undefined) {
        return _.concat(result, item); // just push item   
    }

    if (same.invest.profit >= item.invest.profit) {
        return result; // do nothing if profit is less then already added
    }

    return _.chain(result) // remove item with smaller profit and push item with higher profit
        .reject({ mkBase: same.mkBase, mkComp: same.mkComp })
        .concat(item)
        .value();
}, []);

edited Jan 19, 2018 at 8:56

answered Jan 19, 2018 at 7:12

stasovlas

7,4362 gold badges30 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Martin Carre Over a year ago

Hey @stasovlas! Thanks for your answer but unfortunately, ultimately I'm looking to get rid of all the duplicates keeping only the highest profit one. For instance: I can't get as a base: "Test" with "test2" as comp and another one with base 'test3' and 'test2' as comp. Just as I can't get "test" as base with "test2" as comp and again base "test and comp "test3" for instance ... I'm sorry this is not very clear I'm afraid! But let me know how I can help clarifying things up...

stasovlas Over a year ago

@Ardzii can you add expected result in your question?

Martin Carre Over a year ago

Done! Thanks stasovlas for suggesting this! :)

Martin Carre Over a year ago

That seemed to have done the trick! Thanks mate! I need to get my head around your answer to apply it now! I'll keep you posted! Thanks again!

stasovlas Over a year ago

@Ardzii huh, my english is too bad for explanation, I hope this article helps you medium.freecodecamp.org/reduce-f47a7da511a9

|

kofman · Accepted Answer · 2018-01-18 20:06:13Z

1

var sameBase = {}, sameComp = {};

arr.forEach(item => {
    let existingBase = sameBase[item.mkBase];

    if ( ( existingBase === undefined ) ||  ( existingBase.invest.profit < item.invest.profit ) ) {
            sameBase[item.mkBase] = item;  
    } 

    existingComp = sameComp[item.mkComp];
    if ( ( existingComp === undefined ) ||  ( existingComp.invest.profit < item.invest.profit ) ) {
        sameComp[item.mkComp] = item;  
    }
});

var sameBaseArr = toArr(sameBase);
var sameCompArr = toArr(sameComp);

console.log("SAME BASE: " + JSON.stringify(sameBaseArr, true));
console.log("SAME COMP: " + JSON.stringify(sameCompArr, true));

function toArr(map) {
    let arr = [];
    for (var key in map) {
        arr.push(map[key]);
    }   
    return arr;     
}

answered Jan 18, 2018 at 20:06

kofman

1066 bronze badges

3 Comments

Martin Carre Over a year ago

Hey Kofman! I think you got something there... I'd need to combine your 2 functions though! Let me check the others' answers and try to combine and I'll come back to you! Thanks!

kofman Over a year ago

Based on what criteria you want to combine? Do you need to just concat the two arrays? Or, deduplicate further based on some equality criteria? E.g. obj1 equals obj2 iff obj1.mkBase equals obj2.mkComp? - something like that? Need more info...

Martin Carre Over a year ago

No problem at all Kofman and thanks for the follow-up... I need to remove the duplicates whether it's the mkBase or the mkComp that are duplicated ie. I need to remain with only a unique mkBase object and a unique mkComp object and the one that I keep needs to be the one with the highest profit. Plus, if in the Array (arr) there are non-duplicates objects (no other mkBase or mkComp (as it is the case with option 8) I need to keep them. I hope this clarifies things! Thanks again! :P

Lyçann H. · Accepted Answer · 2018-01-18 16:59:38Z

0

Try this (not tested, but the logic can help):

var mkBases = [], mkComps = [];

for (var i = 0; i < arr.length; i++) {
    var actualBase = arr[i].mkBase;
    var actualComp = arr[i].mkComp;

    if (typeof mkBases[actualBase] !== 'undefined') {
        mkBases[actualBase] = {getAllIndexes(arr, actualBase)}
    }

    if (typeof mkComps[actualComp] !== 'undefined') {
        mkComps[actualComp] = {getAllIndexes(arr, actualComp)}
    }
}

function getAllIndexes(arr, val) {
    var indexes = [], i = -1;
    while ((i = arr.indexOf(val, i+1)) != -1){
        indexes.push(i);
    }
    return indexes;
}

Now you can iterate over mkBases and mkComp to get your duplicates.

edited Jan 18, 2018 at 16:59

answered Jan 18, 2018 at 16:48

Lyçann H.

1131 gold badge5 silver badges15 bronze badges

Collectives™ on Stack Overflow

NodeJS: Extracting duplicates from Array

3 Answers 3

6 Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related