Mongo remove duplicates in array of objects based on field

Question

New to Mongo, have found lots of examples of removing dupes from arrays of strings using the aggregation framework, but am wondering if possible to remove dupes from array of objects based on a field in the object. Eg

{
"_id" : ObjectId("5e82661d164941779c2380ca"),
"name" : "something",
"values" : [
    {
        "id" : 1,
        "val" : "x"
    },
    {
        "id" : 1,
        "val" : "x"
    },
    {
        "id" : 2,
        "val" : "y"
    },
    {
        "id" : 1,
        "val" : "xxxxxx"
    }
]
}

Here I'd like to remove dupes based on the id field. So would end up with

{
"_id" : ObjectId("5e82661d164941779c2380ca"),
"name" : "something",
"values" : [
    {
        "id" : 1,
        "val" : "x"
    },
    {
        "id" : 2,
        "val" : "y"
    }
]
}

Picking the first/any object with given id works. Just want to end up with one per id. Is this doable in aggregation framework? Or even outside aggregation framework, just looking for a clean way to do this. Need to do this type of thing across many documents in collection, which seems like a good use case for aggregation framework, but as I mentioned, newbie here...thanks.

Community · Accepted Answer · 2020-06-20 09:12:55Z

15

Well, you may get desired result 2 ways.

Classic

Flatten - Remove duplicates (pick first occurrence) - Group by

db.collection.aggregate([
  {
    $unwind: "$values"
  },
  {
    $group: {
      _id: "$values.id",
      values: {
        $first: "$values"
      },
      id: {
        $first: "$_id"
      },
      name: {
        $first: "$name"
      }
    }
  },
  {
    $group: {
      _id: "$id",
      name: {
        $first: "$name"
      },
      values: {
        $push: "$values"
      }
    }
  }
])

MongoPlayground

Modern

We need to use $reduce operator.

Pseudocode:

values : {
  var tmp = [];
  for (var value in values) {
      if !(value.id in tmp)
        tmp.push(value);
  }
  return tmp;
}

db.collection.aggregate([
  {
    $addFields: {
      values: {
        $reduce: {
          input: "$values",
          initialValue: [],
          in: {
            $concatArrays: [
              "$$value",
              {
                $cond: [
                  {
                    $in: [
                      "$$this.id",
                      "$$value.id"
                    ]
                  },
                  [],
                  [
                    "$$this"
                  ]
                ]
              }
            ]
          }
        }
      }
    }
  }
])

MongoPlayground

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Mar 31, 2020 at 16:23

Valijon

13.2k4 gold badges40 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

chacmool Over a year ago

Thank you! Is there any difference/preference between "classic" and "modern" in terms of performance or dealing with large collections?

Valijon Over a year ago

@chacmool Need to try. If you would like to see a benchmark, let me know.

talha_ah Over a year ago

Love reacts only <3

Jay Haran Over a year ago

I get the error undefined variable for "value" where are we defining value?

Valijon Over a year ago

@JayHaran $$value and $$this are reserved variables for $reduce operator. If you iterate an array, $$this is iterating item, while $$value is last stored item

|

whoami - fakeFaceTrueSoul · Accepted Answer · 2020-03-31 16:27:50Z

3

You can use $reduce, Try below query :

db.collection.aggregate([
  {
    $addFields: {
      values: {
        $reduce: {
          input: "$values",
          initialValue: [],
          in: {
            $cond: [
              { $in: ["$$this.id", "$$value.id"] }, /** Check if 'id' exists in holding array if yes push same array or concat holding array with & array of new object */
              "$$value",
              { $concatArrays: ["$$value", ["$$this"]] }
            ]
          }
        }
      }
    }
  }
]);

Test : MongoDB-Playground

edited Mar 31, 2020 at 16:27

answered Mar 31, 2020 at 16:23

whoami - fakeFaceTrueSoul

18.1k6 gold badges35 silver badges51 bronze badges

4 Comments

Valijon Over a year ago

Was typing the explanation :D

whoami - fakeFaceTrueSoul Over a year ago

@Valijon : did not get what you're saying ?

Valijon Over a year ago

I was posting the same solution and was writing the explanation. But you've posted first

whoami - fakeFaceTrueSoul Over a year ago

@Valijon :-) :-)

Collectives™ on Stack Overflow

Mongo remove duplicates in array of objects based on field

2 Answers 2

Classic

Modern

6 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Classic

Modern

6 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related