Avoiding duplicates when inserting multiple data into MongoDB

Question

Hi I want to insert multiple data and avoid duplicates in my programme. I have implemented the code to insert multiple data to mongoDB using asp.net core web api and it's working fine. I tried so many things but still I couldn't find a way to avoid duplicate records when inserting multiple records. Can you guys help me please :)

I want to avoid inserting duplicates by using employeeid.

this is my controller

using HelloApi.Models;
using HelloApi.Services;
using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace HelloApi.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    [Produces("application/json")]
    public class EmployeeController : ControllerBase
    {
        private readonly EmployeeService _employeeService;

        public EmployeeController(EmployeeService employeeService)
        {
            _employeeService = employeeService;
        }

        //.....................................Create..............................................

        public Task Create(IEnumerable<Employee> employees)
        {
            return _employeeService.Create(employees);

        }
    }
}

This is my model class

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using System.ComponentModel.DataAnnotations;

namespace HelloApi.Models
{
    public class Employee
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        [Key]
        public string Id { get; set; }

        [Required]
        [BsonElement("employeeid")]
        public string employeeid { get; set; }

        [Required]
        [BsonElement("firstname")]
        public string firstname { get; set; }

        [Required]
        [BsonElement("lastname")]
        public string lastname { get; set; }

        [Required]
        [BsonElement("age")]
        public int age { get; set; }

        [Required]
        [BsonElement("address")]
        public string address { get; set; }

        [Required]
        [BsonElement("telephone")]
        public int telephone { get; set; }

    }
}

This is my service class

using HelloApi.Models;
using MongoDB.Driver;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace HelloApi.Services
{
    public class EmployeeService
    {
        private readonly IMongoCollection<Employee> _employee;

        public EmployeeService(IHelloApiDatabaseSettings settings)
        {
            var client = new MongoClient(settings.ConnectionString);
            var database = client.GetDatabase(settings.DatabaseName);

            _employee = database.GetCollection<Employee>(settings.employeeCollectionName);
        }

        //.....................................Create..............................................

        public Task Create(IEnumerable<Employee> employees)
        {
            return _employee.InsertManyAsync(employees);
        }
    }
}

This is the POST request in Postman and it is working fine but submit duplicates.

Really appreciate if you guys can help me with this :)

Remove Duplicates..? which properties...? Do you want to remove duplicate employeeid or multile propties write your own logic at API and return error message to client — Prashanth Reddy
– Prashanth Reddy, Commented Feb 9, 2021 at 4:16
Hi, yes I want to remove duplicated by looking employeeid. I want not to insert same employeeid twice. — Nirdha
– Nirdha, Commented Feb 9, 2021 at 7:05
public Task Create(IEnumerable<Employee> employees) { // var employee get by id for employeeid //if employee found create new collection with existinng emplyees //insert for new employees //update for existing employees } — Prashanth Reddy
– Prashanth Reddy, Commented Feb 11, 2021 at 3:27
Hi @PrashanthReddy I tried to implement this in your way but still struggling to do the update when inserting multiple documents. Can u please provide me an example of implementation? — Nirdha
– Nirdha, Commented Feb 13, 2021 at 11:06

Prashanth Reddy · Accepted Answer · 2021-02-14 14:01:29Z

This Might be helpful. Use below InsertORUpdateEmployee function to find and update using employeeid.

public Task<string> Create(IEnumerable<Employee> employees)
{
    return Task.Run(() =>
    {
        return InsertORUpdateEmployee(employees);
    });
}

//Insert or Update Employees
private string InsertORUpdateEmployee(IEnumerable<Employee> employees)
{
    try
    {
        foreach (Employee emp in employees)
        {
            var empId = emp.employeeid;
            var DB = Client.GetDatabase("Employee");
            var collection = DB.GetCollection<Employee>("EmployeeDetails");
            
            //Find Employee using employeeid
            var filter_id = Builders<Employee>.Filter.Eq("employeeid", empId);
            var entity = collection.Find(filter_id).FirstOrDefault();
            //Insert
            if (entity == null)
            {
                collection.InsertOne(emp);
            }
            else
            {
                //Update
                var update = collection.FindOneAndUpdateAsync(filter_id, Builders<Employee>.Update
                    .Set("firstname", emp.firstname)
                    .Set("lastname", emp.lastname)
                    .Set("age", emp.age)
                    .Set("address", emp.address)
                    .Set("telephone", emp.telephone));
            }
        }
        return "Insert or Updated Succesfully";
    }

    catch (Exception ex)
    {
        return ex.ToString();
    }

}

CloudEmber · Accepted Answer · 2021-02-09 04:49:31Z

1

I think you need to add an extra step to declare your Id field as a Primary Key. [Key] should only work for RDBMS. For MongoDB i found another post here to define a field as a unique key.

(Actually MongoDB doesn't really allow for a Primary Key besides the auto-generated '_id' field, afaik, but you can create unique keys. )

Try looking through the answers posted here: How to set a primary key in MongoDB?

answered Feb 9, 2021 at 4:49

CloudEmber

994 bronze badges

Comments

R2D2 · Accepted Answer · 2021-02-09 05:02:59Z

1

It seems you need to create unique index on employeeId field to avoid duplicates :

 db.employee.createIndex( { "employeeId": 1 }, { unique: true } )

edited Feb 9, 2021 at 5:02

answered Feb 9, 2021 at 4:44

R2D2

10.9k2 gold badges20 silver badges41 bronze badges

1 Comment

Nirdha Over a year ago

Hi, yes what I want is to avoid inserting same employeeid records twice.

Collectives™ on Stack Overflow

Avoiding duplicates when inserting multiple data into MongoDB

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related