Mongodb aggregation remove null values from object with nested properties

Question

Is there a way to remove literally all null or empty string values from an object? We have an aggregation which creates an object with empty fields and empty objects should the value be null. What we wish to do is remove all null properties and empty objects and recreate the object, in order to keep the data as small as possible.

e.g. in the following object, only 'test' and 'more-nested-data' should be taken into account, the rest can be removed


{
    "test": "some",
    "test2": {
    },
    "test3": {
        "some-key": {
        },
        "some-other-key": {
            "more-nested-data": true,
            "more-nested-emtpy": null
        }
    }
}

which should become:

{
    "test": "some",
    "test3": {
        "some-other-key": {
            "more-nested-data": true
        }
    }
}

I tried a lot, but I think by using objectToArray that something could be done, but I have not found the solution yet. The required aggregation should need to recursively (or by defined levels) remove null properties and empty objects.

? The solution there is for top-level fields only. Your request was to trim out any null field at any level of depth. You need recursion for that; see answer below. — Buzz Moschetti
– Buzz Moschetti, Commented May 17, 2022 at 11:06

Buzz Moschetti · Accepted Answer · 2022-02-02 17:02:32Z

Use the $function operator available in 4.4 (Aug 2021) to do this recursively as you note. Given this input which is a slightly expanded version of that supplied in the question:

var dd = {
    "test": "some",
    "test2": { },
    "test3": {
    "some-key": { },
        "some-other-key": {
        "more-nested-data": true,
            "more-nested-emtpy": null,
            "emptyArr": [],
            "notEmptyArr": [
                "XXX",
                null,
                {"corn":"dog"},
                {"bad":null},
                {"other": {zip:null, empty:[], zap:"notNull"}}
            ]
        }
    }
}
db.foo.insert(dd);

then this pipeline:

db.foo.aggregate([
    {$replaceRoot: {newRoot: {$function: {
        body: function(obj) {
            var process = function(holder, spot, value) {
                var remove_it = false;
                // test FIRST since [] instanceof Object is true!                                   
                if(Array.isArray(value)) {
                    // walk BACKWARDS due to potential splice() later                               
                    // that will change the length...                                               
                    for(var jj = value.length - 1; jj >= 0; jj--) {
                        process(value, jj, value[jj]);
                    }
                    if(0 == value.length) {
                        remove_it = true;
                    }

                } else if(value instanceof Object) {
                    walkObj(value);
                    if(0 == Object.keys(value).length) {
                        remove_it = true;
                    }

                } else {
                    if(null == value) {
                        remove_it = true;
                    }
                }

                if(remove_it) {
                    if(Array.isArray(holder)) {
                        holder.splice(spot,1); // snip out the val                                  
                    } else if(holder instanceof Object) {
                        delete holder[spot];
                    }
                }
            };

            var walkObj = function(obj) {
                Object.keys(obj).forEach(function(k) {
                    process(obj, k, obj[k]);
                });
            }

            walkObj(obj); // entry point!                                                           
            return obj;
    },
        args: [ "$$CURRENT" ],
        lang: "js"
      }}
    }}
]);

produces this result:

{
    "_id" : 0,
    "test" : "some",
    "test3" : {
        "some-other-key" : {
            "more-nested-data" : true,
            "notEmptyArr" : [
                "XXX",
                {
                    "corn" : "dog"
                },
                {
                    "other" : {
                        "zap" : "notNull"
                    }
                }
            ]
        }
    }
}

A convenient way to debug such complex functions is by declaring them as variables outside of the pipeline and running data through them to simulate the documents (objects) coming out the database, e.g.:

ff = function(obj) {                                  
            var process = function(holder, spot, value) {
                var remove_it = false;
                // test FIRST since [] instanceof Object is true!                                   
                if(Array.isArray(value)) {
...
printjson(ff(dd));  // use the same doc as above

You can put print and other debugging aids into the code and then when you are done, you can remove them and call the pipeline to process the real data as follows:

db.foo.aggregate([
    {$replaceRoot: {newRoot: {$function: {
        body: ff,  // substitute here!
        args: [ "$$CURRENT" ],
        lang: "js"
      }}
    }}
]);

SDJ · Accepted Answer · 2022-02-02 14:30:07Z

0

Sounds like the unwind operator would help. Checkout the unwind operator at https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/

answered Feb 2, 2022 at 14:30

SDJ

8991 gold badge7 silver badges14 bronze badges

Collectives™ on Stack Overflow

Mongodb aggregation remove null values from object with nested properties

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related