how can i use a recursion function to handle an array?

You don't return anything within your 'if array' case:

if(Object.prototype.toString.call(up)=='[object Array]'){
    up.forEach(function(d){
        return findRoot(source,d);
    })
    // no return

When you don't specify a return, JavaScript will default to returning undefined.

Also note that the forEach function will not do anything with the value you return from the function. One alternative would be to use the map function instead, then return that array again:

var results = up.map(function(d) {
    return findRoot(source, d);
});
return array;

However, this may also not do exactly you're intending to do. Since the only base case your code has is when the value is null, your function will ever only end up returning null, or arrays containing null, rather then something meaningful. For example, calling findRoot(obj, 'hello'); would return [null, null], which is probably not what you want.

If that is the case, you may want to re-think what exactly your recursive function is meant to do + look at adding more base cases, or modifying your existing base and recursive cases.

The problem is you are returning inside your anonymous function in your forEach() loop which isn't actually returning anything for findRoot() so by default it returns undefined.

up.forEach(function(d){
    return findRoot(source,d);//doesn't return for **findRoot()** just for anonymous function.
});

What you can do is return an array of root nodes if there is more than one. You can push the returns to an array then return the array. If there isn't more than one you can just return like normal. Here is an example:

function findRoot(source,key){
    var up = source[key]['up'];

    if(up==null){

        return key

    }else{
        if(Object.prototype.toString.call(up)=='[object Array]'){
            var temp = new Array();
            up.forEach(function(d){
                temp.push(findRoot(source,d));
            });
            return temp;
        }else{

            return findRoot(source,up)
        }

    }
}

If you have an array it will return like this:

If you don't have an array it will return like this:

You can then check to see if the return is an array or not and do what you need to do with the return. Another option is to always return an array and if there is only one element then there would only be one element in the array.