MongoDB aggregation returns empty array with NodeJS

Question

I want to create a script which is taking the average of the Volume for last 7(for example) days. I'm stuck with aggregation stages since first stage I need to take Date for last 7 days and in second stage calculate Average of Volume

Package list:

Node-schedule - */1 * * * * (Runs the script every minute)

Binance API - Taking data from them.

Screenshot for showcasing how the document looks like in MongoDB.

Aggregation part of the Code.

const average = await dbo.collection(symbol).aggregate([{
                '$match': {
                    'Date': { '$gte': new Date((new Date().getTime() - (7 * 24 * 60 * 60 * 1000))) }
                },
            },
            {
                '$group': {
                    _id: null,
                    'Volume': { '$avg': '$Volume' }
                },
            }
        ]).toArray();

This code returns me an empty array in terminal like this > []

Full Code here.

const { MongoClient } = require('mongodb');
const schedule = require('node-schedule');
const fetch = require("node-fetch");
const symbols = ["ADABTC", "AEBTC", "AIONBTC", "ALGOBTC", "ARDRBTC"];

//a descriptive name helps your future self and others understand code easier
const getBTCData = async symbol => { //make this function accept the current symbol
    //async/await lets us write this much nicer and with less nested indents
    let data = await fetch(`https://api.binance.com/api/v3/klines?symbol=${symbol}&interval=30m&limit=1`).then(res => res.json());
    const btcusdtdata = data.map(d => {
        return {
            Open: parseFloat(d[1]),
            High: parseFloat(d[2]),
            Low: parseFloat(d[3]),
            Close: parseFloat(d[4]),
            Volume: parseFloat(d[5]),
            Timespan: 30,
        }
    });
    console.log(btcusdtdata);
    saveToDatebase(symbol, btcusdtdata);
    //recursive functions are complicated, we can get rid of it here
    //by moving the responsibility to the caller
};

//helper function for an awaitable timeout
const sleep = ms => new Promise(res => setTimeout(res, ms));

const j = schedule.scheduleJob('*/1 * * * *', async() => {
    //expand this function to be responsible for looping the data
    for (let symbol of symbols) {
        //we can pass symbol to getBTCData instead of making it
        //responsible for figuring out which symbol it should get
        await getBTCData(symbol);
        await sleep(8000);
    }
});

//make this a helper function so `saveToDatabase()` isn't also responsible for it
const getDateTime = () => {
    let today = new Date();
    let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
    let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
    return date + ' ' + time;
};

const saveToDatebase = async(symbol, BTCdata) => {
    try {
        const url = 'mongodb://username:password@ipadress:port/dbname';
        let dateTime = getDateTime();
        let db = await MongoClient.connect(url, { useUnifiedTopology: true });
        const dbo = db.db('Crypto');
        const myobj = Object.assign({ Name: symbol, Date: dateTime }, BTCdata[0]);
        await dbo.collection(symbol).insertOne(myobj);
        const average = await dbo.collection(symbol).aggregate([{
                '$match': {
                    'Date': { '$gte': new Date((new Date().getTime() - (7 * 24 * 60 * 60 * 1000))) }
                },
            },
            {
                '$group': {
                    _id: null,
                    'Volume': { '$avg': '$Volume' }
                },
            }
        ]).toArray();
        console.log('1 document inserted');
        console.log(average);
        db.close();
    } catch (e) {
        console.error(e)
    }
};

EDIT1 If I delete $match part my script is working and I receive average of Volume.

Screenshot of terminal after success try without $match

EDIT2

According to the last answer I understand that I need to change Date format from string to object, but I really can't get how I can do it in this part?

 const getDateTime = () => {
    let today = new Date();
    let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
    let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
    return date + ' ' + time;
};

EDIT3

After editing the Date format I receive a Document in MongoDB in strange Date format like - Date:2020-07-20T13:24:02.390+00:00

Code here:

const getDateTime = () => {
    let today = new Date();
    let date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
    let time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
    return new Date();
};

@ManuelSpigolon I edited the main question in the bottom. Yes it's working without match please take a look on updated quesiton. — Hexycode
– Hexycode, Commented Jul 20, 2020 at 10:44

Manuel Spigolon · Accepted Answer · 2020-07-20 12:15:28Z

3

The problem is on the Date field format.

The getDateTime function returns a string so Mongo is managing the field as a string not as a Date object so the $gte check will compare string not dates.

You should change the function to getDateTime = () => new Date(). Mongo will manage the date correctly storing in UTF Timezone.

Tring to query a date-string in the $match field would be really difficult.

Edit:

To update the typing just:

const getDateTime = () => {
    return new Date();
};

edited Jul 20, 2020 at 12:15

answered Jul 20, 2020 at 10:53

Manuel Spigolon

13.4k7 gold badges60 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Jiří Pospíšil Over a year ago

It wouldn't really be all that difficult ($addFields + $toDate) but fixing the data type is the proper way to go.

Hexycode Over a year ago

@ManuelSpigolon Thank you for answer. I understand my mistake, is there a possibility that you can write me a "fullcode" example of it?

Hexycode Over a year ago

@ManuelSpigolon Could you explain on how I suppose change the format from string to object, I currently trying but unsuccessful.

Manuel Spigolon Over a year ago

Updated answer: when you insert the data in Mongo, the data field must be a Date object, so it will be inserted as Date in Mongo and not as a string

Hexycode Over a year ago

@ManuelSpigolon After editing the code as you provide I receive weird Date in MongoDB. Date:2020-07-20T13:24:02.390+00:00 like this

|

Collectives™ on Stack Overflow

MongoDB aggregation returns empty array with NodeJS

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related