0

I have a nodeJS application with a mongoDB database and recently after uploading data into the images database, my api is starting to take 40+ seconds to load data. For some, even if I search through the collection with limits or cursors, the data still takes time to load. Even if I only fetch back some data. If I look for items in mongoDb, I get the error. I have looked at various examples and even researched workers but got confused on its use and was not able to get it working with my code. Any help would be appreciated and thanks in advance.

So although creating an index did improve some of the api's using images, I found that once a couple hundred records are pulled, the items[i].imageData = LZString.decompressFromBase64(image.imageData); is also taking long to decompress all those images. is there a faster way or better way of doing this?

my controller for an api taking too long that fetches images from the database is:

function itemsByCategoryWithImages(req, res, next) {
    cloverService.getAllItemsByCategoryWithImages( req.params.id)
    .then(users => res.send(users))
    .catch(err => next(err));
}

my service is:

async function getAllItemsByCategoryWithImages(category_id) {
    const options = {
        method: 'GET',
        url: some_url,
        headers: {
            Authorization: 'Bearer ' + token,
            ContentType: 'application/json'
        },
        retry: {limit: 3, methods: ["GET", "POST"]},

    };

    const response = await got(options);

    const items = JSON.parse(response.body).elements.map(item=> item.name);

    let promises = [];
    promises.push(response);
    promises.push(imageService.getImageByName(items));
    // const items = JSON.parse(response.body).elements.map(item=> item.id);

    const responses = await Promise.all(promises);


    let images = [];

    const itemsResponse = JSON.parse(await responses[0].body).elements;
    images = responses[1];

    return await this.matchItemImages(itemsResponse, images);
}

my image service funtion is:

async function getImageByName(names, isCategory) {
    if(isCategory){
        return Image.find({$or: [{cloverDataName: { $in: names }}], isCategory: isCategory})
    }
  return Image.find({$or: [{cloverDataName: { $in: names }}]})
}

finally, the match image function:

async function matchItemImages(items, images) {
    console.log(' about to match images', images.length, items[0].name)

    let imagesById = []
    if(images.length > 0) {
        imagesById = new Map(images.map(image => [image.cloverDataId, image]));
        var i = 0, len = items.length;
        while (i < len) {
            const image = imagesById.get(items[i].id);
            if (image) {
                items[i].imageData = LZString.decompressFromBase64(image.imageData);
            }
            ;
            i++;
        }
    }
    return items.sort((a, b) => {
        var textA = a.name.toUpperCase();
        var textB = b.name.toUpperCase();
        return (textA < textB) ? -1 : (textA > textB) ? 1 : 0;
    });
}

when i comment the promises.push(imageService.getImageByName(items));, the speed is restored.

2 Answers 2

0

you need to create index on {cloverDataName: 1}

db.image.createIndex({cloverDataName: 1}, {background: true})
Sign up to request clarification or add additional context in comments.

4 Comments

I will tru that out but can you explain a bit more on what creating an index does and how it helps. I came across this yesterday while researching but after reading up on it, was still confused on how it works.
ALso when will it be best to create index? before the request?
So although creating an index did improve some of the api's using images, I found that once a couple hundred records are pulled, the items[i].imageData = LZString.decompressFromBase64(image.imageData); is also taking long to decompress all those images. is there a faster way or better way of doing this?
please read about indexes on mongodb university, there is a complete course on it's usecase. learn.mongodb.com
-1

I solved the issue with pagination By loading limited items and loading more on scroll as needed.

1 Comment

Please elaborate this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.