Sequential Number Generation Issue in Loop Using Queues

Question

I've encountered a challenge in my current implementation. I'm utilizing queues to invoke a specific function (let's call it function a()) within a loop. The loop triggers the queue, leading to multiple calls of the same function a().

Inside function a(), I'm generating a number based on the totalCountOfSavedDocs + 1. For instance, if the savedOrders are 10, the next one should logically be 1. The generated number is then stored in the database (using mongodb) for the next doc saving.

However, due to the loop and the multiple queue calls, the number generation is not happening sequentially. Consequently, I'm facing an issue where the same number is being assigned to multiple documents.

I'd appreciate any insights or suggestions on how to ensure the sequential generation of numbers despite the loop and multiple queue calls.

Here is the code : This is how queue is called :

 for await (let order of orders) {
const msg: ServiceBusMessage = {
    body: JSON.stringify({
        type: "order-create-single",
        data: order,
        req: reqObj,
    })
};

await sender.sendMessages(msg); 
}

IN queue :

    public async process(orderObj: IOrderAddRequest, req: any) {
    try {
        await this.addOrderService.createOrder(req, orderObj);
    }
    catch (err) {
        throw err;
    }
}

For messages that have been auto-forwarded, this property reflects the sequence number that had first been assigned to the message at its original point of submission. This property is read-only. — P Sampath
– P Sampath, Commented Dec 20, 2023 at 9:34

P Sampath · Accepted Answer · 2023-12-20 15:11:25Z

0

For messages that have been auto-forwarded, this property reflects the sequence number that had first been assigned to the message at its original point of submission. This property is read-only in Azure ServiceBus.

In MongoDB, we can use sequence number as a set and get. So you can update the sequential number in the message body and update in database with an increment. Start the sequential number from 1, get the sequential number from the database, and increment it before sending the queue/database. Refer to this doc for auto-incrementing IDs in MongoDB.

For creating a Service Bus client and sender for a queue. Refer to below:-

const serviceBusClient = new ServiceBusClient(connectionString);
const sender = serviceBusClient.createSender(queueName);

// Sample orders
const orders = [
    { orderId: 1, product: 'Product A' },
    { orderId: 2, product: 'Product B' },
    // Add more orders as needed
];
await sender.sendMessages(message);

getPreviousNumber Function:

async function getPreviousNumber(db, session) {
    const collection = db.collection('sequentialNumbers');
    const result = await collection.findOne({}, { session });
    return result ? result.number : 0;
}

storeNumberInDatabase

async function storeNumberInDatabase(db, session, number) {
    const collection = db.collection('sequentialNumbers');
    await collection.updateOne({}, { $set: { number } }, { upsert: true, session });
}

// Increment the shared counter to generate the next sequential number for Db
                const nextNumber = ++sharedCounter;

enter image description here

Azure: enter image description here

edited Dec 20, 2023 at 15:11

answered Dec 20, 2023 at 11:17

P Sampath

4,2652 gold badges7 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vinayak Sharma Over a year ago

Thanks for your response. The issue lies in my createOrder function, which involves transactions. Let me provide a more detailed explanation from the beginning. I call queues in a loop without specifying an orderNumber initially, as it's autogenerated later. Inside the queue, createOrder runs multiple steps, including getOrderNumber and addOrder. The problem arises because the createOrder function is invoked iteratively, results in multiple transactions. In this scenario, a transaction may not be committed while another transaction is waiting with the same orderNumber, leading to duplications.

P Sampath Over a year ago

The issue is with the asynchronous nature of the calls, and the order number generation occurs within the loop. The sequential number generation is implemented to temporarily lock the generation process so that only one instance of the function (a()) can generate the next number at a time.

Collectives™ on Stack Overflow

Sequential Number Generation Issue in Loop Using Queues

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related