Data consistency across multiple microservices, which duplicate data

Question

I am currently trying to get into microservices architecture, and I came across Data consistency issue. I've read, that duplicating data between several microservices considered a good idea, because it makes each service more independent.

However, I can't figure out what to do in the following case to provide consistency:

I have a Customer service which has a RegisterCustomer method.
When I register a customer, I want to send a message via RabbitMQ, so other services can pick up this information and store in its DB.

My code looks something like this:

             ...
            _dbContext.Add(customer);
            CustomerRegistered e = Mapper.Map<CustomerRegistered>(customer);
            await _messagePublisher.PublishMessageAsync(e.MessageType, e, "");
            //!!app crashes
            _dbContext.SaveChanges();
            ...

So I would like to know, how can I handle such case, when application sends the message, but is unable to save data itself? Of course, I could swap DbContextSave and PublishMessage methods, but trouble is still there. Is there something wrong with my data storing approach?

Saptarshi Basu · Accepted Answer · 2019-03-26 17:14:08Z

0

Yes. You are doing dual persistence - persistence in DB and durable queue. If one succeeds and other fails, you'd always be in trouble. There are a few ways to handle this:

Persist in DB and then do Change Data Capture (CDC) such that the data from the DB Write Ahead Log (WAL) is used to create a materialized view in the second service DB using real time streaming
Persist in a durable queue and a cache. Using real time streaming persist the data in both the services. Read data from cache if the data is available in cache, otherwise read from DB. This will allow read after write. Even if write to cache fails in worst case, within seconds the data will be in DB through streaming

answered Mar 26, 2019 at 17:14

Saptarshi Basu

9,3694 gold badges42 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Deb · Accepted Answer · 2019-11-07 12:40:54Z

0

NServiceBus does support durable distributed transaction in many scenarios vs. RMQ.Maybe you can look into using that feature to ensure that both the contexts are saved or rolled back together in case of failures if you can use NServiceBus instead of RMQ.

answered Nov 7, 2019 at 12:40

Deb

7671 gold badge7 silver badges22 bronze badges

Comments

bz5314520 · Accepted Answer · 2021-11-09 09:31:31Z

0

I think the solution you're looking for is outbox pattern, there is an event related database table in the same database as your business data, this allows them to be committed in the same database transaction, and then a background worker loop push the event to mq

answered Nov 9, 2021 at 9:31

bz5314520

261 silver badge4 bronze badges

Collectives™ on Stack Overflow

Data consistency across multiple microservices, which duplicate data

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related