2

I was trying to model a fairly simple real world model inside a distributed system and got stuck thinking about timing and order and would appreciate some external view on it.

Assuming I have this simple model

enter image description here

  • The actor produces a fixed "income" at the end of each month
  • The account produces a "balance" at the start of each month

These objects are connected via an event broker that adds latency, caching and all the other complexity of distributed systems.

I've tried to design this solution with a message broker. I also added an actor that produces "time" events. The time-actor creates two events: "end of month" and "start of month" to inform all listeners that time has passed. See it as a logical clock.

However, while producing "income" events and "balance" events appear straight forward on paper, I run into order of execution issues, which turns out, is essentially a race condition between the individual topics (channels) of the system:

enter image description here

In this image you see that the time producer produces the "end of month" event that causes the actor to produce an "income_produced (1000)" event.

Very shortly after, the next event "start of month" event is produced. The account object has not yet caught up with the income_produced event. Because the time events were very close to each other.

Now there is a race condition for account, if it receives the income_produced event before the "start of month" event, it will report the correct account balance. If the order is reversed, it will report the wrong balance.

This is intentionally a "bank account" example and I fail to see a good solution to be strongly time ordered.

There is no way for this system to ensure "enough" processing time has passed between "end of month" and "start of month" to let all objects "catch up" with the state.

How is this usually solved?

I am aware, that one workaround would be to use not three but only one topic / channel in a system were at least the order of operations inside one topic would be correct enough, when the order of writes is correct. (Kafka would for instance also require it to be a single partition, AFAIK).

But if I would design a more complicated system, then there would be many more topics / channels, on which the objects operate. Forcing them all into one single topic / channel would introduce a lot of complexity, since all objects would need to read and potentially discard a lot of messages, which are not relevant for them. Also the entire system would essentially become one single sequential system, which feels odd.

How would somebody with more experience approach this?

Edit: Minor clarification: This is not an approach to fully model how banking or accounting (in the US or anywhere) works. It is just a prototype I am thinking about to explore the problem-space where I was checking whether eventual consistency could work or where I am making wrong assumptions about the concept.

Improve this question
2
  • "I was checking whether eventual consistency could work" - could work for what application? Schemes like credit cards were built around eventual consistency, but a consequence of it is that the credit limit is technically unenforceable - instead, the bank accepts the financial risk of the cardholder exceeding their limit. But whether that approach "works" depends on the context. Commented 19 hours ago
  • Consider that blockchains were invented to solve this problem... Commented 3 hours ago

4 Answers 4

Reset to default
8

To be honest, I don't think the example application is modeling the requirements correctly. From what I gather, there are two events: one to produce income, and another to balance the account. The context of your example is finance, and we have specific rules for this: accounting.

You can still have a distributed system here - banks were arguabley the original distributed system - I just don't think you have two message queues. You only need one queue that accepts transactions. Each transaction will add to the account or take away from the account. Each transaction needs a timestamp, so that when some background process runs at the start of each month, it can grab all the transactions from the last month and calculate the balance. And then you need rules about how to handle late transactions; for that you would need to consult someone who runs a bank (again, we have specific rules governing accounting and bank transactions).

Obviously, you don't want the message queue being your permanent storage. The queue can be a buffer so you can accept messages while you are persisting others incrementally.

Instead of working through a concrete design, have a look at the following technologies and concepts:

  • Eventual consistency — this is the heart and soul of micro services or any system that eschews ACID transactions in favor of distributed transactions.

  • Event sourcing — captures every change to application state as a sequence of events.

  • Time series database — essentially a data store optimized for storing timestamps with a corresponding value. This isn't a silver bullet, but it's another option when you need to capture time-related data which might work better than your run-of-the-mill relational database.

There is no single way of designing distributed systems, and when you do design such a system, you need to fully understand the real world requirements. This implies you need access to subject matter experts so you can fully quantify the constraints your system needs to operate within. Beyond that, the concepts above, while abstract, provide the foundation for tackling these kinds of problems.

Improve this answer
2
  • "banks were arguabley the original distributed system" - yes but in that condition they don't have strong ordering, and they can have unobservable "in flight" transactions. Commented 19 hours ago
  • Moreover, the "eventual consistency" in banks is a process called "reconciliation", which to this day is incredibly manual. It is one of the reasons why blockchains have been floated as a solution to this problem Commented 3 hours ago
5

There essentially is no strong ordering with a distributed system. A strong ordering is a property of a centralised system, because a centralised system is capable of deciding a single consistent ordering, and in such a system some central function has the authority to make such decisions and impose those decisions upon the other peripheral parts.

The pattern these developments often follow is that the developer starts with a "distributed" system, realises it can't do what is necessary for their purposes, then tries to build their own ad-hoc centralised system using non-local hardware elements, rather than simply restarting the development with standard centralised technologies running on a single machine, with telecoms to allow remote control.

Your example also isn't a "real world" example of how bank accounts really work.

All systems have the concept of a "settlement time", which is the budgeted duration between things happening, and the effects of these happenings propagating to all other necessary parts of the system when the system is working correctly.

In centralised systems which are highly non-local - that is, there are geographically distributed parts which are yoked together by telecoms links into one system - the settlement time can be large and variable, and the risk of breakdown (that is, a condition in which the system has detectably failed to settle within the duration expected in its design) can be very high.

Improve this answer
4

Your contrived example doesn't appear to be about GAAP finance accounting, and is just trying to get at causality in a distributed system. So I will ignore the XY aspect and move on, inventing some tighter assumptions for the example.

Thank you for the narrative description of the problem setup; that's good. However, we don't quite see what the Invariants and the Correctness Requirements are, so I will flesh those out. Typically it is reasoning about invariants that will let us decide whether a distributed system is correct or is buggy / racy.

A (not shown) actor will consume the Account Balance channel and take actions like sending a statement via Postal Service or via e-mail, for January and later for February. That balance consumer should avoid sending twice in a month, and it must send an accurate account balance figure which goes with the given month. The consumer can have a local state machine which helps it suppress duplicate sends, and helps it retry / recover when it reboots or other exciting events happen, like healing of a network partition.

The chief difficulty with your "Correct" and "Incorrect Order" examples is that those messages omit info which the balance consumer would need in order to properly interpret their meaning. You have stripped timestamp / causality information from them. So let's revisit the design of this Public API and its Messages.

Actor hears a message and in response to that sends an Income message. The income message should mention the clock message which triggered it, perhaps as the tuple:
(end_of_January, 1000).

When the Account Service sends a message, it should mention the clock message that triggered it, as well:
(start_of_February, (end_of_January, 1000))

We're worried about this racy sequence of arrivals heard by the Account Service:

  • Clock: start_of_February (what?!? no income?), followed by
  • Actor: 1000

Depending on the business rules of your Use Case, and CAP requirements, you may wish for the Account Service to be in an "inhibited" state until all Actors have sent in an end_of_January income update. Keep that Clock message buffered, ready to be delivered as soon as we're no longer inhibited.

As a practical matter, many distributed system servers will locally increment a serial number and send that as a message ID. That lets peers conveniently summarize "I have seen all of Alice's message up through serial 7", or perhaps holes such as "I have seen through 7, and also 9, but not yet 8". TCP SACK does this all the time.

To properly interpret and act on a message, an actor like the Balance Consumer may need to see several IDs in that message. Perhaps IDs from Clock, Actor, Clock, and Account Balance. We might call this a Vector Clock, or Lamport Clock. A well-designed Account Balance service might offer the computational service of eliminating some race ambiguity, such as by being inhibited or by retrying, and that can let it get away with sending smaller messages containing fewer upstream IDs.

To clarify, I'm not contemplating sending NTP synchronized wallclock readings anywhere in this, as they only cause trouble when trying to reason about causality. When I speak of a Network Clock or a Lamport Clock, that is in the generally accepted sense of some server locally incrementing a sequence number and sending that number as its clock. Essentially time ticks forward by one with every message sent by that server.

Suppose that Alice's server A was at sequence number 7, and Bob's server B was at 1003. Then we might see messages like

  • A: (7, launch_missle)
  • B: (1003, missle_launched)
  • A: (8, acknowledged)

We tend to think of wallclock time as global. Network clocks are always w.r.t. the sending server. So we might speak of these three observed network clocks:

  • (A, 7)
  • (B, 1003)
  • (A, 8)

Viewing a number like 7 or 8 in isolation isn't meaningful, because a distributed system doesn't really have any "global" timestamps. We always mention the sender when logging a network clock or when sending it in a message. It is legitimate for Bob to forward Alice's (A, 7) onward to Charlie's server C, to show Charlie the causal chain.

Improve this answer
3
  • The problem with using timestamps is the need for a centralised clock. If you partition the clocks, you lose strong ordering again. Commented 19 hours ago
  • 1
    @Steve. Right, a centralised clock doesn't help a distributed system. That's why we need Lamport clocks. I made an edit to try to clarify. Commented 13 hours ago
  • I was more referring to the fact that there isn't any solution that allows a distributed system to have the properties of a centralised system - such as simplicity, consistency, and a definite ordering of events. Commented 11 mins ago
-1

Distributed systems should be fault tolerant (for philosophical discussions see reactive manifesto), that is they should support event lose, everything else could be implemented by enhancing the implied entities with states that are changed by received events according to afore configured transitions, the way transitions diagram is, and optionally acknowledgment means (maybe bidirectional event driven communication?).

For reference the scrap of reactive manifesto:

Responsive: The system responds in a timely manner if at all possible.

Resilient: The system stays responsive in the face of failure.

Elastic: The system stays responsive under varying workload.

Message Driven: Reactive Systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling, isolation and location transparency.

Improve this answer
New contributor
user481404 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.