Postgres CDC failure and WAL filling up disk space?

Question

Suppose a CDC data loading tool (Fivetran, Airbyte, Debezium, etc) gets into a failure state and is not successfully running its incremental syncs of a Postgres data source. Also suppose that max_slot_wal_keep_size is set to -1. As far as I can tell, this means that the amount of WAL retained for the tool's replication slot will be allowed to grow without bounds.

My question: assuming the tables continue to grow, is the end state of this scenario that the WAL eventually fills up all disk space, causing total catastrophe for the production database? Does the answer depend on whether the tool is pointed at read-only replica of the database?

When the database server runs out of disk space, it will stop. The tool used for the CDC process doesn't matter. — Frank Heikens
– Frank Heikens, Commented Feb 24 at 17:28
@FrankHeikens Then I guess what I am asking is: in the case where 1. some CDC tool is pointing to the read-only replica of a Postgres database 2. that CDC tool stops working 3. max_slot_wal_keep_size is set to -1.... will the hard drive of the database begin to completely fill up? — just-some-questions
– just-some-questions, Commented Feb 24 at 19:45
Yes, that's what's going to happen. This is not related to the CDC; it's related to your configuration. There are many ways to bring down a server; this is just one of them. — Frank Heikens
– Frank Heikens, Commented Feb 24 at 20:26

Laurenz Albe · Accepted Answer · 2025-02-24 20:53:02Z

1

If you configure PostgreSQL to retain all WAL required for logical decoding, and the progress of logical decoding is stalled, the disk will eventually fill up and PostgreSQL will crash. To prevent that from happening, there is the parameter max_slot_wal_keep_size, which will make PostgreSQL abandon the replication rather than die.

answered Feb 24 at 20:53

Laurenz Albe

257k22 gold badges312 silver badges388 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sourabh Gupta · Accepted Answer · 2025-03-27 03:53:42Z

0

Yep, if max_slot_wal_keep_size = -1 and the CDC tool is stuck, WAL will keep growing until it fills up disk—can totally bring down the primary. Using a read replica doesn’t help; WAL still piles up on the primary. Always monitor replication slots or set a limit to avoid surprises.

answered Mar 27 at 3:53

Sourabh Gupta

1

Collectives™ on Stack Overflow

Postgres CDC failure and WAL filling up disk space?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related