An inactive Postgres replication slot is filling your disk

· updated · Vladimir Chemeris, the developer of HeapDeck

Disk usage on the primary climbs in a straight line. Checkpoints run, the logs show nothing, the tables grow at their normal pace, and still pg_wal holds tens of gigabytes and keeps growing. The usual suspect is a replication slot whose consumer went away.

The slot and the WAL it keeps

A replication slot records how far its consumer has read: restart_lsn. The primary keeps each WAL segment from that point on, so the consumer can resume where it stopped. Physical slots serve streaming replicas; logical slots serve logical replication subscriptions and change-data-capture tools such as Debezium.

While the consumer is connected, restart_lsn moves forward and PostgreSQL recycles old WAL at each checkpoint. Then the consumer disappears: someone decommissions a replica, switches off a CDC connector, or drops a subscription on the subscriber and forgets the publisher. The slot stays. Its restart_lsn stops moving, and the primary keeps each byte of WAL written since. PostgreSQL sets no limit by default: the server keeps WAL until the disk is full, and a primary that can’t write WAL stops.

A slot can hold back more than WAL. A physical slot used with hot_standby_feedback and a logical slot (through catalog_xmin) keep vacuum from removing rows their consumer might still need. An abandoned slot can therefore also mean growing bloat and a transaction ID age that never goes down.

Finding the slot

Run this on the primary:

SELECT slot_name,
       slot_type,
       database,
       active,
       wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;

Look for active = false with a large retained_wal. On PostgreSQL 17 and later, inactive_since shows the slot’s last use, and the date often tells you who owned it.

wal_status shows the danger the slot is in:

  • reserved: the WAL it needs fits within max_wal_size;
  • extended: it needs more than that, and the server keeps the extra because of the slot;
  • unreserved: the slot will lose required WAL at the next checkpoint;
  • lost: the WAL it needs is gone, and the consumer can’t resume from the slot.

Deciding what to do

First find the owner. Slot names help: a replica’s name, debezium, the name of a subscription. A connected physical consumer shows up in pg_stat_replication; a logical subscriber has the subscription in its own pg_subscription.

If the consumer is gone for good, drop the slot.

SELECT pg_drop_replication_slot('debezium_orders');

PostgreSQL releases the retained WAL, and the disk space comes back at the next checkpoint. If the consumer still exists and comes back later, it can’t resume: you rebuild a replica from a base backup, and a CDC pipeline needs a fresh snapshot.

PostgreSQL refuses to drop a slot in use and names the process that holds it, for example replication slot "x" is active for PID 182. Stop the consumer first.

If the consumer is coming back, keep the slot and watch the disk. Since PostgreSQL 13, max_slot_wal_keep_size caps the WAL a slot can hold. PostgreSQL invalidates a slot that would need more (wal_status = lost) and keeps the disk from filling. You trade an outage for a rebuild of that one consumer, the better deal in most cases.

Keeping it from happening again

  • Alert on slots that are inactive or retain more than a few gigabytes of WAL; a disk usage alert comes too late.
  • Set max_slot_wal_keep_size to a size your disk can afford.
  • Add “drop its replication slot” to the checklist for decommissioning a replica or a CDC connector.

On the phone

In HeapDeck, the Replication tile on Health turns amber the moment a slot goes inactive, days before the disk alert in most cases. The Replication screen lists inactive slots first with the WAL each retains and shows a red banner: An inactive slot makes PostgreSQL keep WAL forever. This is the most common cause of a disk filling up silently. Slot details show wal_status, the time since the slot was last active on PostgreSQL 17 and later, and for HeapDeck Pro a Drop slot button that works after you type the slot name.

More about the screen: replication lag and slots on your iPhone.

Questions

Why does my pg_wal directory keep growing?

Check for a replication slot whose consumer went away. Other causes are a failing archive_command and a high wal_keep_size. Check pg_replication_slots first, since it takes one query.

How do I clean up pg_wal safely?

Fix what holds the WAL, and let PostgreSQL recycle it at the next checkpoint. Don't delete files from pg_wal by hand: that can make the server unable to recover.