Six plants, one event log, and what it actually guarantees

Six plants. Twenty-four systems that need to know what the others are doing, which wired pairwise gives 276 possible connections. Nobody builds 276. They build about forty over five years, each one written by whoever had a free week, and those forty become the only accurate description of how the business runs.

The diagram on the wall is wrong. The forty scripts are right, and no single person has read all of them.

An event backbone is the argument that the forty should have been one thing: a durable ordered log that every site writes to and every consumer reads from at its own pace. It is a good argument. It is also more expensive than the people proposing it usually admit, and the expense is not the servers.

The mesh has no memory

Point-to-point integration moves a record from A to B and forgets it happened. That is the whole problem. When the quality team asks for the same production events six months later there is nothing to hand them, because the transfer was the event and the transfer is gone.

A log inverts that. The producer appends. Consumers hold their own position and read forward at whatever speed they can manage. Adding a seventh consumer does not require touching the producer, does not require a deployment window at six plants and does not require anyone to reconstruct what the producer's author intended in 2021.

Retention is what makes this real rather than rhetorical. Kafka's broker default is log.retention.hours at 168, one week, with deletion enforced a segment at a time and log.segment.bytes shipping at one gibibyte. That week is the actual product. A consumer that broke on Friday catches up on Monday, and a consumer that did not exist on Friday can be born on Monday and still see Friday.

What you bought in exchange is one shared dependency where you previously had forty small independent ones. Forty small failures are irritating. One shared failure stops six plants at once.

There is no ordering across partitions, so there is no ordering across a topic, so there is no global order of anything.

One topic per event type, one field for the plant

The obvious topology is a topic per site. shenzhen.events, ogun.events, and so on down the list. It survives until the seventh site, at which point every consumer needs a code change to subscribe to a topic that did not exist when it was written, and the subscription list becomes a configuration file nobody will touch without a meeting.

Name topics after what happened, not where it happened. production.order.completed. equipment.state.changed. material.consumed. The plant becomes a field in the record and a header on the message, which means a consumer that wants one site filters and a consumer that wants all of them does nothing at all.

There are honest reasons to break that rule and there are about three of them: a legal requirement that one country's records never share a disk with another's, a retention profile that differs by an order of magnitude and a single site whose volume would starve every other tenant. Everything else is a team wanting its own sandbox.

A topic name becomes public API the moment the first consumer subscribes, and it will outlive the people who chose it. Spend the extra hour.

Ordering is a per-partition promise

The guarantee, stated in Kafka's own introduction, is scoped to a topic-partition: events come back in the order they were written to that partition. Read it again for what it does not say. There is no ordering across partitions, so there is no ordering across a topic, so there is no global order of anything.

This makes the partition key the most consequential decision in the design, and it is usually made in an afternoon. The key must be the entity whose history has to stay linear. An asset ID. A work order. A serial number. Key by plant instead and you get one partition per plant, the site with 400 machines gets a hot partition and your parallelism is capped at the number of factories you own.

Causality is what people discover late. Event X in partition 3 caused event Y in partition 11 and the broker will never tell you that. If a consumer must see them in the right sequence you either co-locate them in one partition, which couples their throughput forever, or you carry a sequence number in the payload and sort downstream, which means buffering and a decision about how long to wait for a straggler that may never arrive.

Partition counts also do not really go up. Increase them and hash routing changes, so a key that lived on partition 3 starts landing on partition 7 while its history stays behind on 3. Per-key ordering breaks across that boundary and no tool will warn you. Over-provision at the start.

On the producer side the defaults are now sane and worth knowing exactly. enable.idempotence is true by default. max.in.flight.requests.per.connection defaults to 5 and must stay at or below 5 for idempotence to hold, because the broker retains only five batches per producer ID. Switch idempotence off, leave retries on, keep more than one request in flight, and the documentation says plainly that a failed send can reorder your records.

Replay is a loaded tool

Resetting a consumer group to an old offset and reprocessing is the best debugging instrument in the building. It is also the fastest way to push four thousand duplicate goods movements into an ERP on a Tuesday afternoon.

Split the consumers that compute from the consumers that act. A consumer building a materialised view can be replayed without ceremony. A consumer that issues a command to a PLC or posts a financial document cannot be replayed at all without a guard, and the guard belongs in the sink, not in the pipeline where it will be refactored away by somebody who does not know why it is there.

Three traps sit in the machinery itself. Committed offsets expire: offsets.retention.minutes defaults to 10080, so a group that sits empty for a week returns with no position and resumes wherever auto.offset.reset points. Retention is enforced per segment rather than per record, so the real deletion boundary is coarser and less predictable than the retention setting implies. And a replay written back into the same topic is indistinguishable from new reality to everything downstream unless you stamp it, at which point every consumer needs to understand the stamp.

If replay beyond a week matters, tiered storage is the mechanism. Completed segments move to object storage and consumers reading historical offsets get them through the same API, which converts retention from a disk-sizing argument into a monthly bill. That is usually the trade you want.

Schemas drift slower than the machines do

Confluent's Schema Registry defaults to BACKWARD compatibility, meaning a consumer on the new schema can read data written under the old one. You may add optional fields and remove fields that carried defaults. Adding a required field is rejected. The transitive variants check each new version against every prior version rather than only the most recent, and on a system meant to hold years of plant history, transitive is the honest setting even though it is the annoying one.

None of that catches the failure that actually happens.

A vendor pushes firmware to a torque controller at one site. The field is still called torque, still a float, still passes every compatibility check, and now reports in newton-centimetres instead of newton-metres. The registry is content. Six dashboards are wrong by a factor of a hundred, one of them feeds a maintenance threshold, and nothing fires.

So carry units in the record. Carry a semantic version alongside the schema version, one that a human increments when the meaning changes rather than the shape. Structural compatibility is a tooling problem and it is solved. Semantic compatibility is a governance problem and it is not.

The site that goes dark at 21:00

I have sat in an office in Ogun State where the changeover from grid to generator takes roughly forty seconds, several times in a working day, and the switches come back long before the uplink does. Nobody in that building treats it as an incident. It is Tuesday.

The design that survives this is a small local cluster at each site that keeps accepting writes while the wide-area link is down, with MirrorMaker 2 pulling into a central aggregate cluster once it returns. Kafka's operations documentation lists this shape by name, "feeding edge clusters into a central, aggregate cluster", alongside disaster recovery and cloud migration. It is a documented pattern, not a clever one.

NATS answers the same constraint with a different shape, and its documentation uses precisely this example: a factory floor behind a firewall that cannot accept inbound connections. A leaf node dials outward to a hub and bridges subject interest across that single link. No inbound rule, no public address on the plant side. If your actual problem is connectivity rather than history, that is a far smaller thing to run.

Two details bite on reconnection. The backlog arrives at line rate, and while per-partition order still holds, wall-clock time and event time have now diverged by hours, so any window, join or aggregation keyed on processing time will produce confident nonsense. And producer.id.expiration.ms defaults to one day: a gateway that buffers for thirty hours and then retries has quietly lost the deduplication window it was relying on.

The Chinese manual for one such gateway described its store-and-forward buffer in a single sentence and gave no capacity figure at all. The English datasheet did not mention the buffer. I have stopped assuming a number exists just because the feature does.

Exactly-once ends where Kafka ends

Kafka's design documentation is unusually direct about this. It warns that many systems advertise exactly-once semantics and that the claims deserve a careful reading, because they often quietly assume no producer failure, no consumer failure and no multiple consumer processes. That paragraph sits in the project's own docs, and most people repeating the phrase have never opened it.

What you get is specific. The broker assigns each producer an ID and deduplicates on a sequence number, so a retried send does not append twice. Transactions let a producer write across several partitions atomically. A stream processor can commit its input offset in the same transaction as its output records, and a consumer configured for read_committed will not see records from aborted transactions.

The scope of all that is Kafka to Kafka. Read a topic, process, write a topic. The moment your sink is an MES, an ERP or a Postgres table, the transaction does not reach it, and the documented answer is to store the offset in the same place as the output so the two move or fail together.

Two numbers to check before you rely on any of it. transaction.max.timeout.ms defaults to fifteen minutes, a hard ceiling on how long a processing step may hold a transaction open. transactional.id.expiration.ms defaults to seven days. And the consumer isolation level defaults to read_uncommitted, so unless somebody deliberately changed it, your consumers are already reading records from transactions that were rolled back.

The approach that survives contact with a factory is duller than any of this. Give every event a natural idempotency key, plant plus line plus asset plus a monotonic counter, and make every sink upsert on it. At-least-once delivery then suffices, and you have not bought a distributed transaction to solve a primary key problem.

Most of these plants should run a database instead

Here is the strongest version of the case against everything above. Six sites, a few thousand events a minute, four engineers on the team. An append-only table with an auto-incrementing id, one polling reader per consumer that remembers the last id it saw, and an index. It will work. It will keep working for years. No partition rebalance, no consumer lag dashboard, no broker certificate expiring on a public holiday and no schema registry to govern.

The threshold is not throughput and it is never throughput. A single database instance will out-run the event volume of most factories by a comfortable margin. The threshold is the number of independent consumers and how badly you need to add the next one without touching the producer. Below roughly five consumers the table wins on every axis that matters. The log starts earning its keep when a sixth team asks for the data and the answer has to be yes, this week, with no release at six sites.

The load nobody puts in the business case is human. Kafka ships a minor release about every quarter: 4.1.0 in September 2025, 4.2.0 in February 2026, 4.3.0 in May 2026, 4.3.1 in June. Somebody owns those upgrades across every plant, plus certificate rotation, plus disk headroom on edge nodes, plus lag alerting that can distinguish a dead consumer from a slow one, plus the schema review that stops the torque field changing units. That is a person, not a sprint, and it never goes away.

Engine choice matters far less than that, which is the part vendors do not lead with. Kafka 4.0, in March 2025, was the first major release to run entirely without ZooKeeper, so a cluster is one process type instead of two, and that is the largest operational improvement in a decade. Redpanda speaks the same wire protocol from a single binary with Raft under each partition. Pulsar separates serving from storage. NATS is smaller in every dimension including what it promises. The managed Kafka endpoints from the cloud vendors move the pager, which is worth real money, but their semantics diverge at the edges in ways that will find you: retention ceilings tied to the tier you bought, compression codecs, transaction support that is not always generally available.

The gateway I looked at in Dongguan had firmware compiled in 2017 and no concept of a partition. It will outlive the backbone we put behind it, probably twice over, and each replacement will be written up in the proposal as a simplification.

Sources

Apache Kafka Downloads (release versions and dates, 4.3.1 released 25 June 2026) : https://kafka.apache.org/downloads

Apache Kafka 4.0.0 Release Announcement (first major release without ZooKeeper, KRaft by default) : https://kafka.apache.org/blog/2025/03/18/apache-kafka-4.0.0-release-announcement/

Apache Kafka 4.3 Design documentation (message delivery semantics, transactions, isolation levels) : https://kafka.apache.org/43/design/design/

Apache Kafka 4.3 Producer configuration reference (enable.idempotence, max.in.flight.requests.per.connection) : https://kafka.apache.org/43/configuration/producer-configs/

Apache Kafka 4.3 Broker configuration reference (log.retention.hours, offsets.retention.minutes, producer.id.expiration.ms, transaction timeouts) : https://kafka.apache.org/43/configuration/broker-configs/

Apache Kafka 4.3 Geo-Replication and Cross-Cluster Data Mirroring (MirrorMaker 2, edge clusters into a central aggregate cluster) : https://kafka.apache.org/43/operations/geo-replication-cross-cluster-data-mirroring/

Confluent Schema Registry: Schema Evolution and Compatibility Types (BACKWARD default, transitive modes) : https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html

NATS Documentation: Leaf Nodes (outbound-only bridging from a factory floor behind a firewall) : https://docs.nats.io/running-a-nats-service/configuration/leafnodes