The Protobuf Schema Registry as the Single Source of Truth

Every service you run already has a contract. The only question is where it lives. In a wiki page nobody has opened since the author left. In the request handler of whichever service happened to be written first. In a shared types package that three teams edit and one team owns. A protobuf schema registry moves that contract into a file the build can read, and that relocation is the whole argument. Not the binary encoding. Not the wire size. The fact that the contract stops being a description of the system and becomes an input to it.

I have worked both ways. Without a schema file the failures are quiet. Someone widens a field from an enum to a free string on the producer, a consumer three hops away keeps switching on the old values, and the bug surfaces as a null in a dashboard owned by people who have never read the producer's code. With a schema under change control the same edit fails on the pull request that made it. That is the only behavioural difference worth paying for, and it is worth being precise about which part of the toolchain produces it, because most of the toolchain does not.

The service written first becomes the spec by accident

Contracts do not go missing. They accumulate in the wrong place. The most common wrong place is the first service somebody wrote, because every later service was built by reading its responses and matching them. Its field names become the vocabulary. Its optionality rules become the validation. Its accidents, like emitting an empty string where it means unknown, become load-bearing.

Documentation loses to this because nothing checks it. A README describing a payload is a claim about the system with no mechanism attaching it to the system. It rots at exactly the rate the system changes, which is fastest during the period the document matters most.

The distinction that matters is not file format. It is direction. If you generate OpenAPI from your Go structs or your Java annotations, the code is still the source of truth and the spec is a report about it. Nothing prevents the report from being regenerated to match whatever the code now does, which is precisely the change you wanted to catch. A schema file earns the title only when it is upstream: the code is generated from it, so editing the schema is the only way to change the payload, and an edit that breaks a consumer shows up as a diff on one file. Protobuf is a good format for this, but the reason to reach for it is that its tooling assumes that direction and enforces it.

buf breaking compares you to your previous self. It has no opinion about whether your previous self was right.

What generated types actually remove

Run buf generate against a proto module and you get types in Go, TypeScript, Java, Python and the rest from one file. Configuration moves out of protoc's flag soup into buf.gen.yaml, where each protoc --<lang>_out and --<lang>_opt pair becomes a plugin entry. You do not have to restructure an existing tree to start: the -I paths you were already passing become module entries in buf.yaml, and buf build -o writes a binary image that is wire-compatible with a FileDescriptorSet if something downstream still wants one.

The category of bug that disappears is the hand-written data transfer object. Every language boundary in a JSON system has one, each is a human transcription of a payload shape and each drifts independently. The classic version is casing. The canonical ProtoJSON mapping emits field names in lowerCamelCase unless json_name says otherwise, and parsers accept both the lowerCamelCase name and the original proto field name. So a hand-written client that sends snake_case works, a hand-written client that expects snake_case back from a compliant serialiser does not, and nobody finds out until the serialiser version changes.

There is a sharper asymmetry underneath. Binary Protobuf propagates unknown fields through a parse and re-serialise cycle. The ProtoJSON parser is specified to reject unknown fields by default, with an option to ignore them. If you have a proxy or an enrichment service that deserialises and re-emits, it preserves data silently in one encoding and either drops it or hard-fails in the other. That one is difficult to find because both services look correct in isolation.

This is also where a coding assistant quietly makes things worse. GitHub Copilot or Cursor will produce a plausible client struct on request, and it will be plausible in the way that hurts most: right field names, wrong optionality, wrong casing on the two fields somebody renamed last quarter. Generated code cannot be plausible. It either matches the schema or it does not compile.

buf lint is taste, buf breaking is enforcement

The Buf CLI (v1.72.0 at the time of writing) bundles several things that get discussed as one thing. buf lint runs more than forty style and correctness rules, grouped into MINIMAL, BASIC and STANDARD, with STANDARD applied by default when no lint section appears in buf.yaml, plus two non-hierarchical categories, COMMENTS and UNARY_RPC. buf format rewrites to a canonical style. buf push publishes a module to a registry. None of those change anybody's behaviour.

buf breaking does. It compares the current schema against a past version, which can be a Git branch or tag, a tarball, a prebuilt image or a module in a registry. The working invocation is buf breaking --against '.git#branch=main', run on every pull request. Its fifty-odd rules sit in four categories forming a strictness hierarchy: FILE, PACKAGE, WIRE_JSON and WIRE. Passing a stricter category implies passing every looser one.

Choosing the category is a statement about who your consumers are, and it is the one configuration decision worth arguing about. FILE detects breakage to generated source per file and is the default, which makes it too strict for a lot of teams, because moving a message between files in the same package trips it even though nothing on the wire moved. PACKAGE gives the same guarantee at package granularity and is usually what an internal platform actually needs. WIRE_JSON covers the binary format and the JSON encoding. WIRE covers only the binary format and is correct only if nobody compiles your generated code and nobody speaks JSON, which is a stronger claim than most people mean when they select it.

The part that changes behaviour is that the check fails the build. A lint job that posts a warning is theatre. The official Buf GitHub Action comments inline on the offending lines of the .proto file, which is good, and its default workflow also honours a buf skip breaking label on the pull request, which is honest of them and dangerous for you. That label is a fine escape hatch for a deliberate, coordinated break. It becomes the normal way to merge faster than anyone expects if nobody watches the count. The failure mode I keep meeting is not a team without breaking-change detection. It is a team with breaking-change detection and a habit of skipping it.

Field numbers are the API, names are decoration

The wire format identifies fields by number, not by name. Valid numbers run from 1 to 536,870,911, with 19,000 through 19,999 reserved for the Protobuf implementation itself, and numbers 1 through 15 encode their tag in a single byte. That last detail is a real design constraint. The fifteen cheapest slots in a hot message belong to fields present on every message, not to the debug metadata somebody added first because it was convenient.

Because the number identifies the field, it cannot change once the message type is in use. The official guidance treats renumbering as identical to deleting the field and adding a different one, and treats reuse of a retired number as a deserialisation hazard rather than a style preference. The reason is unglamorous. Old messages exist. They sit in log archives, in a topic with long retention, in a dead letter queue somebody will replay next quarter, in the flash of a device you shipped and cannot reach. Confirming that nothing currently sends field 7 tells you nothing about what was sent last March.

So you reserve, with reserved 2, 15, 9 to 11; for numbers and reserved "foo", "bar"; for names, and those statements accumulate forever. Buf enforces the discipline directly. FIELD_NO_DELETE_UNLESS_NUMBER_RESERVED, in the WIRE and WIRE_JSON categories, permits deleting a field only when its number is reserved. FIELD_NO_DELETE_UNLESS_NAME_RESERVED does the same for names under WIRE_JSON, because JSON identifies fields by name. RESERVED_MESSAGE_NO_DELETE stops somebody tidying the accumulated reserved lines away once they get long, which they will. FIELD_SAME_TYPE, FIELD_SAME_CARDINALITY and FIELD_SAME_JSON_NAME cover the rest of the ways a field changes identity without changing its number.

The compatible type changes are the part people half-remember and get wrong. int32, uint32, int64, uint64 and bool are mutually compatible on the wire, but a 64-bit value parsed as int32 truncates exactly as a C++ cast would, so widening is safe and narrowing corrupts quietly. sint32 and sint64 are compatible with each other and with nothing else, and an out-of-range sint64 read as sint32 truncates and then zigzag-decodes into a value that is not merely truncated but different. string and bytes are compatible while the bytes are valid UTF-8. fixed32 pairs with sfixed32, fixed64 with sfixed64. enum is compatible with the integer types. Singular and repeated of the same scalar look interchangeable and are not for numerics, bools or enums, because repeated fields of those types serialise packed by default and a packed field will not parse correctly into a singular one.

required did not survive into proto3, and the reasoning generalises past Protobuf. A required field is a permanent constraint imposed by whoever wrote the message first, enforced in every reader that ever parses it, and unremovable without breaking writers already deployed. You cannot know how long a message type will live, or who will eventually be forced to fill your required field with an empty string to get past your validator. The current guidance is to document the requirement in a comment and enforce it in the application layer. Protovalidate, now at v1.0, is the standard way to do that: field options for the common constraints and CEL expressions for anything cross-field, evaluated identically across Go, TypeScript, Java, Python and C++. Semantic validation belongs there. It does not belong in the wire format, where it is load-bearing forever. Editions change how you spell these choices, with edition 2023 generally available from protoc 27.0 and edition 2024 landing in the v32 line, replacing the proto2 and proto3 labels with per-feature defaults. They do not change the choices.

Migrating when JSON over HTTP is already everywhere

Almost every team arriving at this has a working system, which constrains the migration more than the technology does. The order that works is not the order people pick.

Start by describing what you already send. Write .proto files for the existing payloads and do not clean them up while you write them. If a field is called usr_id and holds a string that is sometimes numeric, model that. The urge to fix the schema and the transport in one change is where these migrations die, because afterwards you cannot tell a serialisation bug from a semantics bug.

Then wire buf lint and buf breaking into CI, before a single byte of traffic changes. This is the step teams skip and the step that pays. From that commit onward the shape of your payloads is under review, and you get the benefit whether or not you ever ship binary encoding. A proto file plus buf breaking in CI is a strictly better JSON contract than an OpenAPI document nobody diffs.

Generated types go in next, on one consumer, still speaking JSON. Here you pin json_name on every field whose existing wire name is not the lowerCamelCase form of the proto field name, which will be most of them if your API is snake_case. FIELD_SAME_JSON_NAME then guards what you just pinned. Transport comes last and is often optional. Connect is the pragmatic step because it serves application/proto and application/json from the same endpoint over plain HTTP/1.1 or HTTP/2, so binary becomes content type negotiation rather than a rewrite, and the curl habit your on-call rota depends on survives. The version I would not ship again is the reverse of this: transport first, gRPC everywhere, then discovering what your load balancers do with trailers, what your tracing does with a binary body and what your browser clients cannot do at all. Teams who go that way conclude Protobuf was the mistake. The mistake was doing the hardest and least valuable part first.

The case for stopping short of a protobuf schema registry

For one team, one language and one deployable, this is real overhead against benefits you will never collect. Say that plainly. Your compiler already gives you the guarantee the registry is selling. A change to a Go struct breaks compilation at every call site in the same build. There is no cross-language drift because there is no second language, and no version skew because there is one deployable.

The costs are not only conceptual. buf.yaml, buf.gen.yaml and buf.lock to maintain. A generation step in every build and on every developer machine. Generated code either committed, where it produces enormous diffs nobody reads, or produced in CI, where plugin version drift yields a build that fails only for the person who pulled last. A registry to host or pay for. And a review culture that understands why a renumbered field is a production incident, which is the expensive part and the part no tool supplies.

OpenAPI is a legitimate stopping point and the enforcement mechanism transfers cleanly. OpenAPI 3.1 aligns with JSON Schema, and oasdiff diffs two specs, classifies breaking changes across paths, parameters, request bodies, responses, headers and security, and exits non-zero when it finds one. That is the same CI gate, on a format your HTTP stack already speaks, against a documented rule set in the hundreds. Run it as a GitHub Action against the spec on main and you have most of the behavioural change described here.

One condition attaches. Write the OpenAPI document first and generate from it. A spec produced from code annotations is a report, and diffing two reports tells you the code changed, which you knew. The threshold I would use, and it is a threshold rather than a rule: more than one language reading the same payload, or a payload that outlives the process that produced it. A message on a queue, an event in a log with long retention, a firmware image on a machine in a plant that will not be reflashed for three years. Below that line, OpenAPI and discipline. Above it, the wire rules stop being pedantry and start being the reason the system still parses.

What breaking-change detection cannot do

buf breaking compares you to your previous self. It has no opinion about whether your previous self was right.

Turn it on and every early mistake becomes permanent at the moment you turn it on. The field that should have been an enum. The enum whose zero value means something instead of meaning unset. The message that should have been two messages. The tool converts design debt into a fixed cost you pay in perpetuity, and it does that on day one, reliably before anybody understands the domain well enough to have designed it correctly.

The standard answer is the package version suffix, which Buf's STANDARD lint category enforces, so your package is acme.orders.v1alpha1 and you are free to break it. That only works while v1alpha1 is genuinely disposable, and it stops being disposable the first time an external consumer integrates against it, which usually happens before you have decided it is stable. So two true statements sit against each other. The right time to add the registry is after the schema settles, and the schema does not settle until something stops people changing it. I do not think that resolves. You pick which failure you would rather have. A schema registry does not make your schema good, it makes it expensive to change, and those are the same thing only if you were right the first time. Field number 3 in the first message you write will still be field number 3 long after everyone who chose it has left, still carrying whatever you put there on the afternoon you were mostly thinking about something else.

Tools referenced

GitHub Copilot, reviewed here: GitHub Copilot review.

Cursor, reviewed here: Cursor review.

Sources

Protocol Buffers, Language Guide (proto 3): field numbers, reserved, updating a message type: https://protobuf.dev/programming-guides/proto3/

Protocol Buffers, Proto Best Practices: never reuse a tag number, do not add required fields: https://protobuf.dev/best-practices/dos-donts/

Protocol Buffers, ProtoJSON format: lowerCamelCase mapping, json_name, unknown field handling: https://protobuf.dev/programming-guides/json/

Buf Docs, Detecting breaking changes: FILE, PACKAGE, WIRE_JSON and WIRE categories: https://buf.build/docs/breaking/

Buf Docs, Breaking rules reference: FIELD_NO_DELETE_UNLESS_NUMBER_RESERVED and related rules: https://buf.build/docs/breaking/rules/

Buf Docs, Lint overview: MINIMAL, BASIC, STANDARD, COMMENTS and UNARY_RPC: https://buf.build/docs/lint/

Connect Protocol Reference: application/proto and application/json on one endpoint: https://connectrpc.com/docs/protocol/

oasdiff, OpenAPI breaking changes rule set and CI exit codes: https://www.oasdiff.com/docs/breaking-changes