Kafka Schema Registry Migration Without Breaking the Wire Format

Almost every team running Kafka at scale has a schema registry in front of it. Far fewer have moved one in production without an outage, and the ones who tried usually learned the hard way that a registry migration is not a configuration export.

We get pulled into these projects after the first attempt has already stalled. The team treated the migration as a copy job: dump the schemas from the old registry, load them into the new one, repoint the clients, done by lunch. Then consumers across three services start throwing deserialization errors, the on-call channel lights up, and someone rolls back. The schemas copied fine. The problem was never the schemas.

Why a registry move is a stateful migration, not a config change

The contrarian framing we lead with on every one of these engagements: a Kafka schema registry migration is closer to a database migration than to a settings change, because the registry holds state that is embedded in your data at rest.

Here is the coupling most teams miss. When a producer serializes a record with the Confluent serializer, the bytes on the wire are not just the Avro or Protobuf payload. The serializer prepends a 5-byte header: one magic byte set to zero, then a 4-byte big-endian integer that is the schema ID assigned by the registry. The consumer reads that ID, calls the registry to fetch the matching schema, and only then can it deserialize. The Confluent wire format documents this layout precisely. That schema ID is now written into every message sitting in every topic, including messages with a retention of weeks or months.

So when you stand up a fresh registry, it assigns its own IDs starting from a low number. The schema your old registry called ID 4127 might become ID 1 in the new one. Every consumer reading a backlog of older messages will look up 4127, find nothing or find the wrong schema, and fail. The migration broke not because a schema was wrong, but because the identifier baked into your retained data no longer resolves.

What you are actually migrating

A schema registry holds three things that have to survive the move, and teams routinely account for only the first.

The schemas themselves are the easy part. They are versioned text, and any registry will accept them. The schema-to-ID mapping is the hard part, because those IDs are foreign keys held inside your message bytes. The third thing is the compatibility configuration: the global and per-subject compatibility level that governs which future schema changes the registry will accept. Confluent supports seven compatibility modes, from NONE through BACKWARD, FORWARD, and FULL, each with a transitive variant. A migration that resets every subject to the default BACKWARD level can silently start rejecting producer deployments that the old registry would have allowed, or worse, start accepting changes the old policy forbade.

Two more pieces of state deserve a mention because they cause quiet drift. Subject naming strategy determines how subjects map to topics: the default TopicNameStrategy ties one schema lineage to one topic, while RecordNameStrategy and TopicRecordNameStrategy allow multiple types per topic. If the source and target registries disagree on strategy, the same producer ends up writing to different subjects on each side. And in Confluent's implementation, all of this lives in a single compacted Kafka topic named _schemas, which means the registry's entire state is itself a Kafka log you can read, copy, or replay.

The counter-take on preserving IDs

Conventional migration advice says to re-register your schemas against the new registry and accept whatever IDs it hands out, then re-encode or reprocess old data if needed. We disagree for any topic with non-trivial retention.

The counter-take we offer is this: preserve the original schema IDs, do not regenerate them. Both Confluent's Schema Registry and AWS Glue support an import path that lets you set a schema with an explicit ID rather than an auto-assigned one. Confluent calls this IMPORT mode, and the registry must be empty and explicitly switched into that mode before you load schemas with their original IDs. AWS documents an analogous registry-level migration story for the Glue Schema Registry. Preserving IDs means a consumer reading a three-week-old message resolves the same ID to the same schema it always did. No reprocessing. No re-encoding. The wire-format contract your retained data depends on stays intact.

One case makes regenerating IDs acceptable: a topic whose retention is shorter than your migration window, where every message written under old IDs will have aged out before you cut over consumers. For those topics, and only those, the copy-and-repoint approach is safe. Knowing which topics qualify requires you to actually inventory retention, which is the step the failed first attempt skipped.

A migration sequence that holds

The sequence we run looks more like a dual-write database cutover than a config push. It assumes you cannot stop producers and cannot reprocess history.

First, inventory. List every subject, its current compatibility level, its naming strategy, and the retention of the topics it serves. This inventory is the artifact the whole migration plans against. Second, stand up the target registry empty and load the existing schemas with their original IDs preserved, using import mode. Verify by fetching a sample of IDs from both registries and diffing the returned schemas byte for byte.

Third, run both registries in parallel. Point the new registry at the same underlying state or keep it synchronized as new schemas register, so that a schema added during the migration window appears in both. Confluent's Schema Linking exists for exactly this continuous-sync case. Fourth, migrate consumers before producers. Consumers only read schema IDs; a consumer pointed at the new registry will resolve IDs identically if the IDs were preserved, so this step is low risk and builds confidence. Fifth, migrate producers. A producer writing to the new registry may register a new schema and receive a new ID; because consumers are already reading from the same synchronized registry, that ID resolves for them immediately. Sixth, decommission the old registry only after a full retention period has elapsed, so that no consumer can still be reading messages whose IDs originated on the old side.

The ordering matters. We have watched teams migrate producers first, which strands consumers still pointed at a registry that never received the producer's newly registered schema, and the symptom is an intermittent deserialization failure that only shows up for the newest messages. Consumers first inverts that risk.

What goes wrong, and where

Three failure modes recur often enough that we now check for them before writing a plan.

ID collisions are the first. If the target registry is not empty when you import, an incoming schema with explicit ID 4127 can collide with a schema the new registry already auto-assigned to 4127. Import mode mitigates this only on a registry that started empty. Mixing an import into a live registry is the fastest way to corrupt the ID space.

The second is compatibility drift. A migration that does not copy per-subject compatibility overrides falls back to the global default, and the first producer to ship a schema change discovers the new policy the hard way. We copy compatibility settings as deliberately as we copy schemas, and we diff them post-migration.

The third is the dual-registry consistency gap. During the parallel-run window, any schema registered against only one side creates a split brain: the ID exists on one registry and not the other. Without continuous synchronization or a hard freeze on schema registration during the window, this gap is almost guaranteed on a busy platform. A short registration freeze, measured in hours, is often cheaper than building synchronization for a one-time move.

Tooling reality

The tooling has improved, but none of it removes the need to understand the wire format. Confluent's Schema Linking and the export/import API cover the Confluent-to-Confluent case well. Karapace, the open-source registry from Aiven, is wire-compatible with Confluent's REST API and can act as a drop-in target, which makes a Confluent-to-Karapace move tractable without rewriting client code. AWS Glue Schema Registry uses a different client and serializer, so a move onto Glue is a client-library migration as well as a state migration, and the two should be sequenced separately rather than attempted at once.

Whatever you choose, the registry is not the risky part. The risky part is the several terabytes of already-serialized messages sitting in your topics, each carrying a schema ID that has to keep resolving. Pick the tool after you have mapped that exposure, not before.

What to do this quarter

If a schema registry migration is on your roadmap, four moves de-risk it before you touch a client config.

1. Inventory every subject, its compatibility level, its naming strategy, and the retention of the topics it feeds. Treat any topic with retention longer than your migration window as ID-preservation-mandatory.
2. Decide ID preservation per topic, not globally. Preserve IDs for long-retention topics; allow regeneration only where every old message will age out before consumer cutover.
3. Prove the import on a copy first. Load schemas with original IDs into an empty target, then diff a sample of IDs byte for byte against the source before any client moves.
4. Sequence consumers ahead of producers, run both registries in parallel through a full retention period, and decommission the old registry last.

Get the ID mapping and the cutover order right, and a registry migration becomes a quiet, boring change. Get either wrong, and the schemas will copy perfectly while your consumers fail on data you migrated months ago.