Schema Registry

Available for all CloudKarafka customers without any setup

Schema Registry is a widely used solution for managing schemas of the data that you write to your Kafka topics.

A schema is used to ensure that the producer and consumer can understand each other, even if data changes over time. A schema informs your Apache Kafka clients about how the data should be formatted, which fields should be included in a hash map, and how these fields should appear (e.g. whether they should be strings or numbers).

The main use of Schema Registry

When you register a new schema under a subject, you get back an ID that you embed in your message. Then, the consumer reads the message, extracts the ID, and downloads the schema from Schema Registry.

Schema compatibility

When you register a new schema, you register it under a subject. To safely evolve your schemas, you can configure the compatibility level for that subject. If you don't configure a compatibility level, it will use the global level, which can also be changed and defaults to BACKWARD.

The compatibility will be checked based on the configured level for each new version of the schema that you register. If the new schema is not compatible with the configured level, it will not be allowed to register.

The Schema Registry has seven different compatibility levels:

NONE

don't check compatibility

BACKWARD

make sure the schema is backwards compatible with the previous version

BACKWARD_TRANSITIVE

make sure the schema is backwards compatible with all the previous versions

FORWARD

make sure the schema is forward compatible with the previous version

FORWARD_TRANSITIVE

make sure the schema is forward compatible with all the previous versions

FULL

make sure the schema is backward and forward compatible with the previous version

FULL_TRANSITIVE

make sure the schema is backward and forward compatible with all the previous versions

Backward compatibility

For a schema to be backward compatible means that the consumer will still be able to read the data even if the producer has changed the structure. For example, if you add a field to a hashmap, the reader will still be able to read and understand the data; they will just ignore the new field.

Using Schema Registry with CloudKarafka

Schema Registry is available for all our customers without any setup.
Schema Registry URL:
https://schemaregistry.cloudkarafka.com

Below, you can find some examples of how to use Schema Registry on CloudKarafka. Not all endpoints are listed as examples, but it does support all REST endpoints that other implementations of Schema Registry support.

Authentication

To access the Schema Registry you need to authenticate, the Kafka clients that has build-in support for Schema Registry also support authentication using Basic Auth so that is what we use. Using the same API KEY that you use to manage your cluster at CloudKarafka is used to authenticate to Schema Registry and you pass that API KEY in the password field in Basic Auth.

Examples

Register a new version with the subject "my-subject"

curl -X POST -H "Content-Type:
application/vnd.schemaregistry.v1+json" \
    -u :API-KEY \
    --data '{"schema": "{\"type\": \"string\"}"}' \

https://schemaregistry.cloudkarafka.com/subjects/my-subject/versions

List all your subjects

curl -u :API-KEY https://schemaregistry.cloudkarafka.com/subjects

Fetch a schema by id

curl -u :API-KEY https://schemaregistry.cloudkarafka.com/schemas/ids/1

Delete a version for subject "my-subject"

curl -X DELETE -u :API-KEY  https://schemaregistry.cloudkarafka.com/subjects/my-subject/versions/3

Test compatibility of a schema with the latest version of "my-subject" Delete a version for subject "my-subject"

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -u :API-KEY \
  --data '{"schema": "{\"type\": \"string\"}"}' \

https://schemaregistry.cloudkarafka.com/compatibility/subjects/my-subject/versions/latest

Get compatibility level for subject "my-subject" Delete a version for subject "my-subject"

curl -u :API-KEY https://schemaregistry.cloudkarafka.com/config/my-subject

Update compatibility level for subject "my-subject" Delete a version for subject "my-subject"

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -u :API-KEY \
  --data '{"compatibility": "BACKWARD"}' \

https://schemaregistry.cloudkarafka.com/config/my-subject