This guide will cover how to run Schema Registry on your server in
AWS using the hosted Kafka Cluster at Cloudkarafka.
You need a server running Ubuntu in your AWS account that you can access with
ssh. To run Schema Registry without memory issues the server needs to have at
least 1Gb of memory.
Create the Kafka cluster at cloudkarafka.com, make sure to select a subnet that doesn’t conflict with the subnet that your machines (in you account) is using.
See this Guide on how to set up VPC Peering connections Guide: VPC Peering
Schema Registry is part of the Confluent Platform and not available as standalone. So we'll go ahead and download the latest version of the Confluent Platform which is version 5.5.1. Download the tarball and extract into /opt
wget https://packages.confluent.io/archive/5.5/confluent-5.5.1-2.12.tar.gz
tar -xzvf confluent-5.5.1-2.12.tar.gz -C /opt
cd /opt/confluent-5.5.1
vim etc/schema-registry/schema-registry.properties
# /opt/confluent-5.5.1/etc/schema-registry/schema-registry.properties
listeners=http://0.0.0.0:8081
kafkastore.bootstrap.servers=PLAINTEXT://10.56.72.161:9092,PLAINTEXT://10.56.72.51:9092,PLAINTEXT://10.56.72.225:9092
kafkastore.topic=_schemas
debug=true
/opt/confluent-5.5.1/bin/schema-registry-start /opt/confluent-5.5.1/etc/schema-registry/schema-registry.properties
Run Schema Registry as a Systemd service for better reliability.
# /etc/systemd/system/schemaregistry.service
[Unit]
Description=Schema Registry
[Service]
Type=simple
PIDFile=/var/run/schemaregistry.pid
User=ubuntu
Group=ubuntu
ExecStart=/opt/confluent-5.5.1/bin/schema-registry-start /opt/confluent-5.5.1/etc/schema-registry/schema-registry.properties
ExecStop=/opt/confluent-5.5.1/bin/schema-registry-stop
Restart=on-failure
SyslogIdentifier=schemaregistry
[Install]
WantedBy=multi-user.target
Now enable the service and start it
sudo systemctl enable schemaregistry
sudo systemctl start schemaregistry
And now the service will start automatically every time the server is rebooted.
To check the status of the service
sudo systemctl status schemaregistry
Instead of having Schema Registry listen to http://0.0.0.0:8081 you can change that to http://127.0.0.1:8081 and you can put nginx in front. This allows you to use an encrypted connection and it also adds the possibility to use a custom path, for example, different port, subdomain or a custom path to reach Schema Registry. You can also add some security to the endpoint by configuring nginx to check for basic auth header which will then force the user to use username and password to access the http service. More about that here: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/