Guide: Kafka Rest Proxy

This guide will cover how to run Kafka Rest Proxy on your server in AWS using the hosted Kafka Cluster at Cloudkarafka.
You need a server running Ubuntu in your AWS account that you can access with ssh. To run Kafka Rest Proxy without memory issues the server needs to have at least 1Gb of memory.

Create a Kafka cluster

Create the Kafka cluster at cloudkarafka.com, make sure to select a subnet that doesn’t conflict with the subnet that your machines (in you account) is using.

Setup VPC peering

See this Guide on how to set up VPC Peering connections Guide: VPC Peering

Download

Rest Proxy is part of the Confluent Platform and not available as standalone. So we'll go ahead and download the latest version of the Confluent Platform which is version 5.5.1.

wget https://packages.confluent.io/archive/5.5/confluent-5.5.1-2.12.tar.gz
tar -xzvf confluent-5.5.1-2.12.tar.gz -C /opt

Configure

# /opt/confluent-5.5.1/etc/kafka-rest/kafka-rest.properties

listeners=http://0.0.0.0:8082
bootstrap.servers=PLAINTEXT://bootstrap.servers=10.56.72.161:9092,PLAINTEXT://10.56.72.51:9092,PLAINTEXT://10.56.72.225:9092

Run

/opt/confluent-5.5.1/bin/kafka-rest-start /opt/confluent-5.5.1/etc/kafka-rest/kafka-rest.properties

Run with systemd

Run Rest Proxy as a Systemd service for better reliability.

# /etc/systemd/system/kafkarestproxy.service

[Unit]
Description=Kafka Rest Proxy

[Service]
Type=simple
PIDFile=/var/run/kafkarest.pid
User=ubuntu
Group=ubuntu
ExecStart=/opt/confluent-5.5.1/bin/kafka-rest-start /opt/confluent-5.5.1/etc/kafka-rest/kafka-rest.properties
ExecStop=/opt/confluent-5.5.1/bin/kafka-rest-stop
Restart=on-failure
SyslogIdentifier=kafkarest

[Install]
WantedBy=multi-user.target

Now enable the service and start it

sudo systemctl enable kafkarestproxy
sudo systemctl start kafkarestproxy

And now the service will start automatically every time the server is rebooted.

To check the status of the service

sudo systemctl status kafkarestproxy

Use nginx as proxy

Instead of having Kafka Rest Proxy listen to http://0.0.0.0:8082 you can change this to http://127.0.0.1:8082 and put nginx in front. This allows you to use an encrypted connection and it also adds the possibility to use a custom path, for example, different port, subdomain or a custom path.

Here’s a sample snippet on how to configure a location for nginx:

location = /rest {
    return 302 /rest/;
}
location /rest/ {
    gzip on;
    gzip_types application/json;
    auth_basic "Authentication required";
    auth_basic_user_file /opt/.htpasswd;
    proxy_pass http://127.0.0.1:8082/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    add_header X-Content-Type-Options "nosniff";
    add_header Strict-Transport-Security "max-age=631138519";
}

You can also add some security to the endpoint by configuring nginx to check for basic auth header which will then force the user to use username and password to access the http service. More about that here: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/