Configuring Apache Kafka with Hyperledger Fabric 1.2 on Multiple Hosts

Introduction

In this article, I am going to integrate apache Kafka with Hyperledger Fabric 1.2 and use Kafka-manager to monitor installed Kafka brokers and topics.

Topology

Below topology will be used as a reference throughout the document. However, associated scripts can be configured to work with any topology.

  • Two Organizations each with 2 peers

Deployment Topology

Host-1

  • Orderer0 (orderer0.example.com) -port 7050

Host-2

  • CA1 (ca.org1.example.com) -port 7054

Host-3

  • CA2 (ca.org2.example.com) -port 7054

Host-4

  • Kafka3 (kafka3.example.com) — Port 9092

Make sure above specified ports are not blocked with firewall. Open 2377,7946,4789 ports for docker swarm.

Prerequisites

Follow this link to install all the dependencies on each host.

List of components that I used

  • Ubuntu-16.04

Fabric 1.2 Setup with Apache Kafka

Execute below steps 1–3 on all hosts (host-1,2,3,&4)

1) Clone Hyperledger fabric-samples

2) go to fabric-samples directory

3) clone Fabric Multi Network source code from

https://github.com/mallikprojects/fabric-multi-network.git

git clone -b release-1.2 https://github.com/hyperledger/fabric-samplescd fabric-samplescurl -sSL http://bit.ly/2ysbOFE | bash -s 1.2.0git clone -b fabric-1.2-kafka-couch-support https://github.com/mallikprojects/fabric-multi-network.gitcd fabric-multi-network

4) configure environmental variables in bymn.sh and ensure bymn.sh is same across all hosts

  • export ORDERER0_HOSTNAME=<host name of host-1>

To get hostname run the following command

hostname

5) Fabric configuration for Apache kafka

set OrderType to kafka in configtx.yaml

OrdererType: kafka

update configtx.yaml with kafka brokers

Brokers:- kafka0.example.com:9092- kafka1.example.com:9092- kafka2.example.com:9092- kafka3.example.com:9092

Rename kafka_scripts_bkup to kafka_scripts and zk_scripts_bkup to zk_scripts

mv kafka_scripts_bkup kafka_scripts
mv zk_scripts_bkup zk_scripts

Update Kafka brokers in case of any addition/removal of brokers.

6) Generate crypto data, docker compose templates and copy to all the hosts

./bymn.sh generate crypto-config.yaml//if you want to use couchDB as state database use./bymn.sh generate -s couchdb crypto-config.yaml

crypto-config — copy to all hosts

channel-artifacts — copy to all hosts

6) Create & setup Swarm

On host-1

docker swarm init  --advertise-addr <PC-1 IP address>docker swarm join-token manager

Output of this command should be executed on host-2, host-3 & host-4 as follows

docker swarm join — token SWMTKN-1–3anjn4oxwcn278hie3413zaakr4higjdqr2x89r5605p1dosui-a4u407pt6c5ta2ont7pqdnm 137.116.147.36:2377 –advertise-addr <host-2 Ip Address>docker swarm join — token SWMTKN-1–3anjn4oxwcn278hie3413zaakr4higjdqr2x89r5605p1dosui-a4u407pt6c5ta2ont7pqdnm 137.116.147.36:2377 –advertise-addr <host-3 Ip Address>docker swarm join — token SWMTKN-1–3anjn4oxwcn278hie3413zaakr4higjdqr2x89r5605p1dosui-a4u407pt6c5ta2ont7pqdnm 137.116.147.36:2377 –advertise-addr <host-4 Ip Address>

Replace token with the token you got in the previous command response

7) Create an overlay network

on host-1

docker network create --attachable  --driver overlay fabric

8) Start Fabric network with kafka

on host-1

./bymn.sh up

Check the logs and make sure service is up.

9) Test your network

To test network on PC-2 open cli

docker exec -it <org2cli docker container id> bash
$ ./scripts/script.sh

If you see above success message in your console, then network setup is proper and is running correctly

Install and configure Kafka-Manager to manage installed Kafka brokers

This section lists the steps required to Install Kafka manager on host-4. Kafka-manager uses port 9000. Make sure this port is opened

Reference: https://github.com/yahoo/kafka-manager

on host-4 run the following commands

1)cd ~2) git clone https://github.com/yahoo/kafka-manager.git3) cd kafka-manager4) sudo apt-get update5) sudo apt install default-jdk6) vim conf/application.confconfigure zookeeper settingskafka-manager.zkhosts="<ip address of zookeepr1>:2181,<ip address of zookeepr2>:2181,<ip address of zookeepr3>:2181"close file$ ./sbt clean dist

After extracting the produced zip file, and changing the working directory to it, you can run the service like this:

bin/kafka-manager

This will start kafka-manager on port 9000

This completes the setup of Kafka Manager

Open browser and open http://<host4 -ipadress>:9000/ and add cluster with <ip address of zookeepr1>:2181,<ip address of zookeepr2>:2181,<ip address of zookeepr3>:2181.

On successful creation, you should be able to see cluster details with list of channels created in fabric as below

Success. Your Kafka manager is configured with Fabric network

Technology Entrepreneur ,Architect, Full stack engineer