Kafka: Difference between revisions

From Halfface
Jump to navigation Jump to search
 
(15 intermediate revisions by the same user not shown)
Line 2: Line 2:
  akhq            Apache Kafka HQ. web-based user interface and management tool for Apache Kafka clusters.
  akhq            Apache Kafka HQ. web-based user interface and management tool for Apache Kafka clusters.
  strimzi          Apache Kafka on Kubernetes
  strimzi          Apache Kafka on Kubernetes
cdc              change data capture
=using kafka client=
==List topics==
kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list
==Describe topic==
kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --describe --topic <topic>
=Read topic=
export TOPIC=<topic> ; kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-console-consumer.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --topic ${TOPIC} --property print.key=true --max-messages 10 --from-beginning --timeout-ms 10000
==Describe all topics==
kubectl exec -i -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list | while read i ; do echo '*' $i ; kubectl exec -i -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --describe --topic $i </dev/null ; done
=consumergroups=
List consumer groups
kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list


=quick overview=
=quick overview=
  kubectl api-resources | grep -i kafka | awk '{print $1}' | while read i ; do echo '*' $i ; kubectl get $i -A ; done
  kubectl api-resources | grep -i kafka | awk '{print $1}' | while read i ; do echo '*' $i ; kubectl get $i -A ; done
=which users exit=
=which users exit=
  kubectl get kafkausers -n kafka
  kubectl get kafkausers -n kafka
Line 12: Line 28:
=list topics=
=list topics=
  kubectl get kafkatopics -o wide
  kubectl get kafkatopics -o wide
=Kafka version=
kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --version
=bootstrap server internally=
kubectl get kafka kafka-cluster -o=jsonpath='{.status.listeners[?(@.type=="tls")].bootstrapServers}{"\n"}'
=bootstrap sever external=
kubectl get kafka kafka-cluster -o=jsonpath='{.status.listeners[?(@.type=="external")].bootstrapServers}{"\n"}'
=produce to topic=
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic <topic>
Produce to topic with key.
kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bash
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic <topic> --property "parse.key=true" --property "key.separator=:"
>emp_info: {"emp_id":100,"first_name":"Keshav","last_name":"Lodhi","designation":"DataEngineer"}
CTRL + c
kubectl exec -it -n kafka kafka-cluster-kafka-2 -- bin/kafka-console-consumer.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --topic customer-blacklistmail-dotnet-v1 --property print.key=true --max-messages 10 --from-beginning
=read topic=
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FedoraTopic --from-beginning
=zookeeper=
Who is the leader.
for i in 0 1 2  ; do echo kafka-cluster-zookeeper-$i $(kubectl exec -it -n kafka kafka-cluster-zookeeper-$i -- bash -c 'echo stat | nc 127.0.0.1 12181' | grep Mode:) ; done
Get various info from zookeeper.
kubectl exec -it -n kafka kafka-cluster-zookeeper-0 -- bash -c 'echo envi | nc 127.0.0.1 12181'
The following commands are availablew.
conf Print details about serving configuration.
cons List full connection/session details for all clients connected to this server. Includes information on numbers of packets
received/sent, session id, operation latencies, last operation performed, etc...
crst Reset connection/session statistics for all connections.
dump Lists the outstanding sessions and ephemeral nodes. This only works on the leader.
envi Print details about serving environment
ruok respond with imok if it is running.
srst Reset server statistics.
srvr Lists full details for the server.
stat Lists brief details for the server and connected clients.
wchs Lists brief information on watches for the server.
wchc Lists detailed information on watches for the server, by session
wchp Lists detailed information on watches for the server, by path.
mntr Outputs a list of variables that could be used for monitoring the health of the cluster.
=kafka-exporter=
Get stats from kafka-exporter.
kubectl -it exec deployment/kafka-cluster-kafka-exporter -- curl -s localhost:9404/metrics
=connectors=
These are the actual plugins that Kafka Connect uses to interface with external systems. Connectors can be either sources (pulling data into Kafka) or sinks (pushing data from Kafka). (io.debezium.connector.postgresql.PostgresConnector)
=Kafka Connect=
Is a framework and set of tools for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Latest revision as of 11:40, 6 September 2024

What does it mean:

akhq             Apache Kafka HQ. web-based user interface and management tool for Apache Kafka clusters.
strimzi          Apache Kafka on Kubernetes
cdc              change data capture

using kafka client

List topics

kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list

Describe topic

kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --describe --topic <topic>

Read topic

export TOPIC=<topic> ; kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-console-consumer.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --topic ${TOPIC} --property print.key=true --max-messages 10 --from-beginning --timeout-ms 10000

Describe all topics

kubectl exec -i -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list | while read i ; do echo '*' $i ; kubectl exec -i -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --describe --topic $i </dev/null ; done

consumergroups

List consumer groups

kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-consumer-groups.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --list

quick overview

kubectl api-resources | grep -i kafka | awk '{print $1}' | while read i ; do echo '*' $i ; kubectl get $i -A ; done

which users exit

kubectl get kafkausers -n kafka

which password does user have

kubectl get secret ifs -o json | jq -r .data.password | base64 -d

list topics

kubectl get kafkatopics -o wide

Kafka version

kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bin/kafka-topics.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --version

bootstrap server internally

kubectl get kafka kafka-cluster -o=jsonpath='{.status.listeners[?(@.type=="tls")].bootstrapServers}{"\n"}'

bootstrap sever external

kubectl get kafka kafka-cluster -o=jsonpath='{.status.listeners[?(@.type=="external")].bootstrapServers}{"\n"}'

produce to topic

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic <topic>

Produce to topic with key.

kubectl exec -it -n kafka kafka-cluster-kafka-0 -- bash
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic <topic> --property "parse.key=true" --property "key.separator=:"
>emp_info: {"emp_id":100,"first_name":"Keshav","last_name":"Lodhi","designation":"DataEngineer"}
CTRL + c
kubectl exec -it -n kafka kafka-cluster-kafka-2 -- bin/kafka-console-consumer.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --topic customer-blacklistmail-dotnet-v1 --property print.key=true --max-messages 10 --from-beginning

read topic

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FedoraTopic --from-beginning

zookeeper

Who is the leader.

for i in 0 1 2  ; do echo kafka-cluster-zookeeper-$i $(kubectl exec -it -n kafka kafka-cluster-zookeeper-$i -- bash -c 'echo stat | nc 127.0.0.1 12181' | grep Mode:) ; done

Get various info from zookeeper.

kubectl exec -it -n kafka kafka-cluster-zookeeper-0 -- bash -c 'echo envi | nc 127.0.0.1 12181'

The following commands are availablew.

conf Print details about serving configuration.
cons List full connection/session details for all clients connected to this server. Includes information on numbers of packets 
received/sent, session id, operation latencies, last operation performed, etc...
crst Reset connection/session statistics for all connections.
dump Lists the outstanding sessions and ephemeral nodes. This only works on the leader.
envi Print details about serving environment
ruok respond with imok if it is running.
srst Reset server statistics.
srvr Lists full details for the server.
stat Lists brief details for the server and connected clients.
wchs Lists brief information on watches for the server.
wchc Lists detailed information on watches for the server, by session
wchp Lists detailed information on watches for the server, by path.
mntr Outputs a list of variables that could be used for monitoring the health of the cluster.

kafka-exporter

Get stats from kafka-exporter.

kubectl -it exec deployment/kafka-cluster-kafka-exporter -- curl -s localhost:9404/metrics

connectors

These are the actual plugins that Kafka Connect uses to interface with external systems. Connectors can be either sources (pulling data into Kafka) or sinks (pushing data from Kafka). (io.debezium.connector.postgresql.PostgresConnector)

Kafka Connect

Is a framework and set of tools for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems.