CMAK（ Formerly known as Kafka Manager） It's a way to manage Apache Kafka Clustering tools . For more information about name changes , See below .

CMAK Support the following ：

• Manage multiple clusters
• Easily check cluster status （ The theme 、 consumer 、 Offset 、 agent 、 Copy distribution 、 Partition distribution ）
• Run preferred replica election
• Generate partition assignments using the option to select the agent to use
• Running partition reallocation （ Based on the generated allocation ）
• Use the optional theme configuration to create a theme （0.8.1.1 And 0.8.2+ There are different configurations ）
• Delete theme （ Support only 0.8.2+ And remember to match... In the agent ​​ Centering setting delete.topic.enable=true）
• The topic list now indicates topics marked for deletion （ Support only 0.8.2+）
• Batch generate partition allocation of multiple topics , You can choose which proxy to use
• Batch run partition reassignment of multiple topics
• Add a section to an existing theme
• Update the configuration of the existing theme
• （ Optional ） Enable... For agent level and subject level metrics JMX polling .
• Optionally filter out zookeeper There is no ids/owners/&offsets/ Catalog consumers .

1. Cluster management

2. List of topics

3. Theme view

4. Consumer list view

5. Consumption theme view

6. Broker List

7.Broker Exhibition

requirement

Kafka 0.8.. or 0.9.. or 0.10.. or 0.11..
Java 11+

To configure

The minimum configuration is for CMAK（pka kafka manager） State of zookeeper host . This can be done in conf In the directory application.conf Found in file . The same files will be packaged in the distribution zip In file ; You can unzip the file on the desired server and modify the settings .

cmak.zkhosts="my.zookeeper.host.com:2181"

You can specify multiple by separating them with commas zookeeper host , As shown below ：

cmak.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181"

perhaps ,ZK_HOSTS If you don't want to hard code any values , Please use environment variable .

ZK_HOSTS="my.zookeeper.host.com:2181"

You can choose to modify application.conf The default list in the / Disable the following functions ：

application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]

• KMClusterManagerFeature - Allow from CMAK（pka Kafka Manager） add to 、 to update 、 Delete cluster
• KMTopicManagerFeature - Allow from Kafka Cluster add 、 to update 、 Delete theme
• KMPreferredReplicaElectionFeature - Allow for Kafka The cluster runs the preferred replica
• KMReassignPartitionsFeature - Allows generation of partition assignments and reallocation of partitions

Consider enabling jmx Set these parameters for larger clusters ：

cmak.broker-view-thread-pool-size=<3 * number_of_brokers>
cmak.broker-view-max-queue-size=<3 * Total number of partitions for all topics >
cmak.broker-view-update-seconds=< cmak.broker-view-max-queue-size / (10 * number_of_brokers) >

Here's what has 10 A broker 、100 A theme kafka An example of a cluster , Each theme has 10 Zones , In the activation of JMX A total of 1000 Zones ：

cmak.broker-view-thread-pool-size=30
cmak.broker-view-max-queue-size=3000
cmak.broker-view-update-seconds=30

The following controls the thread pool and queue of the consumer offset cache ：

cmak.offset-cache-thread-pool-size=< The default is the number of processors >
cmak.offset-cache-max-queue-size=< The default value is 1000>
cmak.kafka-admin-client-thread-pool-size=< The default is the number of processors >
cmak.kafka-admin-client-max-queue-size=< The default value is 1000>

For a large number of consumers with consumer polling enabled , You should increase the above value . Although its main impact is based on ZK Consumer voting .

Kafka Managed consumer offsets are now managed by “__consumer_offsets” In the theme KafkaManagedOffsetCache consumption . Please note that , This has not been tested to track large offsets . One thread per cluster uses this topic , Therefore, it may not be able to keep up with the large number of offsets pushed to the topic .

Use LDAP Validate users
Warning , You need to use CMAK（pka Kafka Manager） To configure SSL, To ensure that your credentials are not passed unencrypted . By associating user credentials with Authorization The header is passed together , have access to LDAP Authenticate users .LDAP Authentication is completed on the first access , If it works , Is set cookie. In the next request ,cookie Value and Authorization Compare the credentials in the header .LDAP Support is achieved through basic authentication filters .

Configure basic authentication

basicAuthentication.enabled=true
basicAuthentication.realm=< basic authentication realm>

Encryption parameter （ Optional , Otherwise, it will be randomly generated at startup ）：

basicAuthentication.salt="some-hex-string-representing-byte-array"
basicAuthentication.iv="some-hex-string-representing-byte-array"
basicAuthentication.secret="my-secret-string"

To configure LDAP/LDAPS Authentication

basicAuthentication.ldap.enabled=< Boolean flag to enable/disable ldap authentication >
basicAuthentication.ldap.server=< fqdn of LDAP server>
basicAuthentication.ldap.port=< port of LDAP server>
basicAuthentication.ldap.username=< LDAP search username>
basicAuthentication.ldap.password=< LDAP search password>
basicAuthentication.ldap.search-base-dn=< LDAP search base>
basicAuthentication.ldap.search-filter=< LDAP search filter>
basicAuthentication.ldap.connection-pool-size=< number of connection to LDAP server>
basicAuthentication.ldap.ssl=< Boolean flag to enable/disable LDAPS>

（ Optional ） Limit to specific LDAP Group access
basicAuthentication.ldap.group-filter=< LDAP group filter>
basicAuthentication.ldap.ssl-trust-all=< Boolean flag to allow non-expired invalid certificates>

Example （ On-line LDAP Test server ）：

basicAuthentication.ldap.enabled=true
basicAuthentication.ldap.server="ldap.forumsys.com"
basicAuthentication.ldap.port=389
basicAuthentication.ldap.username="cn=read-only-admin,dc=example,dc=com"
basicAuthentication.ldap.password=" password "
basicAuthentication.ldap.search-base-dn="dc=example,dc=com"
basicAuthentication.ldap.search-filter="(uid=$capturedLogin$)"
basicAuthentication.ldap.group-filter="cn=allowed-group,ou=groups,dc=example,dc=com"
basicAuthentication.ldap.connection-pool-size=10
basicAuthentication.ldap.ssl=false
basicAuthentication.ldap.ssl-trust-all=false

Deploy

The following command will create a... That can be used to deploy the application zip file .

./sbt clean dist

See about production deployment / Configured playback frame document .

If java Not in your path , Or you need to be specific ​​ Of java Build version , Please use the following （ The example is assumed to be zulu java11）：

$ PATH=/usr/lib/jvm/zulu-11-amd64/bin:$PATH \
JAVA_HOME=/usr/lib/jvm/zulu-11-amd64 \
/path/to/sbt -java-home /usr/lib/jvm/zulu-11-amd64 clean dist

This ensures that the... In the path is found in the right place first “java” and “javac” Binary . Next , Listen only for all JAVA_HOME Downstream tools , It points them to java11 Location . Last , It tells sbt Also used java11 Location .

Start the service

Decompress the generated zipfile And change the working directory to it , You can run services like this ：

$ bin/cmak

By default , It will select the port 9000. This is covered , The location of the configuration file is the same . for example ：

$ bin/cmak -Dconfig.file=/path/to/application.conf -Dhttp.port=8080

Again , If java Not in your path , Or you need to target different versions of java function , Please add -java-home Options , As shown below ：

$ bin/cmak -java-home /usr/lib/jvm/zulu-11-amd64

Start the service safely

for SASL add to JAAS To configure , Please add the profile location at the beginning ：

$ bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf

Be careful ： Make sure to run CMAK（pka kafka manager） Users of jaas The configuration file has read permission

packing

If you want to create a Debian or RPM software package , You can run one of the following commands ：

sbt debian:packageBin
sbt rpm:packageBin

consumer / Producers lag behind
Producer offset is polled . Consumer bias is based on Kafka The offset theme of the consumer reads . This means that the lag in reporting may be negative , Because we consume offsets from offset topics faster than polling producer offsets . It's normal , Not a problem .

from Kafka Manager Migrate to CMAK
Copy the configuration file from the old version to the new version （application.conf、consumer.properties）
Change the startup script to use bin/cmak instead of bin/kafka-manager

2. Description of common parameters

2.1 Add management kafka colony

Including the name of the cluster , Managing clusters zookeeper Address ,kafka Version of

among Enable JMX Polling, Open or not JMX Training in rotation , This part directly affects the part kafka broker and topic The acquisition of monitoring indicators （ The premise of effectiveness is kafka It's on at startup JMX_PORT. It mainly affects the view of the following indicators ：
1.broker/topic Of combined metrics Comprehensive indicators

2.Poll consumer information
Whether to enable to obtain consumption information , Direct impact can be found on consumer pages and topic View consumption information on the page

Partition information and the offset of each partition record ,leader Number , A copy of each partition , Number of copies synchronized , Whether it is the first choice leader, Whether it is lower than the copy

Click on topic Page Consumers consuming from this topic

Meaning of each field in consumption details （Partition：topic The partition ,LogSize： Total data of the partition ,Consumer Offset： The data that has been consumed under the partition ;Lag： Remaining data under partition .LogSize=Consumer Offset+Lag）

1.Enable Active OffsetCache
Open or not offset cache , decision kafka-manager Whether to cache topic The correlation offset of .

2.3topic management

Broker Skew: reflect broker Of I/O pressure ,broker When there are too many copies on , Relative to others broker , The broker Frequently from Leader Partition fetch Fetching the data , Disk operations are relative to other broker More , If the indicator is too high , explain topic All the partitions are not good ,topic The stability is weak ;
Broker Leader Skew： The process of data production and consumption is as follows Leader Dealing with different areas , If broker Of Leader Too many partitions , The broker The inflow and outflow of data relative to other broker All of them should be big , The index is too high , explain topic It's not good enough ;
Under Replicated: When the index is too high , indicate topic It's easy to lose your data , The data is not copied to enough broker On .

2.4topic details

Preferred Replicas
In the copy of the partition , Take the first copy in the copy list as Leader The proportion of

2.5topic operation

operation explain
Delete Topic Delete topic
Reassign Partitions Balance cluster load
Add Partitions Add partitions
Update Config Topic Configuration information update
Manual Partition Assignments Manually assign the replica under each partition broker
Generate Partition Assignments The system automatically allocates the replica under each partition broker

generally speaking , Manual adjustment 、 The system automatically assigns partitions and adds partitions , All need to be called ?Reassign Partition.
Manual Partition Assignments

Generally, there should be Broker Skew Or Broker Leader Skew It can be adjusted with the help of this function , Earlier in this article Broker Skew and Broker Leader Skew With the help of this tool .

Generate Partition Assignments
This function is generally used in batch movement partition It's more convenient when , For example, new clusters broker perhaps topic newly added partition after , Move the partition to the specified location broker.

2.6 Consumption monitoring

kafka manager Be able to get the current consumption kafka Information about cluster consumers .