Big data learning monk 2022-02-13 07:16:44 阅读数:575
CMAK( Formerly known as Kafka Manager) It's a way to manage Apache Kafka Clustering tools . For more information about name changes , See below .
Kafka 0.8.. or 0.9.. or 0.10.. or 0.11..
Java 11+
The minimum configuration is for CMAK(pka kafka manager) State of zookeeper host . This can be done in conf In the directory application.conf Found in file . The same files will be packaged in the distribution zip In file ; You can unzip the file on the desired server and modify the settings .
cmak.zkhosts="my.zookeeper.host.com:2181"
You can specify multiple by separating them with commas zookeeper host , As shown below :
cmak.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181"
perhaps ,ZK_HOSTS If you don't want to hard code any values , Please use environment variable .
ZK_HOSTS="my.zookeeper.host.com:2181"
You can choose to modify application.conf The default list in the / Disable the following functions :
application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"]
Consider enabling jmx Set these parameters for larger clusters :
cmak.broker-view-thread-pool-size=<3 * number_of_brokers>
cmak.broker-view-max-queue-size=<3 * Total number of partitions for all topics >
cmak.broker-view-update-seconds=< cmak.broker-view-max-queue-size / (10 * number_of_brokers) >
Here's what has 10 A broker 、100 A theme kafka An example of a cluster , Each theme has 10 Zones , In the activation of JMX A total of 1000 Zones :
cmak.broker-view-thread-pool-size=30
cmak.broker-view-max-queue-size=3000
cmak.broker-view-update-seconds=30
The following controls the thread pool and queue of the consumer offset cache :
cmak.offset-cache-thread-pool-size=< The default is the number of processors >
cmak.offset-cache-max-queue-size=< The default value is 1000>
cmak.kafka-admin-client-thread-pool-size=< The default is the number of processors >
cmak.kafka-admin-client-max-queue-size=< The default value is 1000>
For a large number of consumers with consumer polling enabled , You should increase the above value . Although its main impact is based on ZK Consumer voting .
Kafka Managed consumer offsets are now managed by “__consumer_offsets” In the theme KafkaManagedOffsetCache consumption . Please note that , This has not been tested to track large offsets . One thread per cluster uses this topic , Therefore, it may not be able to keep up with the large number of offsets pushed to the topic .
Use LDAP Validate users
Warning , You need to use CMAK(pka Kafka Manager) To configure SSL, To ensure that your credentials are not passed unencrypted . By associating user credentials with Authorization The header is passed together , have access to LDAP Authenticate users .LDAP Authentication is completed on the first access , If it works , Is set cookie. In the next request ,cookie Value and Authorization Compare the credentials in the header .LDAP Support is achieved through basic authentication filters .
Configure basic authentication
basicAuthentication.enabled=true
basicAuthentication.realm=< basic authentication realm>
Encryption parameter ( Optional , Otherwise, it will be randomly generated at startup ):
basicAuthentication.salt="some-hex-string-representing-byte-array"
basicAuthentication.iv="some-hex-string-representing-byte-array"
basicAuthentication.secret="my-secret-string"
To configure LDAP/LDAPS Authentication
basicAuthentication.ldap.enabled=< Boolean flag to enable/disable ldap authentication >
basicAuthentication.ldap.server=< fqdn of LDAP server>
basicAuthentication.ldap.port=< port of LDAP server>
basicAuthentication.ldap.username=< LDAP search username>
basicAuthentication.ldap.password=< LDAP search password>
basicAuthentication.ldap.search-base-dn=< LDAP search base>
basicAuthentication.ldap.search-filter=< LDAP search filter>
basicAuthentication.ldap.connection-pool-size=< number of connection to LDAP server>
basicAuthentication.ldap.ssl=< Boolean flag to enable/disable LDAPS>
( Optional ) Limit to specific LDAP Group access
basicAuthentication.ldap.group-filter=< LDAP group filter>
basicAuthentication.ldap.ssl-trust-all=< Boolean flag to allow non-expired invalid certificates>
Example ( On-line LDAP Test server ):
basicAuthentication.ldap.enabled=true
basicAuthentication.ldap.server="ldap.forumsys.com"
basicAuthentication.ldap.port=389
basicAuthentication.ldap.username="cn=read-only-admin,dc=example,dc=com"
basicAuthentication.ldap.password=" password "
basicAuthentication.ldap.search-base-dn="dc=example,dc=com"
basicAuthentication.ldap.search-filter="(uid=$capturedLogin$)"
basicAuthentication.ldap.group-filter="cn=allowed-group,ou=groups,dc=example,dc=com"
basicAuthentication.ldap.connection-pool-size=10
basicAuthentication.ldap.ssl=false
basicAuthentication.ldap.ssl-trust-all=false
The following command will create a... That can be used to deploy the application zip file .
./sbt clean dist
See about production deployment / Configured playback frame document .
If java Not in your path , Or you need to be specific Of java Build version , Please use the following ( The example is assumed to be zulu java11):
$ PATH=/usr/lib/jvm/zulu-11-amd64/bin:$PATH \
JAVA_HOME=/usr/lib/jvm/zulu-11-amd64 \
/path/to/sbt -java-home /usr/lib/jvm/zulu-11-amd64 clean dist
This ensures that the... In the path is found in the right place first “java” and “javac” Binary . Next , Listen only for all JAVA_HOME Downstream tools , It points them to java11 Location . Last , It tells sbt Also used java11 Location .
Decompress the generated zipfile And change the working directory to it , You can run services like this :
$ bin/cmak
By default , It will select the port 9000. This is covered , The location of the configuration file is the same . for example :
$ bin/cmak -Dconfig.file=/path/to/application.conf -Dhttp.port=8080
Again , If java Not in your path , Or you need to target different versions of java function , Please add -java-home Options , As shown below :
$ bin/cmak -java-home /usr/lib/jvm/zulu-11-amd64
for SASL add to JAAS To configure , Please add the profile location at the beginning :
$ bin/cmak -Djava.security.auth.login.config=/path/to/my-jaas.conf
Be careful : Make sure to run CMAK(pka kafka manager) Users of jaas The configuration file has read permission
If you want to create a Debian or RPM software package , You can run one of the following commands :
sbt debian:packageBin
sbt rpm:packageBin
consumer / Producers lag behind
Producer offset is polled . Consumer bias is based on Kafka The offset theme of the consumer reads . This means that the lag in reporting may be negative , Because we consume offsets from offset topics faster than polling producer offsets . It's normal , Not a problem .
from Kafka Manager Migrate to CMAK
Copy the configuration file from the old version to the new version (application.conf、consumer.properties)
Change the startup script to use bin/cmak instead of bin/kafka-manager
Including the name of the cluster , Managing clusters zookeeper Address ,kafka Version of
among Enable JMX Polling, Open or not JMX Training in rotation , This part directly affects the part kafka broker and topic The acquisition of monitoring indicators ( The premise of effectiveness is kafka It's on at startup JMX_PORT. It mainly affects the view of the following indicators :
1.broker/topic Of combined metrics Comprehensive indicators
2.Poll consumer information
Whether to enable to obtain consumption information , Direct impact can be found on consumer pages and topic View consumption information on the page
Partition information and the offset of each partition record ,leader Number , A copy of each partition , Number of copies synchronized , Whether it is the first choice leader, Whether it is lower than the copy
Click on topic Page Consumers consuming from this topic
Meaning of each field in consumption details (Partition:topic The partition ,LogSize: Total data of the partition ,Consumer Offset: The data that has been consumed under the partition ;Lag: Remaining data under partition .LogSize=Consumer Offset+Lag)
1.Enable Active OffsetCache
Open or not offset cache , decision kafka-manager Whether to cache topic The correlation offset of .
Broker Skew: reflect broker Of I/O pressure ,broker When there are too many copies on , Relative to others broker , The broker Frequently from Leader Partition fetch Fetching the data , Disk operations are relative to other broker More , If the indicator is too high , explain topic All the partitions are not good ,topic The stability is weak ;
Broker Leader Skew: The process of data production and consumption is as follows Leader Dealing with different areas , If broker Of Leader Too many partitions , The broker The inflow and outflow of data relative to other broker All of them should be big , The index is too high , explain topic It's not good enough ;
Under Replicated: When the index is too high , indicate topic It's easy to lose your data , The data is not copied to enough broker On .
Preferred Replicas
In the copy of the partition , Take the first copy in the copy list as Leader The proportion of
operation explain
Delete Topic Delete topic
Reassign Partitions Balance cluster load
Add Partitions Add partitions
Update Config Topic Configuration information update
Manual Partition Assignments Manually assign the replica under each partition broker
Generate Partition Assignments The system automatically allocates the replica under each partition broker
generally speaking , Manual adjustment 、 The system automatically assigns partitions and adds partitions , All need to be called ?Reassign Partition.
Manual Partition Assignments
Generally, there should be Broker Skew Or Broker Leader Skew It can be adjusted with the help of this function , Earlier in this article Broker Skew and Broker Leader Skew With the help of this tool .
Generate Partition Assignments
This function is generally used in batch movement partition It's more convenient when , For example, new clusters broker perhaps topic newly added partition after , Move the partition to the specified location broker.
kafka manager Be able to get the current consumption kafka Information about cluster consumers .
copyright:author[Big data learning monk],Please bring the original link to reprint, thank you. https://en.javamana.com/2022/02/202202130716400645.html