One stop Kafka monitoring and management platform kafkacenter

Big data learning monk 2022-02-13 07:15:39 阅读数:693

stop kafka monitoring management platform


KafkaCenter What is it? ?

KafkaCenter What problems have been solved

The problem we face

Kafka Center What problems have been solved

Function module introduction

Operation steps for use

Use the created test User login system

  Recently doing kafka Monitoring management plug-in research , Today is to write about the excellent open source plug-ins KafkaCenter, Powerful ,git The address is at the end of the article .

kafka Management platform related recommendations : Didi open source software :kafka-manager It is a powerful open-source plug-in for operation and maintenance kafkamanager_ big data practitioners ~-CSDN Blog

A picture gives an overview of kafkaCenter

KafkaCenter What is it? ?

KafkaCenter yes Kafka Cluster management and maintenance , production / Consumption monitoring , A unified one-stop platform for ecological components .

KafkaCenter What problems have been solved

Before we tell you what problems we solve , Let's talk about whether there is KafkaCenter The problems we faced before .

The problem we face

  • establish topic, Manual processing
  • relevant kafka Operation and maintenance , Monitor islanding
  • The existing consumption monitoring tools are not accurate
  • Can't get it Kafka Clustered summay Information
  • Unable to quickly know the health status of the cluster
  • There is no way to know the impact of business on team kafka usage
  • kafka management , Monitoring tools are scarce , Without a good tool, we can directly use
  • Unable to quickly query topic news

    Kafka Center What problems have been solved

  • Unified :  A platform , One stop includes self-service , management , monitor , Operation and maintenance , Use integrated .
  • Process :  establish topic Process , Be right topic Use lifecycle management .
  • Reuse :  The platform supports access to multiple clusters , High reusability .
  • cost :  Just deploy a set of programs , Save machine resources . Lower maintenance cost , Efficient operation and maintenance .
  • ecology :  At present, it has been connected to connect,ksql.
  • convenient :  Provide convenient tools , Make it unnecessary to have kafka People who use experience , Can be easily produced 、 News consumption .
  • overall situation :  You can view the current situation from different dimensions kafka usage
  • jurisdiction :  Perfect authority design , Reduce risk vulnerabilities .

Function module introduction

  • Home->
    View platform management Kafka Cluster Cluster information and monitoring information
  • Topic->
    Users can view their own... In this module Topic, Initiate an application to create a new Topic, At the same time, it can be right Topic Conduct production and consumption tests .
  • Monitor->
    In this module, users can view Topic Production and consumption , At the same time, you can set early warning information for consumption delay .
  • Kafka Connect->
    Enable users to quickly create their own Connect Job, And to their own Connect For maintenance .
  • KSQL->
    Enable users to quickly create their own KSQL Job, And to their own Job For maintenance .
  • Approve->
    This module is mainly used when ordinary users apply to create Topic, The administrator performs approval .
  • Setting->
    The main function of this module is administrator maintenance User、Team as well as kafka cluster Information
  • Kafka Manager->
    This module is used for the normal maintenance of the cluster by the administrator .

Operation steps for use

  1. Use the system to configure the user admin/admin Landing system

2. Log in to the system and enter Kafka Manager -> Cluster, Click on Add New Cluster Fill in the cluster information according to the prompt

3. Get into Setting -> User, Click on Add User Add users to the system (test and testAdmin),test The user role is Member,testAdmin The user role is Admin

Important: Except that the system comes with admin Outside the account , You must create a role for admin The account of , Otherwise, errors will be reported in the system where email is sent to the administrator (eg: establish task) 

4. After adding users successfully , Get into Setting -> Team, Click on New Team, Fill in team name (test_team)

5. add to team After success , Click on team Operation Medium addUser Button , Select the one you just added user, Click on Add User To Team Add users to team in

6. Click the image in the upper right corner to select logout Exit the system

Use the created test User login system

1. Use the user created above (test) Login system

2. establish topic

  • Get into Topic -> My Task, Click on Create Topic Task, Fill in... As prompted topic Information , establish topic(topic_test), The system will send a reminder email to the Administrator

  • Exit the system , Use testAdmin Account login system
  • Get into Approve, Select submitted Topic(topic_test), Click on check, Select the cluster you want to create , Click on Approve.

  • topic Once created , The system will send a successful email to the user test

3. Production news

  • topic Once created , Get into Topic -> Topic List, Choose to create topic(topic_test) choice mock operation

  • Fill in message Of key and value, Click on Send, Prompt on the right PRODUCER TOPIC SUCCESS, Successful production

4. Monitor production status

  • topic(topic_test) After successful message production , Get into Monitor -> Producer
  • Click on topic(topic_test), Enter the production monitoring information , Displays the rate of production

5. News consumption

Important: If not selected Commit The Record Consumed Words , stay consumer Consumption information cannot be viewed in

  • Get into Topic -> Queries
  • choice cluster and topic name, choice Commit The Record Consumed, Click on Query
  • After successful consumption , The right side shows the messages consumed

  • choice Consumer Topic By Offset, choice partition, Fill in offset, Click on Query( Appoint partition Consumption )
  • After successful consumption , The right side shows the messages consumed

6. Monitor consumption

  • Get into Monitor -> Consumer, choice Topic, choice topic(topic_test), View the consumption status of the consumption group

Click... Behind the consumer group alert, by group Group add alert monitor , When lag When the configured value is reached , The system will send an email

Click the consumption group used for consumption above , View consumption lag chart

The optional time interval is ,5 minute 、10 minute 、30 minute 、1 Hours 、6 Hours 、12 Hours 、1 God 、7 God ,

You can also customize the scope

kafkaCenter-git Address :GitHub - xaecbd/KafkaCenter: KafkaCenter is a unified platform for Kafka cluster management and maintenance, producer / consumer monitoring, and use of ecological components.

copyright:author[Big data learning monk],Please bring the original link to reprint, thank you.