Why say KubeMQ Would be Kafka substitute ？

Modern applications are complex , There are many different moving parts . Even for the most basic execution center applications , The front-end interface will also trigger payment processing transactions , The payment processing transaction will trigger manufacturing and transportation Events . These services need a reliable way to communicate with each other , Regardless of the availability of the underlying network or other services .

In order to achieve this complex operation , There must be some type of service “ Post Office ” To track all requests and alerts . The tool to achieve this goal is message queuing .

Message queuing is a specialized application , It acts as an intermediary between different services of distributed applications or between different applications . It separates application services from each other , Ensure that no matter whether the message recipient is available , All of them will be dealt with . Message queuing ensures that all messages are finally successfully received .

Common use cases for message queuing include ：

• Asynchronous processing between different applications .
• Microservice based applications , Among them, reliable communication between different components is very important .
• Transaction sequencing and restrictions .
• Data processing operations that can benefit from the simplified efficiency of batch processing .
• Applications that must be scalable to meet sudden and unexpected changes in requirements .
• Applications must be resilient enough to recover from crashes and unexpected failures .
• Limit resource consumption through long-running processes .

There is no shortage of vendors in the field of message queuing . image Amazon Web Services、Microsoft Azure And Google cloud Such large cloud platforms have their own products (AWS Simple Queue Service、Azure Service bus and Google's Pub/Sub). There is also an independent general message broker , Such as RabbitMQ、Apache Of ActiveMQ and Kafka.

This paper introduces a project named KubeMQ Modern Kubernetes Native message queue , To try to make already in Kubernetes Upper use kafka How your organization can benefit from it .

What is? Apache Kafka

To understand KubeMQ The full value of , First of all, we need to take some time to understand Kafka.Kafka By the first LinkedIn Engineers create , As a tracker LinkedIn User active software bus . It was later released as an open source product , today ,Kafka from Apache Software Foundation Development and management .

Apache Pointed out that , exceed 80% The wealth of 100 Strong company trust and use Kafka. Although it's open source , But it is well known that it is a highly scalable system , Can connect to a wide range of event producers and consumers . It can be configured to perform complex functions using data flow , It works well even in a limited network environment . With support widely available in the online user community ,Kafka It also offers a variety of commercial products . for example ,AWS Provide hosting Kafka,Confluent So it is with .

Kafka The limitations of

Despite the high adoption rate , but Kafka Not always the best choice for a message queuing system . It has a single architecture , Applicable to local cluster or high-end multi virtual machine settings . Whereas Kafka How much memory and storage space is needed , Quickly starting a multi node cluster on a stand-alone workstation for testing can be a challenge .

In short , take Kafka It's not easy to successfully integrate all the complex parts needed to integrate with your infrastructure . Based on Kubernetes This is especially true of our architecture .

As shown in the figure below , be based on Kubernetes Of Kafka Deployment has different active parts . Except for the basic Kubernetes Cluster configuration and underlying computing 、 Network and storage infrastructure （ If you implement it locally ）, You also need to install all Kafka Part and compare it with Helm Wait for package manager Integration . These components can include a coordinator , Such as ZooKeeper or Mesos, Used to manage Kafka Agents and themes . Other things to pay attention to include dependencies 、 journal 、 Partition etc. . If even an element is missing or incorrectly configured , Nothing will work —— Successful deployment Kafka It's not easy .

New Kafka Add nodes to Kubernetes Clusters require complex manual balancing to maintain optimal resource use . That's why there's no easy way to manage and ensure reliable backup and recovery policies ; For those running on a large number of nodes Kafka Cluster disaster prevention is not easy . And Kubernetes The data in the cluster is saved in pod outside , And the orchestrator fails to start automatically pod Different ,Kafka There is no such native fail safe mechanism .

Last , Yes Kafka/ZooKeeper/Kubernetes Effective monitoring of deployment requires third-party tools .

What is? KubeMQ

Kube MQ Is a message service , When you build from scratch, you take into account Kubernetes. Follow container architecture best practices ,KubeMQ Designed to achieve stateless and transient . in other words , One KubeMQ The node will remain unchanged throughout its life cycle 、 Predictable and reproducible . If you need to change the configuration , The node will be shut down and replaced .

This repeatability means , And Kafka Different ,KubeMQ With zero configuration settings , There is no need to adjust the configuration after installation .

KubeMQ Designed to accommodate the widest range of messaging patterns . It is a message broker and message queue , Support the following ：

• With or without persistence Pub/Sub
• request / reply （ Sync 、 asynchronous ）
• One delivery at most
• At least one delivery
• Streaming mode
• RPC

by comparison ,Kafka Only those with persistence and streaming are supported Pub/Sub.Kafka Not at all RPC And request / Reply mode .

In terms of resource use ,KubeMQ Outperform with minimal footprint Kafka.KubeMQ docker The container only occupies 30MB Space . Such a small footprint helps with fault-tolerant settings and simplified deployment . And Kafka Different , take KubeMQ Small development added to the local workstation Kubernetes The environment is very simple . But at the same time ,KubeMQ It has enough scalability , It can be deployed in a mixed environment running on hundreds of local and cloud managed nodes . The core of this easy deployment is kubemqctl, It is KubeMQ Command line interface tool for , Be similar to Kubernetes Of kubectl.

KubeMQ be better than Kafka Another aspect of is its speed .Kafka Yes, it is Java and Scala Compiling , and KubeMQ Yes, it is Go Compiling , Ensure fast operation . In internal benchmark operation ,KubeMQ Handle 100 10000 messages faster than Kafka fast 20%.

go back to KubeMQ Of “ No configuration ” aspect , Channels are the only objects that developers need to create . You can forget the agent 、 Switch and coordinator ——KubeMQ Of Raft Instead of ZooKeeper Do all of this .

From a monitoring point of view , adopt Prometheus and Grafana Dashboard and KubeMQ Fully integrated , So you don't have to manually integrate the extra work of third-party observability tools . however , because KubeMQ Native integration with tools , You can still use existing logging and monitoring solutions , Include ：

• Fluentd、Elastic and Datadog, Used to monitor
• Loggly, Used to record
• Jaeger and Open Tracing, For tracking

because Kafka Not the cloud native Computing Foundation (CNCF) The native part of the environment , Therefore, it is usually not supported to work with CNCF Integration of tools , You have to manually configure .

If configured , It can be through open source gRPC The remote procedure calls the system to connect , And Kubernetes The excellent compatibility of is well known .Kafka Their own proprietary connection mechanism may not provide comparable results .

from Kafka To KubeMQ Transparent migration of

except KubeMQ Besides simple deployment and operation , Put the existing Kafka Migrate settings to KubeMQ It's also very simple. .

So , Developers can use KubeMQ Kafka The connector starts .KubeMQ The target and source connectors are configured to convert from Kafka The news of . At a high level ,KubeMQ The source connector is consumed as a subscriber from Kafka Messages from the source topic , Convert the message to KubeMQ The message format , Then send the message to the internal log .KubeMQ The target connector subscribes to the output log containing the transformation message , Then send the message to Kafka Target theme in . The high-level architecture is shown in the figure below ：

Besides ,Kafka Any messaging mode supported is by KubeMQ Support . for example ,Kafka Only those with persistence and flow are supported Pub/Sub.KubeMQ Is a message queue and message broker , Support Pub/Sub（ With or without persistence ） request / reply （ Sync 、 asynchronous ）、 At least one delivery 、 Stream mode and RPC. therefore , from Kafka Migrate to KubeMQ when , No need to refactor application code and adapt to complex logical changes .

Last

For most workloads ,KubeMQ Built in simplicity 、 Lightweight and container first integration will provide better performance than Kafka Performance of . Besides , The almost zero configuration required will save a lot of management and setup time . As we mentioned , Migration is simple .

KubeMQ It's free to download , Comes with a six-month free development trial . If you use OpenShift, Can be in Red Hat Marketplace Use in KubeMQ . It also applies to all major cloud environments , Include GCP、AWS、Azure and DigitalOcean.