Skip to main content
Apache Kafka

Apache Kafka


What is Apache Kafka?

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical…

Read more
Recent Reviews

TrustRadius Insights

Apache Kafka is a widely-used platform that has proven to be invaluable in various industries and applications. It is relied upon by …
Continue reading

Apache Kafka - FTW

9 out of 10
August 21, 2023
We use Apache Kafka as message broker between our two client facing applications. We used ActiveMQ before but it had shortfalls of high …
Continue reading
Read all reviews

Reviewer Pros & Cons

View all pros & cons
Return to navigation

Product Details

What is Apache Kafka?

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Apache Kafka Technical Details

Operating SystemsUnspecified
Mobile ApplicationNo
Return to navigation


View all alternatives
Return to navigation

Reviews and Ratings


Community Insights

TrustRadius Insights are summaries of user sentiment data from TrustRadius reviews and, when necessary, 3rd-party data sources. Have feedback on this content? Let us know!

Apache Kafka is a widely-used platform that has proven to be invaluable in various industries and applications. It is relied upon by organizations to have real-time communication and keep order information up-to-date. This is particularly useful for organizations that need to process large volumes of data, such as those in the cybersecurity industry. Apache Kafka is also considered the go-to tool for event streaming, generating events and notifying relevant applications for consumption. Additionally, it is used in both first-party and third-party components of applications to address data proliferation and enable efficient notifications.

Another key use case for Apache Kafka is replacing classical messaging software within organizations, becoming the new standard for messaging. This powerful streaming framework plays a crucial role as a queuing mechanism for records in various pipelines, providing a simple yet efficient system for queuing and maintaining records. Moreover, Apache Kafka excels at storing and processing records in dedicated servers, supporting high data loads and offering the ability to replay consumed data. This makes it ideal for buffering incoming records during traffic spikes or in case of data infrastructure failures.

Furthermore, Apache Kafka finds its purpose in driving real-time monitoring by sending log information to feed other applications. Its ability to scale and manage common errors in messaging allows organizations to handle large quantities of messages per second without compromising performance. Another notable use case involves Apache Kafka acting as an efficient stream/message ingestion engine for customer-facing applications, enabling internal analytics and real-time decision-making.

Additionally, Apache Kafka integrates seamlessly with big data technologies like Spark, making it a valuable addition to big data ecosystems. Organizations have successfully replaced legacy messaging solutions with Apache Kafka, thanks to its ability to serve as a messaging and data-streaming pipeline solution. It enables modern streaming API-based applications while ensuring high availability and clustering as a message broker between client-facing applications.

Moreover, Apache Kafka serves as an ingress and egress queue for big data systems, facilitating data storage and retrieval processes. It also acts as a reliable queue for frontend applications to retrieve data and analytics from MapR and HortonWorks. With over five years of being utilized in data pipelines, Apache Kafka has consistently demonstrated excellent performance and reliability.

In summary, Apache Kafka proves to be versatile and essential across various industries and use cases. It facilitates real-time communication, ensures data integrity, enables efficient event streaming, replaces classical messaging software, and supports high scalability and fault tolerance. With its robust capabilities, Apache Kafka continues to be the go-to solution for organizations seeking to streamline their data processing and communication systems.

Fault tolerance and high scalability: Users have consistently praised Apache Kafka for its fault tolerance and high scalability. Many reviewers have stated that Kafka excels in handling large volumes of data and is considered a workhorse in data streaming.

Ease of administration: Reviewers appreciate Kafka's ease of administration, noting that it offers an abundance of options for managing and maintaining queues. Multiple users have mentioned that the platform allows for easy expansion and configuration of cluster growth, making it straightforward to administer.

Real-time streaming capabilities: Kafka's real-time streaming capabilities are seen as a significant advantage by users. Several reviewers have highlighted the platform's ability to handle real-time data pipelines and its resistance to node failure within the cluster. This feature enables users to process asynchronous data efficiently and ensures continuous availability of the system.

Difficulty Monitoring Kafka Deployments: Some users have found it difficult to monitor their Kafka deployments and have expressed a desire for a separate monitoring dashboard that would provide them with better visibility into their topics and messages.

Steep Learning Curve for Creating Brokers and Topics: The process of creating brokers and topics in Kafka has been described as having a steep learning curve by some users, who believe that it could be simplified to make it more accessible.

Outdated Web User Interface: The web user interface of Kafka has not been updated in years, leading some users to feel that it lacks a streamlined user experience. They express the need for a more modern interface instead of relying on third-party tools.

Users have recommended using Apache Kafka for various messaging platform requirements. It integrates easily with multiple programming languages, offers stream processing capabilities, distributed data storage, and the ability to handle multiple requests simultaneously.

Another common recommendation is to consider Apache Kafka as a messaging broker due to its extensive feature set and guaranteed delivery of data to consumers. Users find it highly supported and widely used within the community.

Users also recommend Apache Kafka for streaming large amounts of data. They praise its scalability and ease of use, although they mention that manual rebalancing of partitions may be required when adding or deleting nodes. Additionally, users appreciate that Kafka allows connections between multiple producers and consumers with low resource consumption.

Overall, Apache Kafka is regarded as a practical choice for message processing systems, data streaming, and handling large volumes of data due to its stability, scalability, and diverse features.

Attribute Ratings


(1-18 of 18)
Companies can't remove reviews or game the system. Here's why
Score 8 out of 10
Vetted Review
Verified User
Apache Kafka is really the bedrock of all things streaming and data processing. I cannot imagine if there is any other product that does it better. My last 2 companies used it, and my current one does so as well. If you want your data stream to be organized and sent, Apache Kafka has become the tool of choice. I have dabbled in Azure EventHubs as well, if you are into opensource data streaming, Apache Kafka will take you where you need to be for data lakes and the amount of data that is streamed for the cybersecurity industry that my company is in. Without Apache Kafka, there is no way that my company products can handle the volume of data that we process for our customers.
  • Data streaming is really second to none.
  • Scaling, done right, Apache Kafka is a workhorse.
  • Ease of administration - Although you cannot really compare to Azure EventHubs, but that is comparing between Apples and Oranges.
  • The web UI has not really changed in years. UX has been refreshed, but a more streamlined UX instead of many 3rd party webUX tools, will be most welcome.
  • Webhooks can still be tricky to troubleshoot at times.
  • CLI monitoring is a learning curve to get it right.
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
August 21, 2023

Apache Kafka - FTW

Score 9 out of 10
Vetted Review
Verified User
We use Apache Kafka as message broker between our two client facing applications. We used ActiveMQ before but it had shortfalls of high availability and clustering. Kafka solved it on both fronts and gives a good business continuity.
  • High availability
  • performance
  • Admin user interface
  • zookeeper logs could be better
  • monitoring
It is well suited if you want to use a message broker between two applications with high availability. Its also can be used as streaming replication for data.
Alok Pabalkar | TrustRadius Reviewer
Score 10 out of 10
Vetted Review
Verified User
We use Apache Kafka as an event bus for all our async activities & Micro Service Communication, like sending emails, SMS, and notifications between services and consumers and for event & data processing.
  • Event driven architectures
  • Any use case which requires async data processing
  • Any use case with production and consuming the same data to build business-specific processing
  • Zookeeper services configuration can be simplified
  • Data logging needs to be secured
  • Restarting & overall management needs to be improved
- It's Super fast - Has some learning curve but once mastered it brings scale - All logics that need producer & consumer kind of implementation (Bulk Notification, etc) - Event-driven architectures can be implemented with Apache Kafka
Animesh Kumar | TrustRadius Reviewer
Score 8 out of 10
Vetted Review
Verified User
We use Apache Kafka to stream order information across systems. An order may go through certain updates through its lifecycle. These updates need to be communicated to the systems in near real time and we rely on Kafka for this.Our business use case is to take these orders up with the insurance companies for approval and thus the order information need to be up to date. Kafka has been excellent at doing this so far.
  • Receiving messages from publisher and sending to consumer in FIFO manner
  • Handling of errors using Dead Letter Queue when message could not be consumed on the consumer end
  • Fault tolerance
  • Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
  • Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
  • Learning curve around creation of broker and topics could be simplified
Kafka is well suited in scenarios where a message need to be sent to another system in fault tolerant manner. It is useful when the message size could be large and large number of messages could be floating around.
It would be less appropriate or rather an overkill to use Kafka in scenarios where we are sending short messages to offload certain tasks(like invoice generation and sending email) to a worker(like celery). For such use cases, simple queueing solutions like Amazon SQS should suffice.
Score 6 out of 10
Vetted Review
Verified User
Currently consulting and implementing for a bank, we use a cloud-native Kafka solution (Confluent Kafka) for brokering. The solution is well documented, and liked by the developers but lacks certain technical aspects to improve usability and administration.
  • Brokering
  • Topic definition.
  • Private access to a cluster.
  • Visualisation solutions.
For brokering messages, Confluent Kafka is well suited since it offers a managed solution ready to use. Scenarios where the solution is not very well suited are for example, where pricing is an issue. The solution costs quite a lot for basic usage (for example: for 3 clusters, pricing is above 100k$ a year).
Score 10 out of 10
Vetted Review
Verified User
Apache Kafka is the most powerful and scalable streaming framework on the market. We have used Apache Kafka as a part of many real-time analytics solutions. It has a great performance [and is] easy to integrate with big data technologies like Spark. Due to its distributed nature, Apache Kafka is capable of operating very quickly and can handle millions of messages every second.
  • Real time streaming
  • Performance
  • Scalability
  • Management tools
I have used Apache Kafka for real-time analytics and streaming. It’s highly scalable and integrates well with big data technologies like Spark. I believe Apache Kafka is the best in the market.
Borislav Traykov | TrustRadius Reviewer
Score 8 out of 10
Vetted Review
Verified User
Kafka is an event streaming platform and this is exactly the purpose we use it for in our company. Application data-in-transit goes into Kafka, which generates an even, and all relevant applications (consumers) get notified and then consume said messages. We are really happy with the volume of data we get through and the speed that we get from Kafka. It's used in multiple 1st and 3rd party components of the applications we develop in the entire company. It addresses data proliferation and notifications. If not for Kafka, we'd have to invent a pub/sub model (which multiple people have in the past in this company) - those are complex, hard to maintain, extend and customize. Kafka is fair well documented and used so there is a lot of info about multiple use cases online.
  • The pub/sub model
  • Quick data transfer - regardless of volume (if you have enough resources)
  • Ability to transfer large amounts of data consistently (non-binary)
  • The Kafka Tool is a community-made Java application that looks and feels from the past century.
  • Logging can be confusing. This certainly shows when we have to do troubleshooting.
  • Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.
  • Pub/sub model when more services are involved.
  • A lot of of technologies know how to work with Kafka. There are Kafka libraries for all general-purpose languages.
  • Quick and reliable data transit and notifications.
  • Kafka can have a big memory and/or disk footprint depending on your scenario. Be prepared to delegate resources if your amount of data gets more and more. Kafka is lean by default, but it does require memory (in-mem storage) and disk (offloading) to keep your data.
  • Kafka has a lot of configuration options - be sure to check them if you need to fit Kafka into a specific scenario.
  • The Kafka Tools looks ancient, but it does what it's supposed to.
  • If your developers are debugging, they may unintentionally "steal" events/data from a given queue as they would probably register as a consumer. This is very nasty especially when dealing with a living system There are ways to avoid this, but people need to be aware that it can happen.
Tyler Twitchell | TrustRadius Reviewer
Score 9 out of 10
Vetted Review
Verified User
We use Kafka as the queuing mechanism for records in an indexing pipeline. Previous to using Kafka we were working with tables in SQL Server to handle a queue in a situation that SQL is not really designed for. Kafka provides a simple and efficient system that does the job it was intended for, queuing and maintaining records in a queue, and works very well. We use Kafka for several processes in our organization that require records to be stored and be processed by dedicated servers.
  • Queuing of records
  • Easy expansion of Topic parititions
  • An abundance of options for managing and maintaining queues
  • Easy expansion of cluster for growth
  • A management interface would be nice
  • Built in logging tools
Kafka is a queuing system, plain and simple, and it does its job efficiently and with little fuss. We utilize Splunk logging to keep track of records in queues and how items are being processed and outside of that we generally do not have to mess with Kafka, it just does the job with little maintenance or problems. Any situation where records or information need to be placed in a queue to be accessed and processed by other systems would be well suited to scenarios where Kafka is the right solution.
Score 10 out of 10
Vetted Review
Verified User
My application was dependent on other applications to generate data and those data were needed to be processed immediately. And, processed data were published for other applications. Moreover, data load was very high nearly a hundred thousand a day. And, consumed data may be replayed in the future if required. So, after carefully considering several messaging queues we finally decided to continue with Apache Kafka.
  • Every setting is configurable.
  • Work seamlessly during high data load.
  • Partition mechanism.
  • Easy configurable.
  • Zookeeper configuration.
  • Front-end can be developed to configure properties.
  • UI for administrative configuration.
Kafka can be used as a database but it is not recommended to store data for a long time. Also, if your application has a high data load then only we should utilize Kafka otherwise any other messaging queue is recommended. In addition, Apache Kafka provides far more features than just a simple messaging queue. Using Apache Kafka we can develop loosely coupled, real-time processing, and fault-tolerance architecture.
Score 10 out of 10
Vetted Review
Verified User
Kafka is being used for our IoT data flows as the middle layer to transport data and make it available for consumption. We are implementing it slowly starting project by project and plan to use it globally.
  • Message queue
  • Capture data
  • Make data available
  • Integration between systems
  • More out of the box connectors for various other system integration
Kafka is great for moving data between systems! You can even store data for a while before purging it so you know you have consumed it!
Score 10 out of 10
Vetted Review
Verified User
It is being used for the product mainly. We have huge data pipelines running which depend on Apache Kafka. It is being used for more than 5 years now and we are really happy with the performance and the reliability Apache Kafka has to offer. The experience has been excellent.
  • Data Pipeline
  • Asynchronous processing
  • Data retention for reprocessing
  • Dashboards to monitor the performance
  • ZooKeeper free
  • Connectors for more languages
  • It works overall really well for maintaining data and then processing whenever you want to as it has really good retention options. Multiple consumers can be run and systems can be scaled.
  • Works well when scale is needed
  • Can work well on low hardware requirements
  • Where it can be limiting is while implementing priority queues as it has to be done at the producer level.
Score 10 out of 10
Vetted Review
Verified User
Kafka is being used for sending log information in real time and there[fore] can monitor apps and send these events to feed other apps. It's the core for send[ing] and receiv[ing] messages due to quantity of messages per second. Helps us to scale and manage the common errors in this type of problem.
  • Scalable
  • Fast
  • Performance
  • Open source
  • Performance security
  • Monitoring
  • Configuration
Send a few events in a few time slots: Kafka is designed for high computing events. If you application doesn't work with more [than] 25.000 messages, Kafka isn't the correct solution.

Send events with high size: don't try working with events with more [than] 1 Mb, the performance is very poor.

Send event without compression: if you work with any compression with messages this will help the performance in net traffic and speed of pipeline
Score 7 out of 10
Vetted Review
Verified User
Apache Kafka is used by our company as the "next generation" of messaging/data-streaming pipeline solutions, to replace our old legacy JMS-based messaging solution and enable the modern streaming API based applications. When it is used for messaging purposes, we shift the responsibility of data replay from the message source (publisher application) to the message destination (consumer application). This flexibility resolved the legacy issue of sources replaying the messages but impacting all subscribers to the same topic. When Kafka is used as the streaming pipeline, it is integrated seamlessly with the Spark/Spring Stream-based analytic solutions, as it is also a kind of distributed storage.
  • Undoubtedly, Kafka's high throughput and low latency feature are the highlights.
  • Kafka can scale horizontally very well.
  • The CLI and configuration details need to be worked out more in-depth. The naming convention of configuration is not so good and causing a lot of confusion. Sometimes there are too many configuration parameters to tune--requires the adopter to understand a lot of tricks like NFS entrapment, for example.
  • Lack of a good monitoring solution so far
When it is used as messaging, Apache Kafka is majorly preferred when the use case is Pub/Sub typed. It is not suitable to deal with the end-to-end queue use case nor the request/response paradigm. When Apache Kafka is used for streaming purposes, it doesn't have the native implementation of the query language, it is just a pipeline. You still need to put a lot of programming efforts into your streaming client-side to take care of those analytic requirements.
Viral Patel | TrustRadius Reviewer
Score 9 out of 10
Vetted Review
Verified User
We used it for event logging. It was used for application log collection. Was used with exception tracking and with core microservices of the web application. It helped us reduce cost and simplified operational monitoring.
  • It handles large amount of data simultaneously. Makes application scalable.
  • It is able to handle real time data pipeline.
  • Resistant to node failure within the cluster.
  • Does not have complete set of monitoring tools.
  • It does not support wild card topic selection.
  • Brokers and consumer pattern reduces the performance.
It works well as a replacement for traditional message broker. Used when you want to log simultaneously tracking multiple web activities.
Score 10 out of 10
Vetted Review
Verified User
Apache Kafka is used as a stream/message ingestion engine for all the customer-facing apps including some internal streams company-wide. It is used to ingest close to 2-5 million small (few bytes) messages per second that are then used for internal analytics and decision making in realtime and feed analytics backend (Tibco Spotfire).
  • Apache Kafka is able to handle a large number of I/Os (writes) using 3-4 cheap servers.
  • It scales very well over large workloads and can handle extreme-scale deployments (eg. Linkedin with 300 billion user events each day).
  • The same Kafka setup can be used as a messaging bus, storage system or a log aggregator making it easy to maintain as one system feeding multiple applications.
  • Apache Kafka does take some initial setup and deployment time especially if you haven't bought support from Confluent.
  • It is not a full solution so for an analytics use case, you will still need something like Tibco.
  • It does not have a SQL based query engine out-of-the-box so building/using analytics on top can be a lot of work. It would be great to have something already baked into Kafka out-of-the-box.
Apache Kafka is very well suited where the deployment entails getting a very large number of small messages at extremely high rates—4 million-plus messages a second. It is also very well suited when you need stronger ordering guarantees than a traditional messaging system can provide. It is less suited when you don't need such high message ingestion rates and need to do everything in a public cloud. Apache Kafka will be an overkill for such small/simple deployments.
Score 9 out of 10
Vetted Review
Verified User
We use Kafka for two key features: (1) keeping a buffer of all the incoming records that need to be stored in our data infrastructure, and (2) having a way to replay messages in case our data infrastructure loses some data.
The reason we need to buffer is that when our traffic spikes, we can have up to 1 million messages coming in that need to be processed in some form or fashion. To expect the back-end service to support that is crazy. Instead, we dump them into Kafka to give our data infrastructure time to ingest them. As for replaying events, sometimes the ingestion pipeline fails and drops some messages. I know - that's a huge mistake on our engineering team's part - but when it does happen Kafka has the ability to rewind and replay messages, resulting in delayed processing but no data loss.
  • Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
  • Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
  • Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
  • Doesn't work well with many small topics (on the order of thousands). There is a physical limit due to file handler usage on the number of topics Kafka can have before it grinds to a halt. This is not an issue for most people but it became an issue for us, as we need to have many, many topics and so we weren't able to fully migrate to Kafka except for a few of our big queues.
  • Lack of tenant isolation: if a partition on one node starts to lag on consume or publish, then all the partitions on that node will start to lag. That's what we've noticed and it's really frustrating to our customers that another customer's bad data affects them as well.
  • I don't have tooo much experience here, but I hear from other engineers on my team that the CLI admin tool is a real pain to use. For example, they say the arguments have no clear naming convention so they are hard to memorize and sometime you have to pass in undocumented properties.
Despite the disadvantages I list, I really believe that Kafka is the right choice whenever you need a queueing or message broker system. Kafka is way too battle-tested and scales too well to ever not consider it. The only exception is if your use case requires many, many small topics. Also, Kafka doesn't support delay queues out of the box and so you will need to "hack" it through special code on the consumer side.
January 30, 2019

Kafka quick queue

Score 8 out of 10
Vetted Review
Verified User
We are using Kafka as an ingress and egress queue for data being saved into a big data system. Kafka is also being used as a queue for frontend applications to use in order to retrieve data and analytics from MapR and HortonWorks.
  • Fast queuing
  • Easy to set up and configure
  • Easy to add and remove queues
  • User interface for configuration could be a little better
  • Could be a little more defined when configuring files
  • Logging is a little hard to follow
If you need a queue for ingest or user interfaces Kafka is a great tool. Easy on the admins as well as the developers.
Juan Francisco Tavira | TrustRadius Reviewer
Score 9 out of 10
Vetted Review
Verified User
Apache Kafka is becoming the new standard for messaging at our organization. Originally we limited the use to big data environments and projects but as the technology is becoming more mature we think it will eventually replace classical messaging software.
  • High volume/performance throughput environments
  • Low latency projects
  • Multiple consumers for the same data, reprocessing, long-lasting information
  • Still a bit inmature, some clients have required recoding in the last few versions
  • New feaures coming very fast, several upgrades a year may be required
  • Not many commercial companies provide support
Apache Kafka is extremely well suited in near real-time scenarios, high volume or multi-location projects. It can solve escalation problems for a fraction of the cost other solutions do and it has the flexibility of open source scenarios.
Return to navigation