DataStax throws weight behind Pulsar messaging platform to kill Kafka

DataStax launched an implementation of the open source Apache Pulsar distributed messaging and streaming platform, dubbed Astra Streaming, that is integrated with its database service based on an instance of Cassandra running on top of a serverless computing platform.

Pulsar will supersede Apache Kafka software as the next-generation distributed messaging platform, said Chris Latimer, vice president of product management for DataStax. “We see it as a Kafka killer,” he said.

Kafka has gained a significant amount of traction as a distributed messaging platform that makes it easier to share large volumes of persistent data across a distributed computing platform. Instead of only being able to access data at rest, both Kafka and Pulsar enable applications to also access data in motion. Those capabilities have proven especially crucial within digital business application environments where data needs to be analyzed as close to the points where it is being created and consumed as possible.

Pulsar traces its lineage back to a distributed messaging platform created at Yahoo that advocates claim provides faster throughput and lower latency than Apache Kafka in many use cases. Pulsar is also designed to support multiple tenants, is easier to deploy, and can be employed to replace Kafka using a familiar application programming interface (API) construct. Overall, the total cost of ownership of Pulsar is less, according to a recent analysis published by the market research firm GigaOm.

DataStax is also positioning Astra Streaming, available in beta, as a natural complement to Astra DB, a database-as-a-service based on a version of Cassandra deployed on top of Kubernetes clusters accessing an object-based storage system. That instance of Cassandra has been rearchitected as a set of microservices that can more easily scale up and down as needed, according to DataStax. A key benefit of the integration of Astra Streaming and Astra DB is the ability to interact with event logs in a way that feels familiar to anyone who is experienced with database technologies, said Latimer.

It’s not clear to what degree the decision to select a database will be coupled to the acquisition of a database. DataStax, however, disclosed last month it counts CGi Global Technologies, Energisme, ESL Gaming, Innovapost, Koddi, Overstock.com, Payment Approved, Titanium Intelligent Solutions, T-Mobile, Trading Technologies International, Ubirch, US Bank, and Verizon as customers. It also received an unspecified amount of funding from Goldman Sachs.

Most enterprise IT organizations have yet to commit to Pulsar, Kafka, or any other type of distributed messaging and streaming platform. However, the awareness of the need for these types of platforms has increased considerably thanks to digital business transformation initiatives that require data to be processed in near real time. In many cases, organizations are trying to bring together data management, engineering, storage, and security under a common framework that makes it simpler to treat data as a corporate asset that can be maximized by multiple applications. At the core of that strategy is an effort to leverage AI models that require massive amounts of data to be trained and then deployed in a production environment, noted Latimer.

The challenge organizations now face is aligning all the disparate teams that once managed each of those functions around a set of best practices that are consistently implemented.

Of course, advocates of Kafka are not yet ready to roll over simply because Pulsar has emerged as an alternative. The issue, of course, is many of those same organizations are among the most inclined to adopt a new platform when it becomes stable enough to support distributed applications at scale.