Confluent Introduces Stream Governance Advanced to Securely Extend the Power of Data Streaming

Confluent recently announced new enhancements to its Flow governance product that will improve the ability of engineering teams to discover, understand and trust real-time data. Organizations can use Advanced flow governance to more easily solve problems within complex pipelines with point-in-time traceability, discover and understand topics faster with enterprise metadata, and enforce global quality checks with Schema Registry.

Chad Verbowski, senior vice president of engineering, Confluent, said in a Press release:

Businesses rely heavily on real-time data to make fast and informed decisions. It is therefore essential that the right teams have quick access to reliable data. With Stream Governance, organizations can understand the full extent of the streams flowing through their enterprise so they can quickly leverage that data to power endless use cases.

To learn more about Stream Governance Advanced, InfoQ contacted David AraujoSenior Product Manager at Confluent.

InfoQ: Where does the Stream Governance Advanced level come from? Was it created based on customer feedback?

David Araujo: Stream Governance Advanced was built following the launch of our governance suite in September 2021. Widespread adoption of Stream Governance features has proven it to be a critical business need. Yet customer feedback guided us where and how we needed to continue to evolve the product. Customers looking to expand their use of Apache Kafka and real-time data streaming for even more sophisticated and mission-critical use cases depended on quality and visibility/discovery tools that could scale the enterprise deployment.

InfoQ: What are the clear use cases for Stream Governance Advanced?

Araujo: Stream Governance Advanced provides enterprise-grade data governance and visibility for production workloads, enabling organizations to:

  • Confidently manage mission-critical workloads at any scale with a new 99.95% uptime SLA for Schema Registry available in 28 regions worldwide (Stream Quality)
    • The 99.95% availability SLA for Schema Registry (new for Advanced) enables teams to offload more workloads to the cloud with high certainty that data quality checks for Apache Kafka will be highly available, which which is especially useful for teams self-managing open-source deployments of Kafka that can transition to a highly available, fully managed service.
    • Support for Schema Registry in 28 regions worldwide (extended for Advanced) allows teams to optimize performance on the data streaming platform and further establish data sovereignty with schemas closer to their corresponding Kafka clusters or from any available region of their choice.

  • Improve data discovery in your streaming catalog with user-generated business context and simple declarative search through the GraphQL (Stream Catalog) API
    • Enterprise metadata (new for Advanced) allows individual users to add custom and open details to platform objects such as a topic to help other users understand which team owns the topic, how it is used, who to contact with questions about the data, or any other details they deem necessary.
    • The GraphQL API (new for Advanced) allows users to take advantage of the graphical nature of the Stream Catalog, which is modeled as a graph of entities and relationships, and provides them with a more natural, efficient, and productive way to explore the catalog programmatically.

  • Simplify understanding and troubleshooting complex data streams with lineage research and point-in-time historical information (Stream Lineage)
    • Point lineage (new for Advanced) gives users a look into the past, allowing them to see data flow developments over 24 hours or any one hour window over a 7 day range to answer to questions such as, “What happened to the pipeline on Friday at 3 p.m. when the support tickets started coming in?” or “What was this pipeline like last week when my manager seemed more satisfied with the setup?”
    • Lineage Graph Search (new for Advanced) helps users save time during development or investigation by finding specific objects such as customer IDs or buried topics with complex data pipelines.

InfoQ: Is there full feature parity between the new advanced tier and the essential tier? What are the differences?

Araujo: All features included in the Essentials level are included in Advanced. Stream Governance Advanced introduces new features for Stream Catalog and Stream Lineage not found in Essentials, as well as higher limits, expanded regional coverage, and increased SLA for Schema Registry (stream quality). All the details side by side here.

InfoQ: How does Stream Governance pricing work now?

Araujo: Stream Governance Essentials is available free to all Confluent Cloud customers for up to 1,000 schemas per environment, after which schemas are billed at a rate of $0.002/schema/hour.
Stream Governance Advanced is priced at $1/hour/environment with support for up to 20,000 total schemas per environment.

InfoQ: What is the roadmap for the future of flow governance?

Araujo: Since its launch in 2021, Stream Governance has become a suite of essential features for Confluent customers looking to securely scale data streaming deployments in the cloud to meet growing customer expectations for “all the time real “. As such, we continue to invest heavily in developing more governance features across Confluent’s data streaming platform to simplify customers’ additional efforts to move real-time data into their entire tech stack and achieve this goal.

Helen D. Jessen