Kafka Notifier

The Kafka Notifier is available in v38.1.0 and later.

The Kafka Notifier allows you to send submission notifications to the Kafka topic and partition of your choice. You can configure it to send notifications synchronously with each submission or in batches asynchronously. It also gives you the flexibility to apply any of Kafka’s producer configurations to the delivery of your notifications.

Sample use cases

I want to send data extracted from contracts to specific Kafka topics, with a specific customer group assigned to each topic (requires routing tags and multiple Kafka Notifiers).
I have transformed submission data that I want to send to a Kafka topic, where additional processing will be applied to it before it’s sent to another topic.

Using the Kafka Notifier in SaaS deployments

When using the Kafka Listener in a SaaS deployment of Hyperscience, you need to ensure that Hyperscience has access to your Kafka brokers. The way to do that depends on where your Kafka brokers are located.

If your Kafka brokers are located on..	…then:
A public interface	Ensure that Hyperscience has access to each Kafka broker through the configured port (e.g., opening the firewall for inbound and outbound traffic to that port).
A private network	Enable additional listeners on each of the brokers' instances, and create a way for Hyperscience to reach each of those listeners. For example, you can do so in the following ways: Add a public interface on each of the Kafka instances, and add a Kafka listener on the public interface:port combination, as well as a Kafka advertised listener on the same interface:port. Add a single public interface that routes to each of the Kafka brokers, each on a different port as the existing interface:port combinations configured. Each of the Kafka brokers should then be configured with advertised listeners on the same public interface:port combination routing to the new private interface:port listeners.

If your Kafka brokers are located on..

…then:

A public interface

Ensure that Hyperscience has access to each Kafka broker through the configured port (e.g., opening the firewall for inbound and outbound traffic to that port).

A private network

Enable additional listeners on each of the brokers' instances, and create a way for Hyperscience to reach each of those listeners. For example, you can do so in the following ways:

Add a public interface on each of the Kafka instances, and add a Kafka listener on the public interface:port combination, as well as a Kafka advertised listener on the same interface:port.
Add a single public interface that routes to each of the Kafka brokers, each on a different port as the existing interface:port combinations configured. Each of the Kafka brokers should then be configured with advertised listeners on the same public interface:port combination routing to the new private interface:port listeners.

Contact your Hyperscience representative for more information.

Block settings table

In addition to the settings outlined below, you can also configure the settings described in Universal Integration Block Settings.

Connection settings

Name	Required?	Description
Bootstrap Servers	Yes	Comma-separated list of bootstrap servers the Kafka Notifier sends messages to. Each bootstrap server is a host/port pair the Notifier uses for establishing the initial connection to the Kafka cluster.
Topic	Yes	Name of the topic that will receive the notification’s messages
Client ID	Yes	The client ID that will be used for all producers created for this output connection. It is used mainly for monitoring purposes. If none is provided, a random client ID will be used.
Producer Config (JSON)	No	The Producer Config settings for the connection (e.g., authentication settings) in JSON format. The Kafka Notifier supports all of the settings listed in Apache Kafka’s Producer Configs documentation.
Sync Publish	No	Determines whether the notifier sends a message with each submission or in batches. Specifically, it indicates whether the Kafka producer should block until an acknowledgement is received for each published message. If enabled, a message is sent for each submission. The submission is given a Completed or Halted status based on whether the message was sent successfully. We can guarantee at-most-once delivery if you include acks=all in Product Config (JSON). If disabled, messages are sent in batches. We can guarantee only at-most-once delivery semantics. Enabled by default.
Message Key	No	Determines which topic partition messages are sent to

Specifying Producer Config settings in the “.env” file

You have the option of specifying the Producer Config settings in the “.env” file rather than in the block settings. When doing so, add “HS_KAFKA_PRODUCER_CONFIG_” before the setting name (e.g., ssl.key.password becomes HS_KAFKA_PRODUCER_CONFIG_ssl.key.password).

You do not need to specify Producer Config settings in both the block settings and “.env” file. If the same settings are specified with different values in the block settings and the “.env” file, the values in the block settings are used.

Guaranteeing exactly-once delivery

To prevent duplicate messages—and to make the producer idempotent—configure the Kafka Notifier as follows:

Producer Config (JSON) — Include the following key-value pairs:

enable.idempotency=true,
acks=all,
retries=2137483647,
max.in.flight.requests.per.connection <= 5

Message Key — Assign a message key to all messages. We recommend ${workflow.run.correlation_id}, which retrieves the correlation ID for the flow run and enters it in the Message Key setting. The system assigns a unique correlation ID to all the flow runs executed for a submission.

Health checks

The health check for the Kafka Notifier consists of the following verifications:

Verify the reachability of the server by sending a list_topics request (see Confluent's confluent_kafka API for more information).
Verify that all configured topics exist in the configured Kafka server.

If all of these verifications pass, the connection is considered healthy.

These verifications require that the user assigned to the Kafka Notifier have describe and read permissions on the configured topics.