Kafka as a target

Use the Kafka Target operator to publish processed Mediation events to a Kafka topic.

Use the Kafka Target operator when you want to publish processed events from a Mediation pipeline into Kafka.

Typical use cases include sending enriched usage records to downstream systems, publishing billing-ready events, streaming data to analytics platforms, and integrating with external applications.

Meter volume limits

Configuration

Configure the following fields for the Kafka Target operator:

Name - Enter a user-defined name for the Kafka target.

Example:

Kafka-Target

Event Definition - Select the Event Definition that describes the records being published.

Example Event Definition:

accountNumber: String
totalQuantity: Number
chargeName: String

Connection - Select the Kafka connection that points to the destination Kafka cluster.

Example:

prod-kafka

Topic - Specify the Kafka topic where the target publishes records.

Example:

rated-usage-events

Data Format - Select how records are serialized before they are written to Kafka:

• JSON publishes JSON messages.
• AVRO publishes Avro messages.

Key Fields - Use Key Fields to define the Kafka message key. Kafka uses the message key to decide which partition receives a record. Records with the same key are routed to the same partition, which helps preserve ordering for related events. When you select multiple key fields, their values are combined in the order selected, with an underscore (_) between the values.

Example: Account-based partitioning

Select accountNumber as the key field.

Input event:

{
  "accountNumber": "A100",
  "chargeName": "SMS",
  "totalQuantity": 500
}

Generated key:

A100

All records for account A100 are routed consistently to the same Kafka partition.

Example: Subscription-based partitioning

Select subscriptionNumber as the key field.

Input event:

{
  "subscriptionNumber": "S12345",
  "totalQuantity": 250
}

Generated key:

S12345

All records for subscription S12345 are written to the same partition, preserving event order.

Example: Multiple key fields

Select accountNumber and chargeName as the key fields.

Input event:

{
  "accountNumber": "A100",
  "chargeName": "SMS"
}

Generated key:

A100_SMS

Partitioning uses the combined values of both fields.

Example: Kafka target configuration

Input event:

{
  "accountNumber": "A100",
  "subscriptionNumber": "S100",
  "totalQuantity": 500,
  "ratedAmount": 25.00
}

Example Kafka target configuration:

Connection: prod-kafka
Topic: rated-usage-events
Data Format: JSON
Key Fields: subscriptionNumber

Published Kafka message key:

S100

Published Kafka message value:

{
  "accountNumber": "A100",
  "subscriptionNumber": "S100",
  "totalQuantity": 500,
  "ratedAmount": 25.00
}

Best practices

• Configure Key Fields whenever ordering is important.
• Use stable business identifiers such as accountNumber, subscriptionNumber, or customerId.
• Avoid highly repetitive values that may concentrate traffic into a single partition.
• Use the same Data Format expected by downstream consumers.