> This page is for Docs.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://zuora.ferndocs.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://zuora.ferndocs.com/_mcp/server.

# Kafka as a source

> Use the Kafka Source operator to read messages from a Kafka topic and ingest them into a Mediation pipeline.

Use the Kafka Source operator to read messages from a Kafka topic and ingest them into a Mediation pipeline.

The Kafka Source operator subscribes to a Kafka topic, reads messages, converts them into Mediation events, and sends them through downstream operators such as transformers, enrichments, aggregators, and targets. Kafka sources are commonly used for real-time usage ingestion, event-driven integrations, streaming analytics, data lake and warehouse integrations, and usage-based billing workflows.

## Meter volume limits

<table>
  <tr>
    <th />

    <th>
      API Sandbox
    </th>

    <th>
      Zuora Developer Sandbox
    </th>

    <th>
      Zuora Central Sandbox
    </th>
  </tr>

  <tr>
    <td>
      Kafka
    </td>

    <td>
      * Maximum message payload is approximately 6 MB (tenant configurable)
      * No Kafka throughput throttling (messages-per-minute) limit is implemented
    </td>

    <td>
      The same as API Sandbox
    </td>

    <td>
      Same as API Sandbox. No environment-specific limits exist.
    </td>
  </tr>
</table>

Production follows the same limits as Central Sandbox.

* Message validation uses the tenant-configurable `payloadMaxLength` setting, whose default value is approximately 5 MB.
* Kafka consumer defaults:

  * `fetch.min.bytes = 131072`
  * `fetch.max.wait.ms = 500`
  * `max.poll.records = 1000`

## Configuration

Configure the following fields for the Kafka Source operator:

**Name** - Enter a user-defined name for the Kafka source.

Example:

```
Kafka-Source
```

**Event Definition** - Select the Event Definition that describes the structure of incoming Kafka messages. The Event Definition determines the available fields in the pipeline, data types, and validation rules.

Example Event Definition:

```
accountNumber: String
subscriptionNumber: String
quantity: Number
eventDate: DateTime
```

**Connection** - Select the Kafka connection that contains connectivity and authentication details for your Kafka cluster.

Example:

```
stage-kafka
```

**Topic** - Specify the Kafka topic from which the source consumes messages.

Example:

```
usage-events
```

**Data Format** - Select how the Kafka messages are interpreted:

* **JSON** reads messages as JSON documents.
* **AVRO** reads messages as Avro records.
* **RAW_STRING** reads messages as plain text.

**Offset Reset Strategy** - Select where the source starts reading when no previously processed position exists:

* **Latest** starts reading newly arriving messages. For example, if a topic already contains 10,000 messages, a new source configured with **Latest** starts with the next message that arrives after the source starts. Use this option for real-time processing and production streaming workloads.
* **Earliest** starts reading from the oldest available message. For example, if a topic contains 10,000 existing messages, a source configured with **Earliest** processes those messages. Use this option for historical imports, initial onboarding, and backfill scenarios.

## Examples

Example incoming JSON message:

```json
{
  "accountNumber": "A100",
  "subscriptionNumber": "S100",
  "quantity": 500,
  "eventDate": "2026-06-01T12:00:00Z"
}
```

Example Kafka Source configuration:

```
Connection: stage-kafka
Topic: usage-events
Data Format: JSON
Offset Reset Strategy: Latest
```

The message is converted into a Mediation event and made available to downstream operators.

## Best practices

* Use **Latest** for continuous production ingestion.
* Use **Earliest** for historical imports and backfills.
* Ensure that the selected Event Definition matches the incoming message structure.
* Use **AVRO** when your Kafka ecosystem relies on schema-managed data.