Skip to navigation
ExtensibilityMediationMeter ComponentsProcessorsDeduplicator processor

Configure Deduplicator as a processor

View as Markdown

Use the Deduplicator processor to identify duplicate event data in your Mediation pipeline. When the Deduplicator identifies a duplicate record, the record is marked as an error and discarded.

1

Navigate to Mediation > Meters

2

Create a Custom meter.

3

Select Deduplicator as the processor.

The Deduplicator settings page is displayed.

Deduplicator processor settings

4

In Name, enter a descriptive name for the Deduplicator component.

The Operator ID is a system‑generated, non-editable identifier for this operator, used in audit and API integrations.

5

In Method, choose one of the following options:

  • Exact match (All Fields) to compare the entire record.
  • Specific Fields match to compare only the fields that you select.
6

If you select Specific Fields match, select the fields that the Deduplicator must use to identify duplicates.

7

In Reference Time, choose one of the following options:

  • Processing Time to use the time when the event is received by the system.
  • Event Time to use a timestamp field from the event payload.
8

If you select Event Time, complete the following fields:

  • Event Time Field: Select the field that contains the timestamp value.
  • Time Format: Enter the format used by the timestamp value.

Event Time determines which calendar window is used for duplicate checking. Deduplication state is retained only for a limited period and is not kept forever. By default, the system keeps deduplication state for 60 days.

9

In Deduplication Strategy, choose one of the following options:

  • Rolling Duration to start a time window when the first event arrives and keep it open for the configured duration.
  • Fixed Calendar Window to group events by calendar boundaries such as hour, day, week, or month.
10

If you select Rolling Duration, complete the following fields:

  • Enter the duration value.
  • Select the duration unit.
  • Optional: Select Reset window on duplicate if duplicate arrivals must extend the deduplication window
11

If you select Fixed Calendar Window, complete the following fields:

  • Unit: Select the calendar boundary to use.
  • Timezone: Select the timezone used to calculate the calendar boundary.
12

Click Save to save the deduplicator processor settings.

  • The default configuration is Processing Time with Rolling Duration and a 30-day window.
  • Rolling Duration is supported only with Processing Time.
  • In batch meters, deduplication is still limited to the current batch run.
  • In batch meters, Fixed Calendar Window can still partition records within the batch.