Skip to navigation
ExtensibilityMediationMeter ComponentsProcessorsDeduplicator processor

Deduplicator behavior in batch and streaming meters

View as Markdown

The Deduplicator behaves differently in streaming meters and batch meters.

UI option or combinationStreaming modeBatch modeBehaviorExamples/Notes
Processing Time + Rolling DurationSupportedSupportedRecords are remembered for a moving duration based on arrival time.Ignore duplicate events for 24h after arrival.
Processing Time + Fixed Calendar WindowSupportedSupportedDeduplication resets at fixed calendar boundaries (hour/day/week/month).Allow one record per customer per day.
Event Time + Rolling DurationNot supportedNot supportedThis combination is unsupported.Validation rejects this configuration.
Event Time + Fixed Calendar WindowSupportedSupportedUses timestamp inside payload instead of arrival time.Best for replay/backfill imports.
Deduplication across multiple runsSupportedNot supportedStreaming persists state across executions; batch clears state after completion.Streaming blocks replays across days. Batch only within the current run.
Late events handlingSupportedSupportedLate-arriving records are assigned to the correct event-time window.Yesterday’s delayed event still goes into yesterday’s window.
Replay protectionStrong supportLimited supportStreaming continuously protects against replays. Batch protection ends with the job.Reprocessed files are ignored in streaming but not across batch reruns.
Event Time state retentionRetained for the configured state lifetimeRetained only within the current batch runEvent Time determines window assignment, but deduplication state is not kept forever.By default, persistent state is kept for 60 days. In batch mode, deduplication state is cleared when the batch run ends.

Streaming meters

In streaming meters:

  • The pipeline runs continuously.
  • Deduplication state persists over time.
  • Time windows and retention settings are meaningful across events.
  • A duplicate arriving days later can still be detected.

Use streaming meters when you need replay protection across days or weeks.

Batch meters

In batch meters:

  • The job processes a finite set of records.
  • Deduplication applies only within the current batch run.
  • Deduplication state is cleared when the batch run ends.

Use batch meters when deduplication is required only within the current import or batch file.

Rolling Duration in batch meters

Rolling Duration is supported in batch meters, but the retention settings do not have practical effect after processing completes. During the batch run, records are still compared against other records in the same batch.

Fixed Calendar Window in batch meters

Calendar windows still matter in batch meters. Records can still be partitioned by hour, day, week, or month within the same batch run.

For example, two identical records on Monday are duplicates, while the same record on Tuesday is treated as new again, even within the same batch.

Event Time in batch meters

Event Time with Fixed Calendar Window also works in batch meters. This is useful for historical imports, backfills, replay processing, and multi-day data files.

Event Time determines which calendar window is used for duplicate checking, but it does not imply unlimited historical lookup. In batch mode, deduplication state still ends with the batch run. In persistent mode, retained state is still bounded and not infinite.