dedupe · Docs - Routecraft

← All operations

dedupe(options?: DedupeOptions): RouteBuilder<Current>

Suppress duplicate exchanges based on a key. Duplicate exchanges do not continue downstream - no result is returned and no side effects occur.

Mental model: A persistent, stateful filter. Similar to filter, but maintains state across runs to track which keys have been processed.

// Default: key derived from body hash
craft()
  .id('event-processor')
  .from(eventSource())
  .dedupe() // Skip duplicate events based on body content
  .process(handleEvent)
  .to(destination)

// Explicit key function for stable identity
craft()
  .id('file-processor')
  .from(fileWatcher())
  .dedupe({ key: e => e.headers[HeadersKeys.FILE_CONTENT_HASH] as string })
  .process(expensiveProcessing) // Skip files already processed
  .to(destination)

Options:

key (optional) - Function to derive the deduplication key from the exchange. If omitted, a key is derived by hashing the exchange body. See default key derivation.

Semantics:

Key is reserved immediately (single-flight behavior)
If the key is already reserved or committed, the exchange is dropped
Key is committed only after the full route completes successfully
On failure, the reservation is released or expires

Purpose:

Skip unchanged files
Prevent duplicate work
Prevent duplicate side effects

Note

dedupe vs filter vs cache

filter is stateless - each exchange is evaluated independently based on a predicate. dedupe is stateful across runs - duplicates are dropped entirely. cache is also stateful across runs - duplicates return the cached result instead of being dropped.

Use dedupe when duplicates should do nothing. Use cache when duplicates should return the same result.

Default key derivation:

When dedupe or cache is called without a keyFn, a key is derived automatically by hashing the exchange body:

key = sha256(encode(body))

The key is computed from the body at the moment the operation executes. If the body changes at different points in the route, the derived key will differ.

Supported body types:

Type	Encoding
`Buffer`, `Uint8Array`, `ArrayBuffer`	Hash raw bytes directly
`string`	UTF-8 encode, then hash
Object or array	Canonicalize (sort keys lexicographically at every level), then hash as JSON
Scalars (`string`, `boolean`, `null`, finite `number`)	Hash as JSON

Unsupported types (will throw an error):

NaN, Infinity, -Infinity
Functions, symbols, BigInt
Date or class instances (unless pre-converted to JSON-safe primitives)
Circular references
Streams (must be materialized to bytes/string/JSON first, or provide a keyFn)

When the body contains an unsupported type, a RoutecraftError is thrown indicating that a keyFn is required.

Note

When to provide a keyFn

Use an explicit keyFn when you need stable identity across body changes. For example, if the body is enriched or transformed before dedupe/cache, but identity should be based on a header set earlier by an adapter.