Find Fuzzy Duplicates in Parquet Files Online

How is fuzzy deduplication different from exact deduplication for Parquet files?

Exact deduplication removes rows where all columns match perfectly. Fuzzy deduplication finds rows that are similar but not identical — for example "John Smith" and "Jon Smith" or "123 Main St" and "123 Main Street". It uses string similarity to identify near-matches that represent the same real-world entity.

What similarity threshold should I use when fuzzy-deduplicating a Parquet file?

Start with 0.85 (85% similarity). A lower threshold catches more duplicates but risks merging distinct records. A higher threshold is more conservative and safer when small differences are meaningful. Review the flagged pairs at your chosen threshold before committing.

Can I fuzzy-deduplicate on multiple columns in a Parquet file?

Choose the single column that best identifies the entity — typically a name, address, or description. For multi-column matching such as first name + last name together, use the Concatenate Columns tool to combine them into one column before running fuzzy deduplication.

Is my data private?

Yes — completely. Your file is never uploaded to any server. Everything runs locally in your browser using WebAssembly — processing happens entirely inside your tab. Once you close the tab, nothing is retained.

What is the maximum file size?

The free limit is 50 MB. For larger files, performance depends on your device's available memory — most modern machines handle 500 MB to 1 GB comfortably.