Format-Preserving Permutation Replaces Uniqueness Checks for Duplicate-Free Data Masking in PostgreSQL

A team developing a data masking service for PostgreSQL and SQL Server has replaced random or hash-based replacement with keyed permutations built on Feistel networks. The approach guarantees uniqueness within each format class without checking the target table, preserves referential integrity across tables, and remains fully deterministic and reproducible. Values are decomposed into fixed, variable, and checksum fields; only the variable portion is permuted inside its numeric range using an eight-round Feistel construction with AES as the round function, followed by cycle-walking when the result exceeds the original domain size. The method supports mixed-radix classes such as IBAN, SNILS, IMEI, and email addresses while keeping control digits valid. Because the construction is non-standard, the authors explicitly warn that it provides pseudonymization rather than irreversible anonymization.

HabrTranslated from Russian

Read at source

A team of developers building a production data-masking service for PostgreSQL and SQL Server has published a detailed account of how they solved the uniqueness problem that plagues conventional masking scripts. Instead of generating random or hashed substitutes and then retrying on collisions, they construct a keyed bijection over each format class so that distinct source values always map to distinct target values without any lookup in the destination table.

The classic random-replacement pattern UPDATE clients SET phone = '+7' || lpad((random()*1e10)::bigint::text,10,'0') produces roughly n²/2m collisions on a column of n rows drawn from a domain of size m. For Russian mobile numbers this yields hundreds of duplicates on a million-row table and tens of thousands on ten million rows, immediately breaking unique indexes. Hash-based deterministic replacement avoids cross-table inconsistency and non-reproducibility but still collides once the hash is truncated to the required length.

The chosen solution is a balanced Feistel network operating on the integer interval [0,N). The smallest even bit width 2h such that 2^(2h) ≥ N is selected; each value is split into two h-bit halves that are mixed through eight rounds of L,R → R, L ⊕ AES_k(round‖R). Because every round is invertible, the entire network is a permutation of [0,2^(2h)). When the result lies outside [0,N), the same permutation is reapplied (cycle-walking) until a valid value is obtained; the expected number of iterations for telephone numbers is only 1.07.

Format classes are defined so that fixed prefixes (country code, first digit of area code, card scheme, TAC) are copied unchanged while the remaining digits are concatenated into a single integer, permuted, and written back. Check digits (SNILS, IBAN mod-97, Luhn) are recomputed from the new body, preserving validity for downstream applications. Mixed-radix alphabets (letters and digits in IBAN or email local parts) are handled by treating the entire variable segment as one integer in a mixed base before permutation.

The implementation re-uses AES-128 in ECB mode purely as a fast pseudorandom function; a single masked value costs less than one microsecond on modern hardware, allowing the full read-mask-write pipeline to sustain 300 000 rows per second. Separate 256-bit keys are derived via SHA-256 for each data type so that identical numeric strings belonging to different classes (phone versus SNILS) receive unrelated mappings.

The authors note that the construction deliberately deviates from NIST SP 800-38G FF1 because FF1 cannot directly handle mixed-radix or very small domains and imposes a higher round count. They also caution that the scheme provides only pseudonymization: given the key, every mapping can be reversed, and therefore it does not satisfy requirements for irreversible anonymization.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Privacy & Surveillance

Personal AI Assistants Build Detailed Profiles on Friends and Family Raising Privacy Concerns
AmneziaWG Container for Mikrotik Routers Achieves Major Performance and Feature Upgrades Six Months On
Samsung Bixby Sends Data to Cloud Despite Galaxy AI On-Device Processing Setting