MemPrivacy 1.7B posts 85.97% F1
- MemPrivacy posted on X today that its 1.7B and 4B models use typed placeholders (e.g., <Phone_Number_1>) to redact PII locally for compliance use cases. - MemPrivacy reported an 85.97% F1 score on privacy extraction and said utility accuracy dropped less than 1.6% across benchmarked datasets in experiments. - Results posted May 14 on X compared 1.7B and 4B models; link to benchmark summary. (x.com)
MemPrivacy said on May 14 that its 1.7B and 4B privacy-extraction models are built to replace sensitive spans with typed placeholders on-device before anything is sent to the cloud. The examples in its public materials include tags such as `<Email_1>`, `<Health_Info_1>` and `<Recovery_Code_1>`, with the original values restored locally after the cloud response returns. (github.com) The core claim is a tradeoff claim: preserve more of the sentence’s meaning than blunt masking does, while still keeping raw personal data off remote systems. MemPrivacy’s GitHub README says the cloud side sees placeholders rather than the original values, and its arXiv paper says the approach is meant for edge-cloud agents that use long-term memory systems for personalization. (github.com) The benchmark number MemPrivacy highlighted is an 85.97% F1 score for privacy extraction. In the same set of public materials, the team said utility loss stayed within 1.6% across multiple memory systems, framing the system as an alternative to irreversible masking that can break retrieval and personalization. (arxiv.org) The architecture matters because the company is not describing a cloud-only redaction filter. Its README says detection and replacement happen on the device, the placeholder-to-original mapping is stored in a local SQLite database, and restoration also happens locally. That means the privacy boundary is supposed to sit before prompts and memory entries leave the user’s device. (github.com) The public paper adds more detail on scope. The authors — Yining Chen, Jihao Zhao, Bo Tang, Haofen Wang, Feiyu Xiong and Zhiyu Li — said they built a benchmark covering 200 users and more than 52,000 privacy instances, and introduced a four-level privacy taxonomy for configurable protection policies. The paper also says the system targets edge-cloud deployments where conversation traces may otherwise expose PII, medical or financial information, or credentials. (arxiv.org) The model lineup now visible publicly includes four Hugging Face releases in the IAAR-Shanghai collection: MemPrivacy-1.7B-SFT, MemPrivacy-1.7B-RL, MemPrivacy-4B-SFT and MemPrivacy-4B-RL. That gives outside users a way to inspect the released variants behind the May 14 benchmark claims, though the social post and repository materials do not by themselves establish how the models would perform in every production workflow. (huggingface.co) What this thread adds up to is a specific design choice in privacy tooling for AI agents: do not delete sensitive spans into `***`; swap them for typed stand-ins that preserve role and structure, then keep the lookup table local. MemPrivacy’s own materials say that is how it tries to hold onto memory utility while reducing exposure of raw personal data in cloud memory systems. (github.com) The next place to look is the project’s public paper and repository. The arXiv submission is dated May 10, 2026, and the GitHub repository and Hugging Face collection were updated this week with the benchmark summary and model releases. (arxiv.org)
