IdentityGuard: Disrupting Both Identity Aggregation and Binding against Diffusion-based Personalization

Diffusion-based personalization brings convenience to users in text-to-image generation but it also poses risks of rights infringement and content misuse. To address this issue, researchers have proposed several proactive defense methods by adversarial attacks. However, most of these methods directly attack the noise prediction results during the fine-tuning process, overlooking the unique characteristics of diffusion-based personalization, which results in limited defense performance. Therefore, this paper summarizes the two core tasks of personalized fine-tuning as identity aggregation and identity binding, and proposes a defense method named IdentityGuard to specifically attack these two core tasks. The IdentityGuard designs a training sample decorrelation (TSD) attack and a text-image decoupling (TID) attack respectively for the two core tasks. The TSD attack disrupts the learning of common features by reducing the correlations among training samples. The TID attack targets all tokens by using the value-inverted attention map of each token as its adaptive target, aiming to suppress high-attention regions and strengthen low-attention regions. In addition, a token-level adaptive weighting strategy is designed to dynamically allocate attack weights across different tokens during fine-tuning. Experimental results demonstrate that the IdentityGuard effectively enhances proactive defense performance against diffusion-based personalization, achieving an average improvement of 22.26% in terms of Identity Score Matching (ISM) metric compared to the state-of-the-art (SOTA) methods. The source code is available at https://github.com/imagecbj/IdentityGuard © 1991-2012 IEEE.