The FLEDGE proposal explains how ATPs can leverage browser APIs to join users to, and remove users from, interest groups.

Protection is provided for the publisher to control who can add users of their site to an interest group. However, removing users from an interest group is indicated to be possible without permission of the site owner:

There is a complementary API navigator.leaveAdInterestGroup(myGroup) which looks only at myGroup.name and myGroup.owner, and which only needs permission of the group owner.

I believe this could be abused by an interest group owner to negatively tag users into an interest group and remove them when a positive signal is seen. For example, imagine I would like to collect users who read a popular news site popularNews.com but this news site chooses not to allow me to tag their users. I could tag all users I see for a couple days with interest groups I-have-seen-this-user and not-popular-news-reader across any webpage (except popularNews.com). I could then serve an ad targeted only to popularNews.com with a creative that includes the navigator.leaveAdInterestGroup('not-popular-news-reader'). I would then be able to infer that any user in I-have-seen-this-user and not in not-popular-news-reader interest groups is a reader of popularNews.com.