I think embeddings[1] would be a nice way to create a signature. Essentially, we could dump data about a person's activities into it (words added, namespaces edited, time of day of edits, temporal frequency of editing, # of revisions per session, frequency of citation by type, etc.) and get a signature that could represent several aspects of behavior. The vectors that come out of an embedding would allow us to provide a distance measure between one editor's behavior and another editor's behavior.

That said, I think it is more likely that we would be able to match behaviors that look more like experienced editors generally than one specific editor who might be the primary account of the sock puppet. Still, this might be useful for many aspects of newcomer support and patrolling work. E.g. if a new account looks like an experienced editor, they might not need an invite to the Teahouse. In fact such an editor account may be a sock or a legitimate alternative account. On the other hand, if a newcomer account is getting reverted or warned a lot but doesn't behave like an advertiser or a POV-pusher, we probably want to reach out to them to help.

I'm really interested in investing in embedding-based strategies for tracking the topic-space of content and clustering behaviors but I don't have the resources on the Scoring Platform team[2] to do any sort of serious engineering work with embeddings right now. In the meantime, I'm interested in talking to external researchers about collaborations and possibly even short term contracts to dig into these types of modeling problems. If anyone out there is interested in that, please reach out.

In the meantime, we're working on more rudimentary AIs that can help us sort vandals from everyone else. :)

1. https://en.wikipedia.org/wiki/Embedding 2. https://www.mediawiki.org/wiki/Wikimedia_Scoring_Platform_team

-Aaron

On Fri, Aug 23, 2019 at 12:27 AM Kerry Raymond kerry.raymond@gmail.com wrote:

To reply to my own question .

Can we find a way to create a "signature" of an account's pattern of editing? Perhaps it might be a set of signatures, maybe one for the categories that the account appears to be active in, another for the type of edit, etc. Then if these signatures were calculated for all banned accounts or currently blocked accounts (or at least ones with a long enough contribution history to make it worthwhile - we're not interested in one-edit vandals), then we could have a tool that could be run to quickly compare one account against the signatures of banned/blocked accounts as well as the cumulative edits of a set of known sockpuppets (i.e. treat them as a single account) to determine if this may be a sockpuppet case meriting further investigation. I imagine that it would be too expensive computationally to actually run comparisons of the contribution histories of all "bad guy" accounts against the suspicious account which is why I propose a "signature" approach (but I'm happy to be told otherwise).

If we had such a tool and it proves reasonably reliable in identifying likely sockpuppets (not asking for guarantees but close enough not to be a waste of time to investigate), then we could routinely use it on new accounts or reactivating accounts (i.e. possible sleeper accounts) once they have a long enough editing history to enable the tool to operate effectively to provide automated early warning of new/reactivating accounts appearing suspiciously similar to "bad guy" accounts.

But this is a hard problem, both technically and socially (Assume Good Faith, Privacy, etc), so I welcome the thoughts of others.

Kerry

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l