cross-posted from: https://programming.dev/post/51288619

Hi All,

Due to some ongoing issues with harassment campaigns, we’ve had to setup a rudimentary monitoring system for all new users.

  • When a user’s signup is accepted, they will be automatically enrolled into the monitoring system. The admins team may also add accounts manually if they have been given a strike.
  • The system will monitor all posts, comments and DMs sent by new users, and bring them to the attention of the admin team if it appears suspicious. In egregious cases, it will auto-remove posts and comments if required, but a human admin will always review and reverse any false positives as soon as required.
  • Once we have validated that the user is not a spammer, they will be removed from the system.

We don’t want to go into too much detail on how it all works, but we can say that all the processing is being done locally on the instance server. For most of you, this wont have any impact, but some of you have been impacted by the systems false positives. It is also a good time to point out that DM messages are not private, and should not be used for anything that requires strong privacy.

There will likely be teething problems, but we are actively working on improving the bot to minimize impact and we are always open to feedback.

  • Ategon@programming.dev
    18·
    7 days ago

    Theres no LLM involved. This version of the post doesnt have the tldr to clarify some things so ill just repost my comment here

    • Were not sending data to an LLM but the method is hidden so bad actors cant bypass it (you can disagree on that but the main issue is bots that are able to check blacklisted content and then just edit messages slightly to bypass it)
    • The things that are checked are public (posts, comments, and dms that are also sent to other servers we dont control) and can be moderated on regardless of this. There isnt anything that breaches privacy
    • Worst thing it can automatically do is temporarily remove a post/comment until an admin can approve or deny it just so its not there for people to run into (for example if someone spams slurs at someone else the someone else doesnt see that instead of it being there until an admin is able to come online)
    • southsamurai@sh.itjust.works
      7·
      7 days ago

      Aight, that’s a great response :)

      What I’m taking away from that is that the method is more like what auto mod on reddit did than a separate piece of software running. That, as such, it’s something admins have full control over beyond just choosing to have it in place or not; as in an llm would be a closed box with limited ability to influence what goes on inside. And that it’s serving pretty much as a filter as opposed to an automated decision maker.

      Which is exactly what the fediverse needs tbh, assuming I’m understanding what’s being used correctly.

      With that assumption in place, my only remaining “bullshit” is in a little more transparency on exactly what it is. Not the details of keywords and such that would allow bots to easily bypass, but as in whether it’s some kind of custom built code y’all have put together, vs an externally sourced program. If it’s external, then it should definitely be made known where it comes from so that prospective users can make a decision based on that.

      • Ategon@programming.dev
        9·
        7 days ago

        yeah pretty much exactly that. It’s custom built 100% by our team with no external tools or anything (just a way to notify us using apis). it’s technically a separate running code since we can’t hook it into the base lemmy code that easily but acts the same way