/* Bot Spam and IP Bans */

Discuss general Uzebox topics here: features, wish list. nice to have, etc.
User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Sat May 27, 2017 4:42 pm

This morning I used the program to delete 46 spammy topics. This took barely 2 minutes.

Only 10 or so hours prior I cleared the spam. So, 46 in 10 hours. Wow!

I only encountered one little bug where the program would glitch out. This does not happen often and I can probably fix it. In the meantime I just reloaded the code and it all worked again and I took out the remaining spam topics.

If I used something like setInterval (or setTimeout) I could set this thing to scan every 5-10 minutes and just delete the topics that match the flag1 (spammy words in the topic subject.) This would leave behind topics that may be spammy. An 70-80% auto deletion rate is still pretty good and I may even be able to improve the detection over time.

Moderators: Please contact D3thAdd3r or myself (nicksen782.) This project is going to be important at least until uze6666 gets back to the forums.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: /*************************************Bot Spam and IP Bans*/

Post by D3thAdd3r » Sat May 27, 2017 4:58 pm

Awesome, great work!

I second the 5-10 minute scan as it seems safe to me, and although I believe I have not deleted any good posts, still I probably trust a string scan over possible human error(when that human is in a rush!).

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Sat May 27, 2017 5:26 pm

Yeah, the original version just had a delete button for individual threads with no confirm. Way too easy to mess up. Now with the checkboxes it is less scary.

The code works but I am reviewing it now and adding comments and looking for bugs that I may have missed. Also, I want to fix the XHR sync issue to be async (unfortunate quickfix on my part. I'll fix it.)

And I'll put the auto delete timer on too. It is super easy to update your instance of it. Just click the "Update App" button and it will re-download the JavaScript file.

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Sun May 28, 2017 12:20 am

Here is an update for the anti-spam tool:

Proper async with the multiple form submission. There is also now a count down status bar in the corner that will show you the progress.

Currently there are 3 types of record checks. The first is for "spammy words". These are words we will never use in a topic subject. Second, is if a topic is in the "Off Topic" forum. I flag this visually for right now. Third there is a "Trusted Member" filter. I took anybody who had more than 25 posts. This list can be updated easily. If this is detected then the deletion checkbox is not displayed and instead you see an image of a shield. This should provide quite a bit of protection against accidental deletion.

Next up are the Auto Check features. Auto Check 1 will automatically check the topics that have "spammy words". I have one for the "Off Topic" topics but I've disabled for now.

The delete button will delete all records that are checked (after a confirmation prompt of course.) Then you see the progress bar. The number of topics that are able to be deleted per round of deletions are limited by the number of records displayed in the iframe on the right. So, you can delete 25 per batch.

Automatic Spam deletion: The idea is to test if a certain amount of time has passed (5 minutes? 10 minutes?) and then click the Auto Check1 then the Delete button. I'm still building that.
Attachments
ihs_2.png
ihs_2.png (42.02 KiB) Viewed 3604 times

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Sun May 28, 2017 1:38 am

I have the auto delete functionality working! It defaults to 5 minutes and you can change it if you want. Each time the countdown reaches 0 it will take the minutes value and multiply it by 60 to get the seconds and then continue counting down. If you choose the "NO" radio button then the auto delete feature becomes deactivated.

I have it set to auto delete the "spammy words" topics. So, it may miss a few but not too many. I will add to the spammy words filter if I can and as I see topics that are missed by it.

I'm going to leave this on for an all night test. So, hey! Awesome!

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: /*************************************Bot Spam and IP Bans*/

Post by D3thAdd3r » Sun May 28, 2017 2:15 am

Looking good! Man you have no idea how good that feels to check on the forums after a couple hours and not see it overrun!!!

You have this thing, there is not even much feedback I can give since you have things under control and improving quickly. The only addition I can think of that is probably worth worth adding is the words "levitra" and "cialis" are the most common words used, in those posts not containing "viagra". Again I don't think it is any issue to imagine that any posts talking about boring pharmaceutical stuff are indeed spam.

In the past the bots used to recycle old posts from the forums, but I have not seen that for quite a while luckily as sometimes it was hard to tell for me on a cursory examination. Also, I have not seen for a couple weeks what they used to do, which is spam in Cyrillic text where they did not contain English drug names, but had links to spam. These were ones where they would actually make several replies to the thread, and then it might be 10 posts to delete instead of 1, or sometimes done on an old thread. That might be harder to figure out, as I do not think deleting something just because it contains Cyrillic is any more acceptable than deleting the several instances of German language usage in the past on these forums, in situations where it was acceptable(titles, articles, event names, short replies, etc). Probably there is something else those posts have, if they come back, which could be used as a means to filter...then it seems like your tool would be able to keep everything in check?! :shock:

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Sun May 28, 2017 4:57 am

Thank you! I have a separate browser window dedicated to it now and it can run 24/7. (Separate browser too.)

The automatic delete can clean out quite a bit and the app will not allow you to delete topics from a known good (and long) list of members either. It is safe to use. It may miss things occasionally but otherwise, I see a 90%+ reduction as a great thing!

Lee, are you the only member with moderator privileges?

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: /*************************************Bot Spam and IP Bans*/

Post by D3thAdd3r » Sun May 28, 2017 6:34 am

Harty123 is the only other one I am aware of. I looked at the logs from 2009 onwards, and besides what looks like members deleting their own posts, it seemed to be the case.

Probably 100+ posts I did not have to delete today! I wonder for the few remaining ones I see at the moment, why they don't trigger as spam? Not that I am complaining :D

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Wed May 31, 2017 11:20 am

I'll be improving the spam filter in time.

I didn't have the auto delete on last night and had to clear out a couple pages of spam as a result. Bulk deletions are easy now but still, opps.

I'm thinking that I should just be checking for topics whose last post are not from a known member and then just check that post and delete just the post. If that is the only post then the topic disappears too. If I am just deleting the last post (even if repeatedly) I could be a bit more harsh with the spam filter.

Progress continues. ... And I'll have some game updates for you all soon.

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: /*************************************Bot Spam and IP Bans*/

Post by nicksen782 » Thu Jun 01, 2017 2:53 pm

I find it funny that CurtisTems had posted to THIS topic. That account is one of at least 2 accounts that post to existing topics. (And I've added that IP address to the anti-span bot's banned IP ranges.) I've learned a lot about forum spam.

I dunno why the member ban and IP ban moderator features of phpBB do not work.

But now, my anti-spam bot is aware of these IP ranges. If the post details show that the user posted from one of those ranges then "POW", it gets deleted. (Entire spam networks, not just single IPs.) I don't even check for spammy words if the IP ban check comes back with a match.

The bot now handles topic spam and post spam. I'll be letting a few more spams through today for testing but then I expect that this solution will be complete.

Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests