Damn Spam posts!

OK - I installed phpbb 3.10 on one of my ubuntu servers, examined the operation on the back end, plus some of the source code, and i noticed a few things about ES:

1) you seem to be running default options for the most part. You are losing out a lot of bandwidth/cpu/disk optimizations that could cut the hosting bill down.
2) phpbb's spambot countermeasures are incredibly weak, but you appear to be running a phpbb version from... 2007?
3) there are a good amount of anti-spam mods out there, after some testing i don't notice that you are running any - http://www.phpbb.com/customise/db/modifications/anti-spam-9/ Are these all crap, or has nobody experimented with them?

My offer to help still stands but it has fallen on deaf ears multiple times, even via PMing people in the past. I hope the above 3 points provide some direction for a smoother, most cost effective, and more spam free operation of the board.

Please forward it on to whoever's responsible for server admin. I noticed MRVass hasn't logged in since Dec 07, so i wasn't sure if he's still running the show or not.

-D
 
there's been a bunch of suggestions in the moderator forum, so far none of the ones that would be really helpful have been implemented (most of them are ones discussed in this thread, as well, such as the things I listed).

FWIW, the last two hours I just spent (instead of tracing out the schematic of the meanwell I need to repair, or sleeping) verifying spammers that hadn't posted yet, and banning their IPs (and emails where I could find them on stopforumspam and similar sites, since for some reason that info's not available to me as a mod, even though I get the 'ban emails" choice and it functions), then deleting them so their stupid signature links don't exist in our memberlist profiles.

That was for only 92 of them, just from the last week and a half, at most. I have yet to have had time to go back and nuke hardly any of the hundreds (probably thousands) of sleeper spammers in the memberlist.

While I was doing it, several more had registered but not yet posted, so I nuked them, too.

FWIW, I would bet a large sum of money that I don't have that 99% or more of the members on this page:
http://endless-sphere.com/forums/memberlist.php?sk=c&sd=d&start=50
that show the WWW button for their profile are spammers. Many more are probably spammers that don't use the Website field in the profile, but instead just stick them in their signatures, so you can't see them from the members list and have to actually go into each profile individually to see them, taking even more time.


I personally doubt that we have even 7500 actual members of ES, and I suspect that the rest of the 13000+ registered members are actually spammers.

I'm out of time to do this for today, so someone else will have to take up the torch for the rest of the night. :(



If we simply required moderator approval of each new member, and they had to write out an explanation of why they want to join, it would eliminate almost all of the spammers, with MUCH less work by the mods. If no other mod wanted to do this approval, I'd do it all myself. It would still take me FAR less time than what I have to do currently.

Using the various PHP mods like the ones listed in this thread/etc., the spammers even being able to register could be cut to essentially nothing, while still easily allowing in anyone that actually wanted to be here.
 
neptronix, don't give up on us! It is going to take a while to implement things. As I understand it MRVass is Justin's IT guy with the "keys to the castle." I think he will have to be the guy to make structural changes to the software. I for one don't want to have any hand in anything that might cause a server crash and/or more lost data or posts. So we have to go slowly with the right backups and MRVass would have to implement it.

Thanks for helping us with suggestions, know it is greatly appreciated!

EDIT: I copied your suggestions over onto the mod forum and was talking with Gaston, we are going to contact the chief's and see if they will make some changes to tighten things up.
 
Thanks, Moose. I really do appreciate being listened to :lol: . Thanks for passing my ideas along.

AW: I can't believe there are that many "sleeper cells". How long do those accounts lie dormant until they post? If it's over a month, a simple database query hooked up to a daily cron job could delete unused accounts over a month old. If they hang out for even longer than that... great!

Moderator pre-approval would seem to be a turnoff for new users and also require some manual labor. One forum i joined up on would corral it's new users into a newbie form until they got X amount of posts. That worked pretty well!

http://www.phpbb.com/customise/db/mod/post_count_requirements/

^-- i think this is the mod right here actually. :D

Just another idea.
 
Thanks for the advice, neptronix, an update of the forum is definitely on the todo list. I've personally been in the process of moving to a new place, so I didn't really see this issue until Ypedal e-mailed Justin and I (and for that same reason may have some slow response latency). I've been reading up and it definitely seems like forum customization is one of the most effective countermeasures against spammers, and I'm interested to see the effects of the changes made today to the new user registration process.

In the meantime, please let me know of any more recommendations you have. Anything drastic (such as a noob pool, as suggested) might require some community discussion, and anything requiring more/changed moderator effort would definitely require their sign-off. We're blessed to have such an amazing moderator team, and I think the primary focus needs to be on making their actions fewer and more effective.
 
Another 51 lurker spammers nuked, another hour+ gone. This was only teh obvious ones, not the ones I have to verify on external sites.

Now I've partially cleaned up the member list back to Jan 19th, 2012. Should only take another few hundred hours to get the last year or two of spammers cleaned up. :( But at least their accounts wont' be there for them to come back and post to, anymore.

If we could get some of the Automod tools discussed earlier installed, it would help this kind of thing a lot.
 
Thanks for your response. I don't know what your skill set is like, but if you need any help with php/sql along the way, feel free to shoot me a PM. Glad we finally have an admin here.

Amberwolf, you are amazing. You are one of the pillars that really holds this forum together. Hopefully your job will get easier soon.
 
neptronix said:
AW: I can't believe there are that many "sleeper cells". How long do those accounts lie dormant until they post? If it's over a month, a simple database query hooked up to a daily cron job could delete unused accounts over a month old. If they hang out for even longer than that... great!

There is a Prune Users funciton that would let me do that easily. BUT: there are real members out there that don't post but do read the forums.

What I think is that if we REQUIRED an intro post at the time of registration, but also had the autobanning system setup based on keywords and links and whatnot, so that spammers posting their crap would be nuked automagically, it might be practical to do autodeletion of unused accounts.

But at a guess, there are probably a few thousand zero post accounts, and I am sure some large portion of them are actually real members that read but don't post. So rather than a simple pruning of all zero-posters, we would need to do it as:
--IF post count = zero
--AND joined date = >2 days ago
--AND last visit date = >6 months ago
--THEN prune user

Another thing I'd like to do in lieu of better ideas, but would require a script written to do the notification part:
--notify all NON-BANNED members via the email they registered at ES with that a policy change in profile usage will take place that WILL result in their account removal if they do not act as notified, within one month.
--Implement a policy change that says during registration that if you put anything into the Website field of your profile, your account name and IP will be automatically banned and your account will be deleted.
--Once the notifications have gone out, and members have had the allotted time to make changes,


I would also like a tool that will generate a list of names and IPs for zero-post banned members and the "reason" field for their banning (because I choose "no spamming allowed" in that dropdown when I ban them, except for the last two days of "emergency hack'n'slash" I've been doing), and sorted by that field. Then I could copy/paste that list into the Prune Users page, and remove them completely, including all the links and crap in their profiles.


Moderator pre-approval would seem to be a turnoff for new users and also require some manual labor. One forum i joined up on would corral it's new users into a newbie form until they got X amount of posts. That worked pretty well!
If we made that forum invisible to anyone except members so that no bots can crawl it, it might work ok. However, it would then require the mods to read everything that gets posted there to determine if they are real people or not. A spammer would be able to just script it so their bot always posts at least x number of posts, and easily get around the protection, though.

For those spammers that don't go out of that forum, I guess it doesn't matter, and if it was not visible to the outside or search engines, we wouldn't even *have* to delete it. But their crap would still be in it, and that would certainly bother *me*, and time would still have to be spent cleaning it up. (or i'd go nutz knowing it was still there and I couldn't do anything about it :lol: )

Having a *single* intro "post" required that goes to the mods for preapproval would be less work, and I don't see how it would be a turn-off for new users. Many of the people coming here have projects or problems that they'd like to talk about right away, and it'd be a chance for them to talk about it very first thing without the distractions of the rest of the forum. ;)

But I think the Automod Spam Hammer stuff would obviate the need for either of the above, most of the time.

Really, though, we won't know how well any of them work until we actually TRY something. ;)


FWIW, a slightly similar method to the newbie forum is using a "honeypot" forum, invisible to the outside and only visible to a member. A warning is plastered all thru that forum "DON"T POST HERE!" and that "If you post here you'll be bahleeted and banneded instantosimultaneously and automagically", which of course a spambot isn't going to (be able to) read. Then a script can be run periodically that wipes out anything in that forum including the members that posted it.
 
[/quote]But at a guess, there are probably a few thousand zero post accounts, and I am sure some large portion of them are actually real members that read but don't post. So rather than a simple pruning of all zero-posters, we would need to do it as:
--IF post count = zero
--AND joined date = >2 days ago
--AND last visit date = >6 months ago
--THEN prune user[/quote]

Yeah, that can be done. Or even better, 1 month old and never logged into the forum other than to create the account.. :)

The honeypot idea is real smart, would require lots of coding but would be really effective. Take those IP addresses.. check for posts of the similar time.. and ka-ban.

The Intro post idea would require a lot of manual intervention. But it could be helpful.

User's IPs are stored in the phpbb database table under phpbb_users ( in my config ), you could seek and destroy all users that have the same IP ( kinda extreme measure, but it works ).. also, i dunno if you have this tool, but there would be a way to wipe out *all* posts via 1 user with 1 click using a SQL query as well.

Pretty surprised how simplistic phpbb's internals are. I've worked with worse.

There's a lot of potential here. Do check out those plugins for sure.
 
I don't think the forum should be invisible to non members I think it needs the newbs to read all they can to encourage them to join, and even if they don't join but they learn something that's what the goal is right?
 
I'm not saying make ES itself invisible to non members, just to make the honeypot forum and the new user forum invisible so taht spam there cannot be seen by the search engines and then thus attract yet more spammers.


@Neptronix: unfortunatley you can't simply see if an IP is the same between any members and use that as criteria to remove them all. Often, IPs are dynamic, and so you may get many members on the same ISP (especially celphone internet) in the same area who eventually over time have shared IPs. Iv'e seen this with several members on the board, when looking at the results under the ? button on each post for mods to get detailed info about the posting/source. Sometimes HUNDREDS of different IPs are listed for a member, occasionally nearly one IP per post! Often enough, above the listing it will show that one of those IPs was also used by another member, that I knwo is not the same person as the first one.

I have not yet seen this with *registration* IPs, but at some point it will happen simply becuase it can and probability says it will.

So...matching IPs can't be reliably used for this. :(
 
amberwolf - the IP thing i am referring to is the signup IP, or whatever the most current IP is. I'm not sure how it functions, but in the phpbb database, there is an IP address per user.

If multiple users signup with the same IP, this is a nice tipoff.
I'm not sure how mobile phone IPs work - if they're within a certain block of IPs or what.

I am clueless as to what their tactics are, i am only guessing going off what you've seen. Some SQL kung fu paired with a php interface to seek and destroy would be ultra helpful tho. It would require some collaboration.

If you want to work further, send me a PM later down the line. Until then, good luck with the modifications etc.
 
nuked another 50, lost track of time spent as I dozed off a couple times wiating for the ES server to respond to clicks (was really slow tonight). :?

Again, some had been banned already for posting, but most hadn't.

Just in the first five pages of Memberlist there are still at least 48 more I'm sure are spammers, and at least another 50-100 that probably are but I'd have to actually check. And that only goes back to Jan 12, 2012.
 
neptronix said:
amberwolf - the IP thing i am referring to is the signup IP, or whatever the most current IP is. I'm not sure how it functions, but in the phpbb database, there is an IP address per user.

If multiple users signup with the same IP, this is a nice tipoff.
I'm not sure how mobile phone IPs work - if they're within a certain block of IPs or what.
It doesn't matter which IP you're referring to; it is stll possible for actual users to have a signup IP that is the same as a spammer, so I wouldn't want to do any kind of scripted deletion based solely on IP matches. I'd want to verify first that each user is indeed a spammer.

So far, all the ones with the same signup IP have turned out to be spammers...but eventually that won't be the case. What would be terrible if we had an autodeletion script that also wipes out posts would be if one of the bigtime posters of useful information had signed up years back with a certain IP, and then later has a different one, but their original one happens to be assigned to a person or bot that signs up as a spammer.

Then we run the script and bam...it wipes out teh spammer AND the useful poster and all of their years of informative posts. I don't want any risk of that. :(

It's bad enough when i ban by IP I may end up at least temporarily blocking some real user from accessing the site, because in the future their IP may get changed to one that is banned. :(

But for now it is at least partially effective in keeping spammers away, so I have to use that tool for now, till we get better tools.
 
This just showed up from out of nowhere in the drafts folder on my yahoo acct:

I love Endless-Sphere forum.
Brand new motor, latest, greatest turns up at your door. Do you put it in your bike and be merry? Hell No! Straight to the chopping block, hack it, slash it, mill it, drill it, until it is what it could be. Then put it on your bike. But do you run it with that pay pal ordered stock controller? Hell NO! You got to beef up those traces till you can drive a truck through, rip half the components off the board and replace them with with something that could switch a small town on, then reprogram the program with lies, do a shifty on the shunt until that little brain operating the on off switch is so confused that it lets the current flow like money from Bernanke's printing presses. Hook up a couple of deep cycle gel cells and your done? Hell No! Strap at least 24 nano explosive devices to your whispering death machine, charge to an inch of their life, then prepare to suck them dry at such a rate that the electrons get carpet burn on the way out! Strap all this to your bike with duck tape, make sure it minus 200 outside, hook up your proof of insanity helmet cam, prey to the gods of speed and then, and only then when time and space warp around you as the flux rolls can you be content in the knowledge of what could be, not what has been.

Carpe diem.


I have no idea how it got there????? I sure didn't write it. It's pretty funny though. Weird.
 
mdd0127 said:
This just showed up from out of nowhere in the drafts folder on my yahoo acct:

I love Endless-Sphere forum.
Brand new motor, latest, greatest turns up at your door. Do you put it in your bike and be merry? Hell No! Straight to the chopping block, hack it, slash it, mill it, drill it, until it is what it could be. Then put it on your bike. But do you run it with that pay pal ordered stock controller? Hell NO! You got to beef up those traces till you can drive a truck through, rip half the components off the board and replace them with with something that could switch a small town on, then reprogram the program with lies, do a shifty on the shunt until that little brain operating the on off switch is so confused that it lets the current flow like money from Bernanke's printing presses. Hook up a couple of deep cycle gel cells and your done? Hell No! Strap at least 24 nano explosive devices to your whispering death machine, charge to an inch of their life, then prepare to suck them dry at such a rate that the electrons get carpet burn on the way out! Strap all this to your bike with duck tape, make sure it minus 200 outside, hook up your proof of insanity helmet cam, prey to the gods of speed and then, and only then when time and space warp around you as the flux rolls can you be content in the knowledge of what could be, not what has been.

Carpe diem.


I have no idea how it got there????? I sure didn't write it. It's pretty funny though. Weird.
. Who ever wrote this realy gets it! Lol awesome just like poetry!
 
Arlo1 said:
mdd0127 said:
This just showed up from out of nowhere in the drafts folder on my yahoo acct:

I love Endless-Sphere forum.
Brand new motor, latest, greatest turns up at your door. Do you put it in your bike and be merry? Hell No! Straight to the chopping block, hack it, slash it, mill it, drill it, until it is what it could be. Then put it on your bike. But do you run it with that pay pal ordered stock controller? Hell NO! You got to beef up those traces till you can drive a truck through, rip half the components off the board and replace them with with something that could switch a small town on, then reprogram the program with lies, do a shifty on the shunt until that little brain operating the on off switch is so confused that it lets the current flow like money from Bernanke's printing presses. Hook up a couple of deep cycle gel cells and your done? Hell No! Strap at least 24 nano explosive devices to your whispering death machine, charge to an inch of their life, then prepare to suck them dry at such a rate that the electrons get carpet burn on the way out! Strap all this to your bike with duck tape, make sure it minus 200 outside, hook up your proof of insanity helmet cam, prey to the gods of speed and then, and only then when time and space warp around you as the flux rolls can you be content in the knowledge of what could be, not what has been.

Carpe diem.


I have no idea how it got there????? I sure didn't write it. It's pretty funny though. Weird.
. Who ever wrote this realy gets it! Lol awesome just like poetry!

I like it too. :twisted:

BTW, I changed the registration questions, so we'll see this slows down the spammers for a while.
 
From the quotable quotes...
Kiwi said:
I love Endless-Sphere forum.
Brand new motor, latest, greatest turns up at your door. Do you put it in your bike and be merry? Hell No! Straight to the chopping block, hack it, slash it, mill it, drill it, until it is what it could be. Then put it on your bike. But do you run it with that pay pal ordered stock controller? Hell NO! You got to beef up those traces till you can drive a truck through, rip half the components off the board and replace them with with something that could switch a small town on, then reprogram the program with lies, do a shifty on the shunt until that little brain operating the on off switch is so confused that it lets the current flow like money from Bernanke's printing presses. Hook up a couple of deep cycle gel cells and your done? Hell No! Strap at least 24 nano explosive devices to your whispering death machine, charge to an inch of their life, then prepare to suck them dry at such a rate that the electrons get carpet burn on the way out! Strap all this to your bike with duck tape, make sure it minus 200 outside, hook up your proof of insanity helmet cam, prey to the gods of speed and then, and only then when time and space warp around you as the flux rolls can you be content in the knowledge of what could be, not what has been.

Carpe diem.
 
And that's from April of last year! So why would it just show up in your yahoo drafts folder? It is possible you have either had your account password hacked, or you were viewing that page with the quote on ES *and* had your yahoo mail account open, *and* there is a malware script running on your browser that copies/pastes from open tabs into your mail account tab.

I don't know if any such script exists, but it it would probably not be hard for a bot to be programmed to do. :(
 
amberwolf said:
It's bad enough when i ban by IP I may end up at least temporarily blocking some real user from accessing the site, because in the future their IP may get changed to one that is banned. :(

But for now it is at least partially effective in keeping spammers away, so I have to use that tool for now, till we get better tools.

That is unfortunate that spammers and real users are coming from the same IP blocks. That means there is most likely a botnet involved. That does make life a lot harder for a spam bounty hunter.

It would definitely be a collaborative effort, If it were to ever occur.
 
Unfortunately spammers come from potentially any IP range. Based solely on memory guesstimates: most of them come from Chinese IPs, or Russian. Various other European IPs are next, then India and Pakistan, then the Americas, mostly USA. It'd be nice if they were all from the same blocks of IPs, because that would make fighting them easy, by blocking the whole range. In some cases I *can* do that, after quite a number come from the same ones, but usually not. :(


I have a feeling that some of these bots are malware on people's computers that are unaware of what their computer is doing.

Mostly, I think that there are just immoral people all over the world, and they want money with little care of what they do to get it, so they sign up to do spamming and whatnot. They manually sign up for accounts, and then send in that login info to central locations that then distribute it to spambots around the world. This is the most likely reason for extremely disparate IPs between registration and posting.
 
82 more nuked. Now I think I am back to the end of December 2011.

But even though I already looked thru all the profiles created from 1-1-2012 thru 1-31-2012 yesterday, today when I did the same thing (whcih is several pages of tehm and takes a VERY long time to do, but is necessary), more spammers had popped up.

By that I mean that what looked like normal profiles had sprouted links in signatures or other spam on their profile page, not visible from anywhere else because they haven't posted yet, and thus had not yet been banned for spamming.

Since I have not been spending the several minutes per spammer to verify suspect ones that don't actually show spam links, I have left all of those alone, until they do the above.



I really really really wish the member list showed all of the fields in the profile, so I could just look at them right there. It would GREATLY speed up the process of removing the spammers, as I would only need to glance down the list to find most of them.


Having a nifty little checkbox next to their listing so I could just select a bunch and then remove them all at the same time would be REALLY REALLY handy right about now. ;)
 
Back
Top