It its prime, I was finding around 25,000 spam accounts a day with a 97% accuracy. And that with only looking at less than 1% of public timeline. The bottleneck is the amount of spam Twitter will allow me to report (which is around 2,000 a day). After having conversations with Twitter they were a big help, but not in this area of actually reporting spam.

Given the amount of spam I’m able to report and the operational costs of running the system, it doesn’t make sense to continue the project.

It was a great project and a lot of fun. I far exceed my initial expectations of how much spam I could fine and I learned a lot about Amazon AWS, coding, multithreading, mysql and more. So I’m quite happy with how the project turned out.

I just launched a new project called Oncall. Its an alerting system to augment your IT monitoring solution to help make being on call easier and integrate into your phone via text and voice. This is my first open source project (which there is a lot I am learning about) and I’m pretty excited about it. I really wanted this software to be real easy to implement and work well for small teams of people (like 1 or 2), to hundreds of people and many teams of people.

Essentially how it works is you setup a server running Oncall-server. Your monitoring solution (ie Nagios, Solar Winds, hostmon, etc) sends alerts to oncall via http. Oncall then routes that alert to you phone (using Twilio API) by either calling you or texting you (depends on how you configure the software). The alert naturally escalates unless someone acknowledges it by text message or phone.

People and alerts can be associated with a team to ensure the right alert gets sent to the right people and the entire system is manageable via text messages. So you can do whatever you need to where ever you are.

When my code starts up, it loads a very large list of all active twitter accounts that I’ve already looked at into memory and then is maintained locally after that. This allows me to make only one query to my db instead of thousands (or even hundred of thousands). It seems to of solved my db crashing issue that I was running into. At least, so far it seems that way.

Initial results are quite astounding, as the amount of spam per hour discovered has increased by 8x! There a lot more fine tuning to do and now that I’m finding so much spam, new issues are arising (ie my MySQL db keeps on dropping my connection to it).

More soon.

Multithreading is basically getting a single process to execute code to split off into another thread and operate independently, to run in parallel. Here is the wikipedia article if you want to know more about it. The reason why its so important for Twitter Spam Assassin is because of how I receive tweets from Twitter. Essentially, I open a connection with Twitter asking them to send me tweets from the public timeline. Twitter sends me a tweet, my code looks at it, and when I’m done processing that tweet, Twitter sends me another one.

If I introduce multithreading, each incoming tweet would spin off into a separate thread to be analyzed. This allows me to receive the next tweet immediately and NOT wait for the analyzation of the last tweet to finish (which can take up to 3 seconds). You may to yourself, 3 seconds isn’t a long time, and you’d be right. But if you keep in mind that i’m analyzing thousands of tweets a day and I can get around about 45 tweets per second (best estimate), the numbers get large very quickly.

But with anything, there are fall backs. One of my concerns is that my server(s) cannot analyze as quickly as the tweets are coming in, which Im not sure if thats true or not at the moment. If it is true, there are ways to deal with that like load balancing.

But that isn’t the real reason why I haven’t explored multithreading in the past. The reason why is because of Twitter’s API call limit. If i start processing more tweets on a server, I’ll need to make more API request to Twitter per day. And I’m already hitting my call request cap several times a day. With a little testing, I find that if I enable multithreading I find that I see enormous performance boosts. But I use all of my API call requests very quickly and only have my code search for spam for 10 minutes per hour!

The good news is I am doing my best to work with Twitter and increase my API call limit. Getting some assistance from them would really increase the efficiency of finding and reporting spam.

@TiagoRanieri47

@TiagoRanieri48

@TiagoRanieri50

@TiagoRanieri51

At the time of this posting, three of those twitter accounts are still online. One of them has been suspended. My spam assassin found all of these accounts a couple days ago within an hour of each other and reported them all shortly thereafter. Its interesting to me that all of these accounts have nearly identical activity, have all been reported to twitter the same, but yet, only one of them are offline (despite having several days to take them offline).

In total, my spam assassin has found 51 twitter accounts with “TiagoRanieri” in the name with a very high degree of similarity between them. Out of those, 20 of them are still active (as of the time of this blog post).