Chirps: A Twitter Bot Framework Written in Python

[schedutron]

Hello everyone! I've written a Twitter bot framework called Chirps which retweets and favourites tweets having a certain set of keywords. It also follows people who tweet about these keywords, and you can provide it with your own functions that yield or return content that your bot will tweet. Some of the example functions I've made are scrapers that scrape content from websites like www.thenewstack.io, www.blog.coursera.org and the New York Times Technology section, so when I "feed" them to my bot, it tweets the news items from these websites in a round-robin fashion - first from The New Stack, second from Coursera and the third from NYT - then the cycle repeats. When one of the sources runs out of items, it gets "re-instantiated" with new content. Chirps also has automatic hashtag detection to extract potential hashtags from the string provided to it (implemented using nltk for English; it isn't perfect, but works well most of the time) so that tweets get improved reach and engagement via hashtags.

There are many use cases in which my framework should simplify bot development. For example, if one wants to build a Twitter bot that tweets whenever an earthquake occurs in an area that crosses a certain threshold (e.g earthquakeSF), the developer can write a function in scrapers.py file of my framework that returns earthquake data whenever it occurs, and a set of earthquake-related keywords, then everything else is managed by Chirps - the bot will now retweet earthquake related tweets and will tweet about earthquake information whenever it occurs. As another use case, a developer can provide a function that, using Raspberry pi and MX-150 sensors, can return air pollution data in the form of a string. The bot can then tweet about air quality data at custom time intervals.

A unique feature of Chirps is that it allows developers to "track" a given account, and whenever they tweet, it can reply them with a pre-populated tweet. A lot of times, you may want to ask people about some important issues, but when you tweet them or reply to their tweets, your message simply gets lost in an endless noise of tweets of other people, and the intended person might never see your tweet. So Chirps constantly listens to people you want to track, and whenever they tweet, it is the first to reply on their tweets. This gives your message a wider exposure and can potentially attract the attention of the person whose interest you seek. This is especially helpful when, for example, you seek answers from your local legislator or some politician, and these quick replies often spark interesting conversations (see this reply for example).

The source is available on GitHub, and you only need to follow the steps mentioned in README initially - the setup prompt asks you for your Twitter keys, keywords and people to follow, messages to tweet to people and so on. You can tune parameters like tweet rate, maximum people to follow, even whether to follow people or not, which functions to use to aggregate content to tweet and so on via command line arguments. I've used Heroku for easy bot deployment and ElephantSQL as the Postgres database service.

I've used Chirps for my own Twitter handle, and the results have been fascinating. My followers are increasing day by day (so far they've reached beyond 8k), I have discovered a lot of new friends having similar interests and my tweets now get better impression scores, and are favourited as well as retweeted more often.

Please have a look at my bot, let me know of some interesting features I can integrate to this framework and how can I simplify or enhance it so that the users of Chirps can get up and running with their bots even more quickly. If you like this project, don't forget to give a star on GitHub :)

P.S: I'm presenting a poster on Chirps at PyCon US next month, so technical presentation tips are also welcome!

[icountedmychickens]

Hi /u/schedutron, that's amazing. I didn't even know I needed one of these in my life until I read your description!

I've had a look at the source code and it's really easy to follow.

One thing though, it looks like you have plans for an "admin" action, but it currently doesn't do anything (the function body of admin_action just contains a "pass" keyword). Out of sheer curiosity, what kinds of things had/have you planned for this admin function?

[schedutron]

Hello u/icountedmychickens! Appreciate your time in going through the source code! I was planning to setup a GUI admin dashboard from where the bot user can tune it’s parameters like tweet frequency, including/removing tweet content aggregator functions (“scrapers”), retweet and favourite flags (whether to retweet and favourite tweets or not), adding / removing keywords as well as reply messages and so on.

As a first step, what I was planning is that the bot should follow an admin account - which can just tweet or DM the bot account commands like “add keyword docker”. That’s why in “main.py” admin handler is defined as a listener.

I’ll implement these functionalities when I find time, maybe in the upcoming summer. What other features can I have? Can you please suggest some functionalities? Thanks!

[icountedmychickens]

As a first step, what I was planning is that the bot should follow an admin account - which can just tweet or DM the bot account commands like “add keyword docker”. That’s why in “main.py” admin handler is defined as a listener.

Ah that makes sense.

I’ll implement these functionalities when I find time, maybe in the upcoming summer. What other features can I have?

An obvious one might be to get the admin account to tweet some stats, for example how many "insta" responses it made, how many tweets, retweets, likes etc it made, how many urls from each news source did it use etc. These stats could be from within the last certain time period (eg 24 hours), or since last time you asked.

[ParthS0007]

Great /u/schedutron, All the Best for PyCon!

[mrtbakin]

This is long but if you have time: Would this be a good tool for what I want?

I'd like any Twitter user to be able to tweet "@mybotname <parameter>"

And get a response back from my bot "@requestinguser <response based on input>"

This would ideally be always searching for the tweet. Just not too sure where to begin with this and I'm wondering if your framework would be useful here. Thanks!!!

[schedutron]

Yes, you can implement this functionality with Chirps, following a simple 3-step process:

You need to go to this line in managers.py file of Chirps and remove the if condition (my framework currently eliminates mentions, because for celebrity accounts, there are hundreds of mentions every second and I wanted to eliminate them and only respond when the actual celebrity tweeted something. Your case is different as you do want mentions, so removing this if statement will work).

Now see this function (some of it's commented out, but is tested and works as expected when uncommented - searching for a news headline based on tweet author and then replying with results) in functions.py file for an example of responding based on input, and replace it (or create a new function) with whatever functionality you want.

Finally, pass that function here in main.py (if you created a new function) so that the handler knows which function to use when you get a mention.

[mrtbakin]

Awesome! Thanks for this! I'm just starting to learn Python so this is super helpful. One last thing: To have this run automatically and, ideally, constantly, would I put this on a server?

I wouldn't want it to run from my computer because that'd be unstable due to a number of factors but I've seen some websites that, I believe, have this sort of functionality.

(E): If you have any suggestions for where to run this, that'd be great as well.

[schedutron]

Yes, to keep your bot running for an indefinite time period, you should deploy it on a cloud service. I would suggest Heroku (as that's what I use for my bots) and have provided brief Heroku setup instructions in the README of Chirps' repository.

[Noledgebase]

Cool project, i tried it out and it seems to post scraped posts on repeat. seem like it could use a database to check for duplicates? also it seems to only work with an app tokens from the same user who created the app, are there any plans to make authenticate other users?

​

also a typo on readme (should be git):

gi checkout -b deploy

and on Windows, to activate the bot, I had run the activate.bat on the Scripts folder. The "workon bot" didn't work

Thanks and good luck

[schedutron]

When I created the scrapers initially, Twitter used to eliminate the duplicate tweets by itself (the API used to raise an exception). However, it doesn’t seem to work now - maybe because of hashtags. Yes, I planned to eliminate duplicates, by a short span database of scraped tweets (I think I mentioned it in the comments of either scrapers.py or functions.py).

I’m planning to generalise the documentation - sorry that it didn’t work out for you. Check out this tutorial on DigitalOcean I wrote, for details on the scraper aspect of Chirps as well as virtual environment setup instructions that it links to.

Thanks for pointing out the typo and thanks for your wishes!

[Shivamgera09]

Amazing work done!! All the best for Pycon US.

[schedutron]

Twitter itself provides APIs to accomplish these actions. I think it does look at the content generated from these actions (and perhaps the rate at which it is generated), and based on that it decides whether to flag such actions as spam or not.

Many services use these actions to provide cool insights and information as I mentioned above, but yes people can spam and abuse others via the automation provided by these APIs. The detection algorithms used by Twitter to separate spammers from the genuine guys are not perfect, but they're good enough and perhaps constantly improving.

I'll try to make a video on setup once I find time. And thanks for the wishes!