When I first started out messing around with Engineering Deathmatch, I had two primary goals. First, to help keep my skills sharp and expose me to new areas of technology. Second, was to promote the engineers that were contestants on the show. As part of that second goal, I’ve wanted to promote the EDM contestant’s blog posts through the Engineering Deathmatch Twitter feed. This is one of those tasks that I needed to automate because as more engineers get added to the EDM family I just won’t be able to keep up with all of the blogs and promote them in a reasonable amount of time.
To that end I had been using Twibble (twibble.io) and it had been working well, but I always knew that there would be a time that I would need to build my own environment like Twibble. Partially because I didn’t want to worry about the upper limits of blogs that could be tracked and second because if I could build it myself and host it myself I wasn’t sure if their subscription fees were really worth it in the long run. This past weekend I finally decided to give it a go and write my own little application to give me a good portion of what Twibble does. With this post, I’ll walk you through the process that I used to make this work. Here we go.
There were some things that Twibble did that I really wanted to make sure that my new app did. These were the requirements that I originally defined that actually made me choose Twibble in the first place:
- Ability to monitor multiple RSS feeds. Since for every episode that gets created there is the potential for two more feeds to be added to the blog I needed to be able to scale this solution to handle many blogs. Initially the Twibble service had a top limit, even at their premium pricing tier, to the number of blogs that they would service. They recently changed their pricing and I think that upper limit went up, but if I run my own, then I can duplicate it, throw more resources at it, whatever to make my upper limit whatever I wanted it to be.
- Tweeting pictures from the blog post. Everyone knows that a tweet is more eye catching if there’s a picture to go with it, so I wanted to make sure whatever solution I developed had the ability to grab a picture from the post, if one existed, and include it with the Tweet when it posted to Twitter. I looked through github and while there were several options for propagating RSS feeds to Twitter, I didn’t quite find one that had this functionality.
Because I had some knowledge of Python and I wanted to learn more about it (see, hitting on that first goal of Engineering Deathmatch) I decided to use that to develop the application for this. I decided, even further, on using Python 3 because Python 2’s end of support has been announced for 2020. Yeah, it’s still a ways off, but may as well start coding with the newer stuff!
Additionally, the goal was that I was going to stick this on one of the many Raspberry Pi computers that I have hanging around the house and have it running on a scheduled job. That way I can stuff it in a corner somewhere and have it be out of the way.
The first thing that has to be done, well I guess it doesn’t have to be first, but it needs to be done some time at the beginning, so may as well make it first, is to create your application within Twitter so you can grab some keys! To do that, do the following:
- Go to apps.twitter.com and log in with your Twitter credentials. You will be presented with a screen to name your application. Give it a name, description and a web site. Web site is required because the assumption is that you will be using this to allow other people to use the app. for our purposes, that doesn’t matter.
- Once you have submitted that form, you will be taken to a page with information about your app. Scroll down to the section that talks about keys and click on “Manage keys and access tokens”
- Copy the consumer key and the consumer secret key and put them in a notepad, you will need them later. Towards the bottom of this page you will have a button to generate access token and access token secret. Click that and copy town the values for the two resulting keys.
These values are what will allow your application to communication with the Twitter API. There is a lot of stuff you can do with this, but for this project we will just be sending Twitter status updates. We will use this stuff later, but set it aside for now.
Next, let’s install some Python packages. If you only have Python3 installed, the command should just be: pip install <package>, however if you are using a Raspberry Pi, it has both installed, so the command should be pip3 install <package>. Make sure you know what you have installed because if you have both the system will still allow you to install with the first command, but it will just be installed for Python 2. You will need to install the following packages for Python:
- feedparser – this will read the RSS feeds for you
- configparser – this will read in configuration files that we will develop for this
- bs4 – this helps us to search for image strings in the RSS feed so that we can download that image to post with the tweet
- twython – there are several packages that allow you to use Twitter with Python, this just happened to be the one that had the most understandable documentation to add media into tweets.
Alright, it’s time to code! Open your favorite text editor, the one I used was Atom, but whatever you’re most comfortable with will probably work. I’m not going to go through a lot of the nitty gritty details of how to write out the code for this project, because you can go and look at all of the code that I used up on github here: https://github.com/engdeathmatch/RssToTwitterPy3 I’ve tried to include decent instructions on the repository page and I also tried to be pretty judicious in documenting my code. So go check that out and I think it will get you well on your way. A couple of things that you need to be aware of:
- You need to make sure that you have the auth.cfg file and fill in the details of your Twitter app that you got in the first part of this. Without this, your application won’t be able to talk to Twitter.
- You need to fill out the feeds.cfg file with the usernames and feeds that you want to follow. I used the usernames because I wanted to attribute the engineer to the blog post that I was promoting (for instance, via @samplefive)
Once you have your code all created and on your Raspberry Pi and it works when you execute it (you can do this with: python3 <filename>), then next thing that you will want to do is create a job to schedule it. I spent a good deal of time on this because I couldn’t, for the life of me, figure out how to make a scheduled task from within Linux that would properly kick off my Python file. What I ended up having to do was create a shell script that then called the Python file and schedule the shell script. From the command line on your Raspberry pi, enter:
sudo nano runrss.sh
That will take you into an editor, this file is pretty simple. Here’s an example of what mine looked like:
#!/bin/sh python3 rsstotwitter.py
Hit Ctrl-X to exit and save the file and then you are ready to schedule your application to run regularly. Type the following:
sudo crontab -e
This will take you to the scheduler file. Go towards the bottom of the file and find an extra line and enter something like this:
*/20 9-20 * * * cd /home/pi/appfolder && ./runrss.sh
Make sure that you have a blank line underneath that commad. So what does this do exactly? The first portion */20 9-20 * * * tells cron that this job should run every 20 minutes between the hours of 9 AM and 8 PM on the server. It then changes the directory to where the application is located so that any path statements in the Python script that are just in the same directory will be mapped correctly, and finally it runs the script.
So far the application is working pretty well, and maybe you even stumbled across this post from a Tweet that came through it. There are a couple of things that I will be looking to do in the long run including:
- Configure the application so that it automatically takes older links out of the log, just so that tracking file doesn’t get unbearably large.
- Build in some intelligence so that if the script encounters more than one post at a time to tweet out it can schedule them to be tweeted a little further apart from each other.
But as far as a replacement for the service that I used to be using, it has replaced the core functionality quite well!