Very Basic Twitter Stock Sentiment Analysis

Using Twitter and Ruby to quantify what people are saying about different stocks

One way that investors analyze stocks is by looking at the general sentiment around it. What are people saying? Is it negative or positive? While sometimes it is obvious what the general ‘feeling’ is about a name, what if there was a quantifiable way to determine the sentiment? That would be useful because it would help investors compare different stocks to each other. Twitter is a great way to see what people are saying in real time, and you can easily narrow in on finance-oriented content by searching, for example, “$AAPL” or “$MSFT.”

The first thing I needed to do was get the actual content from Twitter. I wanted just the text of the tweets. I typed $AAPL into the search box and got my search query. I put the URL into the below code in my text editor and examined my now structured html object in my terminal. Below is my $AAPL search result URL. The Nokogiri gem converts the HTML from the URL into nested elements that make it easy to parse through!

I went back to the web page in my browser and right-clicked “Inspect” on the page so that I could find the element that contained the actual text. I found it and successfully retrieved the text in my terminal.

I wrote some code to get the text of each tweet into an array. The problem was that the URL with my search only returned 20 tweets. To get more data, I continued scrolling on the web page until more tweets loaded, and then went back to ‘Inspect’ in the web browser to get the additional html content. I copied the content and put it into a .html file within my project folder and modified my code to read my folder instead of a web page. I had loaded about 100 tweets before copying them into my file. When I tried to copy/paste, my text editor crashed! It was also kind of laborious and I felt like Twitter probably didn’t want people to access their data in this way. I had to come up with a different approach.

The first thing I did was write some code that would count each word in my data set:

I thought one useful and quick application would be to narrow the data set by retrieving only stocks and seeing how many times they were mentioned. That way, someone could get a general idea of similar stocks an investor might be interested in. It seems like $FB and $MSFT were the most frequently mentioned in tweets about $AAPL!

I took the .txt files and put them into my project folder. I didn’t really use any of the code from the repository except the code that helped load the text files. With the .txt file word points, I wrote a very simple method that would simply add up the point values from the txt file to come up with an overall score for the dataset:

Since I didn’t write the sentiment score data or use any code from it except to open the files, I haven’t really defined what these scores mean. But I assume anything below 0 is slightly negative and anything above is positive. Below are sentiment scores for a few stocks…it looks like Netflix has the most positive sentiment and Walmart has the most negative!

$WMT: -22.9
$PCLN: -3.7
$AMZN: 10.6
$TSLA: 13.7
$FB: 16.3
$AAPL: 21.1
$NFLX: 31.8

Related posts:

Surprised? If you know me, you probably are. I normally eschew American-style shoutiness, let alone ‘gurus’, in all their super-slick forms. But there’s something very real and likeable about Shaun…

With the final pull of the lever the capsule disembarked. It’s guidance rockets blasted it safely away from the mothership and towards the waiting planet. Onboard and suited up sat Reagan, who tried…