Twitter Data Cleaning and Preprocessing for Data Science

In the past decade, new varieties of communication, like microblogging and text messaging have emerged and become ubiquitous. While there's no limit to the range of data conveyed by tweets and texts, often these short messages are accustomed share opinions and sentiments that individuals have about what's occurring within the world around them.

Opinion mining (known as sentiment analysis or emotion AI) refers to the employment of tongue processing, text analysis, linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to the voice of the customer materials like reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine.

Both Lexion and Machine learning-based approaches are going to be used for Emoticons-based sentiment analysis. Firstly we get on my feet with the Machine Learning-based clustering. In MachineLearning based approach we are used Supervised and Unsupervised learning methods. The Twitter data are collected and given as input within the system. The system classifies each tweet data as Positive, Negative, and Neutral and also produces the positive, negative and neutral no of tweets of every emoticon separately within the output. Besides being the polarity of every tweet is additionally determined on the idea of polarity.

Collection of information

To collecting the Twitter data, we've got to try to to some data processing processes. therein process, we've created our own application with help of Twitter API. With the assistance of Twitter API, we've collected an oversized no of the dataset. From this, we've got to make a developer account and register our app. Here we received a consumer key and a consumer secret: these are employed in application settings and from the configuration page of the app we also require an access token and an access token secrets which offer the applying access to Twitter on behalf of the account. the method is split into two sub-process. this is often discussed within the next subsection.

Accessing Twitter Data and Strimming

To make the appliance and to interact with Twitter services we use Twitter-provided REST API. We use a bunch of Python-based clients. The API variable is now our entry point for many of the operations we are able to perform with Twitter. The API provides features to access differing types of information. during this way, we will easily collect tweets (and more) and store them within the system. By default, the information is in JSON format, we alter it to txt format for straightforward accessibility.

In case we would like to “keep the connection open”, and gather all the upcoming tweets a couple of particular event, the streaming API is what we'd like. By extending and customizing the stream-listener process, we processed the incoming data. This way, we gather plenty of tweets. this is often very true for live events with worldwide live coverage.

Data Pre-Processing and Cleaning

The data pre-processing steps perform the required data pre-processing and cleaning on the collected dataset. On the previously collected dataset, the are some key attributes text: the text of the tweet itself, created_at: the date of creation,favorite_count, retweet_count: the quantity of favourites and retweets, favourited, retweeted: boolean stating whether the authenticated user (you) have favourited or retweeted this tweet etc.We have applied an intensive set of pre-processing steps to decrease the dimensions of the feature set to create it suitable for learning algorithms. The cleaning method is predicated on dictionary methods.

Data obtained from twitter usually contains lots of HTML entities like < > & which gets embedded within the original data. it's thus necessary to urge obviate these entities. One approach is to directly remove them by the employment of specific regular expressions. Hare, we are using the HTML parser module of Python which may convert these entities to straightforward HTML tags. for instance < is converted to “<” and & is converted to “&”. After this, we are removing this special HTML Character and links. In decoding data, this is often the method of remodeling information from complex symbols to simple and easier to know characters. The collected data uses different varieties of decoding like “Latin”, “UTF8” etc.

In the twitter datasets, there's also other information as retweet, Hashtag, Username and modified tweets. All of this is often ignored and off from the dataset.

Stop words are generally thought to be a “single set of words”. we'd not want these words taking on space in our database. For this using NLTK and employing a “Stop Word Dictionary” . The stop words are removed as they're not useful.All the punctuation marks in step with the priorities should be restricted. as an example: “.”, “,”,”?” are important punctuations that ought to be retained while others must be removed. within the twitter datasets, there's also other information as retweet, Hashtag, Username and Modified tweets. All of this can be ignored and aloof from the dataset. we must always remove these duplicates, which we already did. Sometimes it's better to get rid of duplicate data supported by a collection of unique identifiers. For example, the possibilities of two transactions happening at the identical time, with the identical square footage, the identical price, and also the same build year are near zero.

Thank you for reading.

I hope you found this data cleaning guide helpful. Please leave any comments to allow us to know your thoughts.