Removal of bots from twitter API calls with SpaCy
Specifically related to twitter API calls with cryptocurrency and like keywords
The code and model is available in the following repository: https://github.com/IanLDias/tweet_analysis
Project steps
1) All tweets with a specific keyword (Bitcoin) were collected over the course of 3 days using crontabs to collect the data every 5 minutes. The data was stored on SQLite
2) After enough data was collected, each tweet was labelled by a human to determine if a selected tweet was tweeted by a bot or human.
- This can have bias involved, but other people were able to validate some of the labels.
- Label Studio was a big help over excel
3) SpaCy was used for the text classification
- SpaCy comes with a series of default pipelines for NLP (Tokenizer, POS, NER, DEP, etc) The text-cat was overwritten to fit this data sample.
- This text-cat was added to SpaCy's en_core_web_md
- The training data was passed to the model and a simple-CNN was chosen as the pipe architecture