Tools used:¶
- Pandas
- Jupyter notebook
- Altair
- Sklearn
- Numpy
- NLTK
- Matplotlib
- python built in libraries (re, datetime)
- Tweepy
Data collection:¶
In [96]:
airlines_df.loc[:,['description','statuses_count','user_name','user_screen_name','user_lang']].head(20)
Out[96]:
In [97]:
# filtering the verified and english accounts
top_20_airlines_df = airlines_df[(airlines_df.verified==True) & (airlines_df.user_lang == 'en')].reset_index(drop=True).head(39)
top_20_airlines_df.loc[:,['description','statuses_count','user_name','user_screen_name','user_lang']].head(20)
Out[97]:
In [98]:
# removing non-airlines accounts and non-english accounts
top_20_airlines_df = top_20_airlines_df.drop(top_20_airlines_df.index[[7,12,13,20]]).reset_index(drop=True)
top_20_airlines_df.loc[:,['description','statuses_count','user_name','user_screen_name','user_lang']].head(20)
Out[98]:
Twitter User Gender Classification¶
ref: https://www.kaggle.com/crowdflower/twitter-user-gender-classification¶
Twitter US Airline Sentiment¶
ref: https://www.kaggle.com/crowdflower/twitter-airline-sentiment¶
Number of collected tweets:75070¶
Insights:¶
1- There is a negative relationship between text length and sentiment score.¶
In [ ]:
from nltk.sentiment.vader import SentimentIntensityAnalyzer
sentiment_analyzer = SentimentIntensityAnalyzer()
tweets_df['sentiment_score'] = tweets_df.text.apply(lambda x: sentiment_analyzer.polarity_scores(x)['compound'])
2- Customer service is the most issue that the customer face.¶
In [99]:
from sklearn.pipeline import Pipeline
from sklearn.linear_model import SGDClassifier
from sklearn.feature_extraction.text import TfidfTransformer,CountVectorizer
from sklearn.model_selection import train_test_split,GridSearchCV,cross_val_score
from sklearn.metrics import classification_report
from nltk.stem.snowball import SnowballStemmer
from nltk.stem import WordNetLemmatizer
import re
%matplotlib inline
stemmer = SnowballStemmer("english")
lemmatizer = WordNetLemmatizer()
In [100]:
reason_df = pd.read_csv('Tweets_reason.csv')
reason_df=reason_df[reason_df.airline_sentiment == 'negative'].reset_index(drop=True)
reason_df=reason_df[(reason_df.negativereason_confidence == 1) & (reason_df.airline_sentiment_confidence==1)].reset_index(drop=True)
In [101]:
reason_df.negativereason.value_counts()
Out[101]:
In [103]:
reason_df.loc[(reason_df.negativereason=='Damaged Luggage') | (reason_df.negativereason=='Lost Luggage'),'negativereason']='Lost and Damaged Luggage'
reason_df=reason_df[(reason_df.negativereason!='longlines') & (reason_df.negativereason!='Flight Booking Problems')]
In [104]:
reason_df.negativereason.value_counts()
Out[104]:
In [106]:
text_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', SGDClassifier(loss='hinge', penalty='none',
alpha=0.001,
max_iter=5, tol=None,class_weight='balanced')),])
X_train, X_test, y_train, y_test = train_test_split(reason_df['text'],reason_df.negativereason,test_size=0.33,random_state=53,stratify=reason_df.negativereason)
In [107]:
text_clf.fit(X_train,y_train)
predicted_test = text_clf.predict(X_test)
print(classification_report(predicted_test, y_test))
In [108]:
parameters = {
'vect__stop_words':('english',None),
'tfidf__norm':('l1','l2'),
'clf__loss': ('hinge','log'),
'clf__alpha': (0.001,0.0001,0.00001, 0.000001),
'clf__penalty': ('l1','l2','none'),
'clf__max_iter': [10,100,1000],
}
grid_clf = GridSearchCV(text_clf,param_grid=parameters,cv=5,n_jobs=5)
grid_clf.fit(reason_df['text'],reason_df.negativereason)
grid_clf.best_score_
Out[108]:
In [109]:
grid_clf.best_estimator_.fit(X_train,y_train)
print(classification_report(grid_clf.best_estimator_.predict(X_test), y_test))
3- Airlines gender distribution:¶
In [120]:
gender_df = pd.read_csv('gender-classifier-DFE-791531.csv',encoding='latin1')
gender_df = gender_df[((gender_df.gender == 'male') | (gender_df.gender == "female")) & (gender_df['gender:confidence'] == 1) ].reset_index(drop=True)
In [121]:
def stem_and_lem(text):
# Remove non-ASCII chars.
text = re.sub('[^\x00-\x7F]+',' ', text)
# Remove double spaces.
text = re.sub('\s+',' ',text)
text = "".join([stemmer.stem(word) for word in text])
text = "".join([lemmatizer.lemmatize(word) for word in text])
return text
In [122]:
gender_df.text=gender_df.text.apply(stem_and_lem)
feature_gender_df = pd.DataFrame()
feature_gender_df['features'] = ''
for k,item in enumerate(gender_df.loc[:,['text','description','sidebar_color','link_color']].astype(str).values):
feature_gender_df.loc[k]=" ".join(item)
In [123]:
gender_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', SGDClassifier(loss='hinge', penalty='l2',
alpha=0.001,
max_iter=5, tol=None,
class_weight='balanced')),])
X_train, X_test, y_train, y_test = train_test_split(feature_gender_df.features,gender_df.gender.astype(str),test_size=0.33,random_state=0,stratify=gender_df.gender)
In [129]:
parameters = {
'vect__stop_words':('english',None),
'tfidf__norm':('l1','l2'),
'clf__loss': ('hinge','log','modified_huber'),
'clf__alpha': (0.001,0.0001,0.00001, 0.000001),
'clf__penalty': ('none','l1','l2'),
'clf__max_iter': [10,100,1000],
}
grid_gender_clf = GridSearchCV(gender_clf,param_grid=parameters,cv=5,n_jobs=5)
grid_gender_clf.fit(feature_gender_df.features,gender_df.gender.astype(str))
grid_gender_clf.best_score_
Out[129]:
In [130]:
grid_gender_clf.best_estimator_.fit(X_train,y_train)
print(classification_report(grid_gender_clf.best_estimator_.predict(X_test), y_test))
Gender distribution¶
4- There is a positive relationship between number of followers and number of tweets.¶
5- Spirit airlines has the highest negative to positive ratio out of all the other airlines¶
Future directions and unanswered questions:¶
- What kind of respond did the cutomers got from the airlines?
- If a better accuracy has been achived in both gender and negative reason classifaction, will we find new patterns?
- Are some of the negative tweets contrived by other companies?
- Does the airlines companies treat influencers differently?
Conclution:¶
- There is a negative relationship between text length and sentiment score.
- Customer service is the most issue that the customer face.
- Most of United airlines customers are males
- There is a positive relationship between number of followers and number of tweets.
- Spirit airlines has the highest negative to positive ratio out of all the other airlines