The Data¶
The dataset contains two parts:
- TRAINING: 4327 messages out of which there are 2949 non-spam messages (HAM) and 1378 spam messagees (SPAM), all received from non-spam-trap sources.
SPAMTrain.label contains the labels of the emails, with 1 stands for a HAM and 0 stands for a SPAM.
- TESTING: 4292 messages without known class labels.
The format of the .eml file is definde in RFC822, and information on recent standard of email, i.e., MIME (Multipurpose Internet Mail Extensions) can be find in RFC2045-2049.
Since some data mining techniques only make use of the subject and body of the email to identify spam. In this package, we have included a simple python script (ExtractContent.py) which can help to extract the subject and body of the email.
In a python compatible environment, ( the code is test on python 2.5.1 and should work on python 2.x)
1, invoke the script by command ./ExtractContent.py
2, input source directory -- where you store the source files For exmaple C:\EMAILPro\CSDMC2010_SPAM\TEST
3, input destination directory -- where you want the extracted body to be For example C:\EMAILPro\CSDMC2010_SPAM\TEST_NEW
4, we are done
Note that, the script only extract limited information from the email (no information of fields like to, from, attachment are extract but only the subject and the first part of the body.) By oferring such a script we just want to show a simple preprocessing mehtod where the participants can start from. More advanced method which makes use of email header information or even attachment information are encouraged.
The Script¶
- sklearn for building machine learning models
- re for parsing strings using regular expressions
- os for manipulating filepaths in our computer
- BeautifulSoup for manipulating HTML code
- pandas and numpy for data analysis
In [1]:
import os, sys, sklearn, re
import BeautifulSoup as bs
import pandas as pd
import numpy as np
In [2]:
# load the file into memory
filepath_train = "/Users/apple/Desktop/Kaggle/SpamEmail/CSDMC2010_SPAM/CSDMC2010_SPAM/TRAINING_extract"
In [4]:
# parse the file into a data frame
files = [{'file':f, 'path':os.path.join(filepath_train, f), 'content':'\n'.join(open(os.path.join(filepath_train, f)).readlines())} for f in os.listdir(filepath_train) if os.path.isfile(os.path.join(filepath_train, f))]
df_files = pd.DataFrame(files)
# take a peak at the top 5 rows to make sure we're doing it correctly
df_files.head()
Out[4]:
In [6]:
# load the labels into memory
filepath_label = "/Users/apple/Desktop/Kaggle/SpamEmail/CSDMC2010_SPAM/CSDMC2010_SPAM/SPAMTrain.label"
df_labels = pd.DataFrame.from_csv(filepath_label, sep=" ", header = None, index_col = None)
df_labels.columns = ['class','file']
# take a peak at the top 5 rows
df_labels.head()
Out[6]:
In [7]:
# combine the two tables by merging on the common column, which is the "file" column
df_files_labeled = df_files.merge(df_labels)
df_files_labeled.head()
Out[7]:
In [19]:
#create a hash of the words in each file
file_words = []
for idx, row in df_files_labeled.iterrows():
word_counter = {}
content = row['content']
parsed = re.sub(r'<[^<>]*>', '', content)
words = [w.strip('(').strip(')').strip('"') for w in re.split(r'[!,\.\s\n]',parsed) if len(w)>0]
for w in words:
if w.lower() in word_counter:
word_counter[w.lower()]+=1
else:
word_counter[w.lower()]=1
file_words.append(word_counter)
In [20]:
# if a word isn't represented, make it 0 rather than NA
df_words = pd.DataFrame(file_words).fillna(0)
In [21]:
# Take only words that are represented more than 50 times throughout all documents
df_filtered = df_words[df_words.columns[df_words.sum()>50]]
In [22]:
# check at the matrix sizes look right
print df_words.shape
print df_filtered.shape
In [44]:
#take df_filtered.head()
Out[44]:
Let's split the data into training and validation set¶
In [25]:
from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(df_filtered.as_matrix(), df_files_labeled['class'], test_size=0.3, random_state=0)
Gaussian Naive Bayes model¶
In [30]:
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb = gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
Calculate the accuracy score¶
In [31]:
sklearn.metrics.accuracy_score(y_test, y_pred)
Out[31]:
In [32]:
Random forest model¶
In [34]:
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
In [35]:
sklearn.metrics.accuracy_score(y_test, y_pred)
Out[35]:
But why use one train/validation split? Let' do cross-validation¶
In [38]:
scores = sklearn.cross_validation.cross_val_score(clf, df_filtered.as_matrix(), df_files_labeled['class'])
scores.mean()
Out[38]:
In []:
Extra Trees Classifier¶
In [40]:
clf = sklearn.ensemble.ExtraTreesClassifier(n_estimators=10, max_depth=None,min_samples_split=1, random_state=0)
scores = sklearn.cross_validation.cross_val_score(clf, df_filtered.as_matrix(), df_files_labeled['class'])
scores.mean()
Out[40]:
AdaBoost Classifier¶
In [42]:
clf = sklearn.ensemble.AdaBoostClassifier(n_estimators=100)
scores = sklearn.cross_validation.cross_val_score(clf, df_filtered.as_matrix(), df_files_labeled['class'])
scores.mean()
Out[42]:
Draw an ROC curve¶
In [62]:
clf = sklearn.ensemble.AdaBoostClassifier(n_estimators=100)
clf = clf.fit(X_train, y_train)
In [63]:
y_score = [score[0] for score in clf.predict_proba(X_test)]
In [64]:
fpr, tpr, thresholds = sklearn.metrics.roc_curve(y_test, y_score, pos_label = 0)
In [78]:
import matplotlib.pyplot as plt
%pylab inline
plt.plot(fpr, tpr)
plt.xlim(0,0.3)
plt.ylim(0.5,1)
Out[78]:
In [70]:
pd.DataFrame({'fpr':fpr, 'tpr':tpr, '1-tpr':1-tpr, 'thresholds':thresholds}).sort(columns=['fpr', '1-tpr']).head()
Out[70]:
In [80]:
pd.DataFrame({'fpr':fpr, 'tpr':tpr, '1-tpr':1-tpr, 'thresholds':thresholds}).sort(columns=['1-tpr', 'fpr']).head()
Out[80]:
How can we do better from here?¶
Now report the test error¶
In []:
filepath_train = "/Users/apple/Desktop/Kaggle/SpamEmail/CSDMC2010_SPAM/CSDMC2010_SPAM/TEST_extract"