我正在尝试使用e1071包中的朴素贝叶斯学习器进行垃圾邮件分析。这是设置模型的代码。
library(e1071)emails=read.csv("emails.csv")emailstrain=read.csv("emailstrain.csv")model<-naiveBayes(type ~.,data=emailstrain)
有两组电子邮件,每组都包含一个“statement”和一个“type”。一组用于训练,另一组用于测试。当我运行
model
并直接查看原始输出时,似乎当语句确实是垃圾邮件时,它会给出大于零的概率认为是垃圾邮件;当语句不是垃圾邮件时也是如此。然而,当我尝试使用模型预测测试数据时,使用
table(predict(model,emails),emails$type)
我得到的结果是
ham spamham 2086 321spam 2 0
这看起来不对。我也尝试使用训练集来测试数据,在这种情况下应该会得到相当好的结果,或者至少与模型中观察到的一样好。然而,它给出的结果是
ham spamham 2735 420spam 0 6
这只是比测试集稍好一点。我认为一定是predict函数的使用出了问题。
数据文件的设置方式和其中的一些示例:
type,statementham,How much did ur hdd casing cost.ham,Mystery solved! Just opened my email and he's sent me another batch! Isn't he a sweetieham,I can't describe how lucky you are that I'm actually awake by noonspam,This is the 2nd time we have tried to contact u. U have won the £1450 prize to claim just call 09053750005 b4 310303. T&Cs/stop SMS 08718725756. 140ppmham,"TODAY is Sorry day.! If ever i was angry with you, if ever i misbehaved or hurt you? plz plz JUST SLAP URSELF Bcoz, Its ur fault, I'm basically GOOD"ham,Cheers for the card ... Is it that time of year already?spam,"HOT LIVE FANTASIES call now 08707509020 Just 20p per min NTT Ltd, PO Box 1327 Croydon CR9 5WB 0870..k"ham,"When people see my msgs, They think Iam addicted to msging... They are wrong, Bcoz They don\'t know that Iam addicted to my sweet Friends..!! BSLVYL"ham,Ugh hopefully the asus ppl dont randomly do a reformat.ham,"Haven't seen my facebook, huh? Lol!"ham,"Mah b, I'll pick it up tomorrow"ham,Still otside le..u come 2morrow maga..ham,Do u still have plumbers tape and a wrench we could borrow?spam,"Dear Voucher Holder, To claim this weeks offer, at you PC please go to http://www.e-tlp.co.uk/reward. Ts&Cs apply."ham,It vl bcum more difficult..spam,UR GOING 2 BAHAMAS! CallFREEFONE 08081560665 and speak to a live operator to claim either Bahamas cruise of£2000 CASH 18+only. To opt out txt X to 07786200117
我非常希望得到建议。非常感谢您的帮助
回答:
实际上,predict函数运行得很好。请不要误会我的意思,问题在于您所做的事情。您使用这个公式构建模型:type ~ .
,对吗?公式左侧是什么很清楚,让我们看看右侧。
在您的数据中只有两个变量 – type
和 statement
,因为 type
是因变量,所以唯一算作自变量的是 statement
。到目前为止一切都很清楚。
让我们看看贝叶斯分类器。先验概率很明显,对吗?条件概率呢?从分类器的角度来看,您只有一个分类变量(您的句子)。对分类器来说,这只是一些标签列表。它们都是唯一的,所以后验概率将接近先验概率。
换句话说,当我们得到一个新观察时,我们唯一能说的就是它是垃圾邮件的概率等于训练集中消息是垃圾邮件的概率。
如果您想使用任何机器学习方法来处理自然语言,您必须首先预处理您的数据。根据您的问题,这可能意味着词干提取、词形还原、计算n-gram统计、tf-idf。训练分类器是最后一步。