我正在尝试使用 Spark 的 MLLib 实现词向量化。我遵循了这里给出的示例。
我有一组句子，我想将它们作为输入来训练模型。但我不确定这个模型是接受句子还是仅接受作为字符串序列的所有词。
我的输入如下：
scala> v.take(5)res31: Array[Seq[String]] = Array(List([WrappedArray(0_42)]), List([WrappedArray(big, baller, shoe, ?)]), List([WrappedArray(since, eliud, win, ,, quick, fact, from, runner, from, country, kalenjins, !, write, ., happy, quick, fact, kalenjins, location, :, kenya, (, kenya's, western, highland, rift, valley, ), population, :, 4, ., 9, million, ;, compose, 11, subtribes, language, :, kalenjin, ;, swahili, ;, english, church, :, christianity, ~, africa, inland, church, [, aic, ],, church, province, kenya, [, cpk, ],, roman, catholic, church, ;, islam, translation, :, kalenjin, translate, ", tell, ", formation, :, wwii, ,, gikuyu, tribal, member, wish, separate, create, identity, ., later, ,, student, attend, alliance, high, school, (, first, british, public, school, kenya, ), form, ...但是当我尝试在这个输入上训练我的 Word2Vec 模型时，它不起作用。
scala> val word2vec = new Word2Vec()word2vec: org.apache.spark.mllib.feature.Word2Vec = org.apache.spark.mllib.feature.Word2Vec@51567040scala> val model = word2vec.fit(v)java.lang.IllegalArgumentException: requirement failed: The vocabulary size should be > 0. You may need to check the setting of minCount, which could be large enough to remove all your words in sentences.Word2Vec 不能接受句子作为输入吗？
回答：
您的输入是正确的。然而，Word2Vec 会自动删除在词汇表中（所有句子合并）出现次数不满足最低要求的词。默认情况下，这个值是5。在您的情况下，很可能没有一个词在您使用的数据中出现5次或更多次。
要更改所需的最小词出现次数，请使用 setMinCount()，例如将最小计数设置为2：
val word2vec = new Word2Vec().setMinCount(2)