我正在尝试创建一个关于去货币化数据集的标记语料库，该数据集大约有6250条推文。代码如下，尽管它只能处理一个包含200条记录的小数据集。
df = pd.read_csv('Demonetization_data29th2.csv',encoding = "ISO-8859-1")text = df['CONTENT']sentiment = df['sentiment']a =[]tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]tagged = [nltk.pos_tag(sent) for sent in tagged]print tagged[0]print "---------"brown_tagged_sents = taggedprint brown_tagged_sents[0]size = int(len(brown_tagged_sents) * 0.7)tags = [tag for (word, tag) in brown.tagged_words()]defaultTag = nltk.FreqDist(tags).max()print defaultTagtrain_sents = brown_tagged_sents[:size]test_sents = brown_tagged_sents[size:]tagger = ngramTagger(train_sents, 2, defaultTag)print tagger.evaluate(test_sents)我遇到了如下错误：
File "C:/Users/HP/pos-2.py", line 41, in <module> tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']] File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 130, in word_tokenize sentences = [text] if preserve_line else sent_tokenize(text, language) File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 97, in sent_tokenize return tokenizer.tokenize(text) File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1235, in tokenize return list(self.sentences_from_text(text, realign_boundaries)) File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1283, in sentences_from_text return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)] File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1274, in span_tokenize return [(sl.start, sl.stop) for sl in slices] File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1314, in _realign_boundaries for sl1, sl2 in _pair_iter(slices): File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 312, in _pair_iter prev = next(it) File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1287, in _slices_from_text for match in self._lang_vars.period_context_re().finditer(text):TypeError: expected string or buffer请指导我哪里出错了
回答：
你可能在该列的一个或多个行中遇到了空值。
可以用空字符串填充它，如下所示：
df['CONTENT'] = df['CONTENT'].fillna('')