如果我有大量的文本，并且试图发现其中最常出现的模板，我考虑使用N-Gram方法来解决这个问题，实际上在这个问题中也有人建议使用这种方法，但我的需求略有不同。为了澄清，我有一些这样的文本：

I wake up every day morning and read the newspaper and then go to workI wake up every day morning and eat my breakfast and then go to workI am not sure that this is the solution but I will tryI am not sure that this is the answer but I will tryI am not feeling well today but I will get the work done and deliver it tomorrowI was not feeling well yesterday but I will get the work done and let you know by tomorrow

我试图提取这样的“模板”：

I wake up every day morning and ... and then go to workI am not sure that this is the ... but I will tryI ... not feeling well ... but I will get the work done and ... tomorrow

我正在寻找一种可以处理数百万行文本的方法，所以我想知道我是否可以调整相同的N-gram方法来解决这个问题，或者是否有其他替代方案？

回答：

数百万行文本并不是一个非常大的数字 🙂

你寻找的东西至少与搭配发现相似。你可以尝试计算n-gram的点对点互信息。请参阅Manning & Schütze (1999)了解这个问题和其他方法的解决方案。