我有一个20000×4的数据集,其中4列是字符串。第一列是描述,后三列是类别,最后一列是我希望预测的。我已经将第一列的每个单词进行了标记化,并将其保存在一个字典中,每个单词对应一个整数值,我还将其他列转换为数值。现在我不知道如何将这些数据输入到Flux模型中。
根据文档,我需要使用“用于训练的数据集合(通常是一组输入x和目标输出y)”。在示例中,它将数据x和y分开。但是,我如何将字典加上两个数值列一起使用呢?
编辑:
这是我目前的一个最小示例:
using WordTokenizersusing DataFramesdataframe = DataFrame(Description = ["It has pointy ears", "It has round ears"], Size = ["Big", "Small"], Color = ["Black", "Yellow"], Category = ["Dog", "Cat"])dict_x = Dict{String, Int64}()dict_y = Dict{String, Int64}()function words_to_numbers(data, column, dict) i = 1 for row in range(1, stop=size(data, 1)) array_of_words = tokenize(data[row, column]) for (index, word) in enumerate(array_of_words) if haskey(dict, word) continue else dict[word] = i i += 1 end end endendfunction categories_to_numbers(data, column, dict) i = 1 for row in range(1, stop=size(data, 1)) if haskey(dict, data[row, column]) continue else dict[data[row, column]] = i i += 1 end endendwords_to_numbers(dataframe, 1, dict_x)categories_to_numbers(dataframe, 4, dict_y)
我想使用dict_x和dict_y作为Flux模型的输入和输出
回答:
考虑以下示例:
using DataFramesdf = DataFrame()df.food = rand(["apple", "banana", "orange"], 20)multiplier(fruit) = (1 + (0.1 * rand())) * (fruit == "apple" ? 95 : fruit == "orange" ? 45 : 105)foodtoken(f) = (fruit == "apple" ? 0 : fruit == "orange" ? 2 : 3)df.calories = multiplier.(df.food)foodtoken(f) = (fruit == "apple" ? 0 : fruit == "orange" ? 2 : 3)fooddict = Dict(fruit => (fruit == "apple" ? 0 : fruit == "orange" ? 2 : 3) for fruit in df.food)
现在我们可以将标记的数值添加到数据框中:
df.token = map(x -> fooddict[x], df.food)println(df)
现在你应该可以使用df.token作为输入和df.calories作为输出进行预测了。
========== 在你发布更多代码后的补充:===========
对于你修改后的示例,你只需要一个辅助函数:
function colvalue(s, dict) total = 0 for (k, v) in dict if occursin(k, s) total += 10^v end end totalendwords_to_numbers(dataframe, 1, dict_x)categories_to_numbers(dataframe, 4, dict_y)dataframe.descripval = map(x -> colvalue(x, dict_x), dataframe.Description)dataframe.catval = map(x -> colvalue(x, dict_y), dataframe.Category)println(dataframe)