Julia DataFrames - How to do one-hot encoding?
It is easy enough to do it with basic functions we provide though:
julia> df = DataFrame(x=rand([1:3;missing], 20))
20×1 DataFrame
│ Row │ x │
│ │ Int64? │
├─────┼─────────┤
│ 1 │ 1 │
│ 2 │ 2 │
│ 3 │ missing │
│ 4 │ 1 │
│ 5 │ 3 │
│ 6 │ missing │
│ 7 │ 3 │
│ 8 │ 3 │
│ 9 │ 3 │
│ 10 │ 3 │
│ 11 │ missing │
│ 12 │ 1 │
│ 13 │ 3 │
│ 14 │ 3 │
│ 15 │ 3 │
│ 16 │ 1 │
│ 17 │ missing │
│ 18 │ 1 │
│ 19 │ 1 │
│ 20 │ missing │
julia> ux = unique(df.x); transform(df, @. :x => ByRow(isequal(ux)) .=> Symbol(:x_, ux))
20×5 DataFrame
│ Row │ x │ x_1 │ x_2 │ x_missing │ x_3 │
│ │ Int64? │ Bool │ Bool │ Bool │ Bool │
├─────┼─────────┼──────┼──────┼───────────┼──────┤
│ 1 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 2 │ 2 │ 0 │ 1 │ 0 │ 0 │
│ 3 │ missing │ 0 │ 0 │ 1 │ 0 │
│ 4 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 5 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 6 │ missing │ 0 │ 0 │ 1 │ 0 │
│ 7 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 8 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 9 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 10 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 11 │ missing │ 0 │ 0 │ 1 │ 0 │
│ 12 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 13 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 14 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 15 │ 3 │ 0 │ 0 │ 0 │ 1 │
│ 16 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 17 │ missing │ 0 │ 0 │ 1 │ 0 │
│ 18 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 19 │ 1 │ 1 │ 0 │ 0 │ 0 │
│ 20 │ missing │ 0 │ 0 │ 1 │ 0 │
EDIT:
Another example:
julia> df = DataFrame(col1=102:104, col2=[["a"], ["a","b"], ["c","b"]])
3×2 DataFrame
│ Row │ col1 │ col2 │
│ │ Int64 │ Array… │
├─────┼───────┼────────────┤
│ 1 │ 102 │ ["a"] │
│ 2 │ 103 │ ["a", "b"] │
│ 3 │ 104 │ ["c", "b"] │
julia> ux = unique(reduce(vcat, df.col2))
3-element Array{String,1}:
"a"
"b"
"c"
julia> transform(df, :col2 .=> [ByRow(v -> x in v) for x in ux] .=> Symbol.(:col2_, ux))
3×5 DataFrame
│ Row │ col1 │ col2 │ col2_a │ col2_b │ col2_c │
│ │ Int64 │ Array… │ Bool │ Bool │ Bool │
├─────┼───────┼────────────┼────────┼────────┼────────┤
│ 1 │ 102 │ ["a"] │ 1 │ 0 │ 0 │
│ 2 │ 103 │ ["a", "b"] │ 1 │ 1 │ 0 │
│ 3 │ 104 │ ["c", "b"] │ 0 │ 1 │ 1 │
There is indeed no one-hot encoding function in DataFrames.jl - I would argue that this is sensible, as this is a particular machine learning transformation that should live in a an ML package rather than in a basic DataFrames package.
You've got two options I think:
Use an ML package that does this for you, e.g. MLJ.jl. In MLJ, the
OneHotEncoder
is a model that transforms any table withFinite
features in it into a one-hot encoded version of itself, see the docs hereUse a regression package that automatically generates dummy columns for categorical variables using the StatsModels
@formula
API - if you fit a regression with e.g.GLM.jl
and your formula is@formula(y ~ x)
wherex
is a a categorical variable, the model matrix will automatically be constructed by contrast codingx
, i.e. having binary dummy columns for all but one level ofx
For the second option, you ideally want your data to be categorical (although strings will work as well), and for this DataFrames.jl includes the categorical!
function.
EDIT 17/11/2021: There has since been a definitive thread on this on the Julia Discourse which contains an extensive list of suggestions for doing one-hot encoding: https://discourse.julialang.org/t/all-the-ways-to-do-one-hot-encoding/
Sharing my favourite from there:
julia> x = [1, 2, 1, 3, 2];
julia> unique(x) .== permutedims(x)
3×5 BitMatrix:
1 0 1 0 0
0 1 0 0 1
0 0 0 1 0