Dropout layer before or after LSTM. What is the difference?
As default, Dropout
creates a random tensor of zeros an ones. No pattern, no privileged axis. So, you can't say a specific thing is being dropped, just random coordinates in the tensor. (Well, it drops features, but different features for each step, and differently for each sample)
You can, if you want, use the noise_shape
property, which will define the shape of the random tensor. Then you can select if you want to drop steps, features or samples, or maybe a combination.
- Dropping time steps:
noise_shape = (1,steps,1)
- Dropping features:
noise_shape = (1,1, features)
- Dropping samples:
noise_shape = (None, 1, 1)
There is also the SpatialDropout1D
layer, which uses noise_shape = (input_shape[0], 1, input_shape[2])
automatically. This drops the same feature for all time steps, but treats each sample individually (each sample will drop a different group of features).
After the LSTM
you have shape = (None, 10)
. So, you use Dropout
the same way you would use in any fully connected network. It drops a different group of features for each sample.
A dropout as an argument to the LSTM
has a lot of differences. It generates 4 different dropout masks, for creating different inputs for each of the different gates. (You can see the LSTMCell code to check this).
Also, there is the option of recurrent_dropout
, which will generate 4 dropout masks, but to be applied to the states instead of the inputs, each step of the recurrent calculations.
You are confusing Dropout
with it's variant SpatialDropoutND
(either 1D
, 2D
or 3D
). See documentation
(apparently you can't link specific class).
Dropout
applies random binary mask to input, no matter the shape, except first dimension (batch), so it applies to features and and timesteps in this case.Here, if
return_sequences=False
, you only get output from last timestep, so it would be of size[batch, 10]
in your case. Dropout will randomly drop value from the second dimensionYes, there is a difference, as
dropout
is for time steps whenLSTM
produces sequences (e.g. sequences of10
goes through the unrolled LSTM and some of the features are dropped before going into the next cell).Dropout
would drop random elements (except batch dimension).SpatialDropout1D
would drop entire channels, in this case some timesteps would be entirely dropped out (in the convolution case, you could useSpatialDropout2D
to drop channels, either input or along the network).