How to disable dropout while prediction in keras?
To activate dropout for inference time u simply have to specify training=True
in the layer of interest (Dropout
in our case):
with training=False
inp = Input(shape=(10,))
x = Dropout(0.3)(inp, training=False)
x = Dense(1)(x)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)
X = np.random.uniform(0,1, (1,10))
output = []
for i in range(0,100):
output.append(m.predict(X)) # always the same
with training=True
inp = Input(shape=(10,))
x = Dropout(0.3)(inp, training=True)
x = Dense(1)(x)
m = Model(inp,x)
# m.compile(...)
# m.fit(...)
X = np.random.uniform(0,1, (1,10))
output = []
for i in range(0,100):
output.append(m.predict(X)) # always different
by default, training is set to False
HERE a full example of the usage of droput at inference time
Keras does this by default. In Keras dropout is disabled in test mode. You can look at the code here and see that they use the dropped input in training and the actual input while testing.
As far as I know you have to build your own training function from the layers and specify the training flag to predict with dropout (e.g. its not possible to specify a training flag for the predict functions). This is a problem in case you want to do GANs, which use the intermediate output for training and also train the network as a whole, due to a divergence between generated training images and generated test images.
As previously stated, dropout in Keras happens only at train time (with proportionate weight adjustment during training such that learned weights are appropriate for prediction when dropout is disabled).
This is not ideal for cases in which we wish to use a dropout NNET as a probabilistic predictor (such that it produces a distribution when asked to predict the same inputs repeatedly). In other words, Keras' Dropout layer is designed to give you regularization at train time, but the "mean function" of the learned distribution when predicting.
If you want to retain dropout for prediction, you can easily implement a permanent dropout ("PermaDropout") layer (this was based on suggestions made by F. Chollet on the GitHub discussion area for Keras):
from keras.layers.core import Lambda
from keras import backend as K
def PermaDropout(rate):
return Lambda(lambda x: K.dropout(x, level=rate))
By replacing any dropout layer in a Keras model with "PermaDropout", you'll get the probabilistic behavior in prediction as well.
# define the LSTM model
n_vocab = text_to_train.n_vocab
model = Sequential()
model.add(LSTM(n_vocab*4,
input_shape=input_shape,
return_sequences=True))
# Replace Dropout with PermaDropout
# model.add(Dropout(0.3)
model.add(PermaDropout(0.3))
model.add(LSTM(n_vocab*2))
# Replace Dropout with PermaDropout
# model.add(Dropout(0.3)
model.add(PermaDropout(0.3))
#model.add(Dense(n_vocab*2))
model.add(Dense(n_vocab, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])