PyTorch - How to deactivate dropout in evaluation mode

I add this answer just because I'm facing now the same issue while trying to reproduce Deep Bayesian active learning through dropout disagreement. If you need to keep dropout active (for example to bootstrap a set of different predictions for the same test instances) you just need to leave the model in training mode, there is no need to define your own dropout layer.

Since in pytorch you need to define your own prediction function, you can just add a parameter to it like this:

def predict_class(model, test_instance, active_dropout=False):
    if active_dropout:
        model.train()
    else:
        model.eval()

You have to define your nn.Dropout layer in your __init__ and assign it to your model to be responsive for calling eval().

So changing your model like this should work for you:

class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)

        # define dropout layer in __init__
        self.drop_layer = nn.Dropout(p=p)
    def forward(self, input):
        out,_= self.lstm(input)

        # apply model dropout, responsive to eval()
        out= self.drop_layer(out)
        out= self.linear1(out)

        # apply model dropout, responsive to eval()
        out= self.drop_layer(out)
        out= self.linear2(out)
        return out

If you change it like this dropout will be inactive as soon as you call eval().

NOTE: If you want to continue training afterwards you need to call train() on your model to leave evaluation mode.

---

You can also find a small working example for dropout with eval() for evaluation mode here: nn.Dropout vs. F.dropout pyTorch