Keras - how to get unnormalized logits instead of probabilities
try to change last activation from softmax to linear
model = Sequential()
model.add(embedding_layer)
model.add(LSTM(n_hidden, return_sequences=False))
model.add(Dropout(dropout_keep_prob))
model.add(Dense(vocab_size))
model.add(Activation('linear'))
optimizer = RMSprop(lr=self.lr)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy')
I think I have found a solution
First, I change the activation layer to linear such that I receive logits as outlined by @loannis Nasios.
Second, to still get the sparse_categorical_crossentropy
as a loss function, I define my own loss function, setting the from_logits parameter to true.
model.add(embedding_layer)
model.add(LSTM(n_hidden, return_sequences=False))
model.add(Dropout(dropout_keep_prob))
model.add(Dense(vocab_size))
model.add(Activation('linear'))
optimizer = RMSprop(lr=self.lr)
def my_sparse_categorical_crossentropy(y_true, y_pred):
return K.sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)
model.compile(optimizer=optimizer,loss=my_sparse_categorical_crossentropy)