Why is binary_crossentropy more accurate than categorical_crossentropy for multiclass classification in Keras?

Short answer: it is not.

To see that, simply try to calculate the accuracy "by hand", and you will see that it is different from the one reported by Keras with the model.evaluate method:

# Keras reported accuracy:
score = model.evaluate(x_test, y_test, verbose=0) 
score[1]
# 0.99794011611938471

# Actual accuracy calculated manually:
import numpy as np
y_pred = model.predict(x_test)
acc = sum([np.argmax(y_test[i])==np.argmax(y_pred[i]) for i in range(10000)])/10000
acc
# 0.98999999999999999

The reason it seems to be so is a rather subtle issue at how Keras actually guesses which accuracy to use, depending on the loss function you have selected, when you include simply metrics=['accuracy'] in your model compilation.

If you check the source code, Keras does not define a single accuracy metric, but several different ones, among them binary_accuracy and categorical_accuracy. What happens under the hood is that, since you have selected binary cross entropy as your loss function and have not specified a particular accuracy metric, Keras (wrongly...) infers that you are interested in the binary_accuracy, and this is what it returns.

To avoid that, i.e. to use indeed binary cross entropy as your loss function (nothing wrong with this, in principle) while still getting the categorical accuracy required by the problem at hand (i.e. MNIST classification), you should ask explicitly for categorical_accuracy in the model compilation as follows:

from keras.metrics import categorical_accuracy
model.compile(loss='binary_crossentropy', optimizer='adamax', metrics=[categorical_accuracy])

And after training, scoring, and predicting the test set as I show above, the two metrics now are the same, as they should be:

sum([np.argmax(y_test[i])==np.argmax(y_pred[i]) for i in range(10000)])/10000 == score[1]
# True

(HT to this great answer to a similar problem, which helped me understand the issue...)

UPDATE: After my post, I discovered that this issue had already been identified in this answer.

First of all, binary_crossentropy is not when there are two classes.

The "binary" name is because it is adapted for binary output, and each number of the softmax is aimed at being 0 or 1. Here, it checks for each number of the output.

It doesn't explain your result, since categorical_entropy exploits the fact that it is a classification problem.

Are you sure that when you read your data there is one and only one class per sample? It's the only one explanation I can give.