Machine learning classifying algorithm with "unknown" class

One way to do this can be found in this paper - https://arxiv.org/pdf/1511.06233.pdf

The paper also compares the result generated by simply putting the threshold on the final scores and the (OpenMax) technique proposed by the author.

You should look at One-class classification. This is the problem of learning membership to a class, as opposed to distinguishing between two classes. This is interesting if there are too few examples of a second class ("not-in-class", let's say), or the "not-in-class" class is not well defined.

Where this popped up for me once was classifying Wikipedia articles for being flawed in some way - since it was not clear that an article not flagged as flawed was really not flawed, one approach was one-class classification. I have to add though that for my problem this did not perform well, so you should compare performance with other solutions.