How should I compare my algorithm to others when their code is not available?

Asking the authors for the code is of course always an option, but from my personal experience this almost never works (there is a reason why they did not put it online in first place).

I don't know which area in computer science you are from but normally there is some quantitative measure to determine which method is better and those measures are somehow in the paper. Those can be things like run time or precision-recall curves on certain data sets.

If you want to compare on the same dataset, you can ask the authors for the numbers / plots themselves and then plot your own curves together with them.

If you want to compare on a different data set, you can ask the authors to run the code for you. In my experience, this works more often because many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it.

If you have Bachelor/Master students to supervise, you can give the implementation as a project to them (although you need to give them enough time to do it).

In my personal opinion, it is not worth the effort to implement a method which is not described in enough detail anywhere. It will probably cost you an unreasonable amount of time without even reproducing the quality shown in the paper (for various reasons).


What's wrong with being honest?

We compare our algorithm to A and to B. We would have liked to compare also with X, Y and Z, but code is not available for them.