What happens if the editor cannot find reviewers?
Short answer: the editor will continue to try and find other suitable reviewers, sending them invitations and waiting for replies. At some point, they may come back to you and ask you to suggest names (additional names, if you had already given some when you originally submitted the paper).
It can happen that an editor has a hard time finding suitable reviewers that accept to review a manuscript. There are a few factors in play, such as:
- the journal is not well-know;
- the editor is new, or not well-know in the field, or does not have a very good network;
- the research reported is atypical, in a very narrow subfield, or joins different areas (so that no one reviewer feels confident in accepting)
- the elements sent by the editor (title and abstract) are rather boring or tend to confuse the potential reviewers
Note that most of these factors do not reflect badly on the manuscript, so there is no need to feel bad about it. The editor is likely to continue his search of reviewers. I see two other options that the editor could choose from, but I judge them as rather unlikely:
- Evaluate the manuscript himself, and make a decision based on his own review. After all, it's his job, and the reviewers' role is only to help the editor reach a decision.
- Refer the manuscript to another editor, or to the editorial board, so they can make a decision on it.
Finally, note that it may take quite some time to find suitable (and willing) reviewers. In the case of one paper of mine, it took the editor 3 months to find adequate reviewers, and that was actually for a prestigious journal. However, the paper was atypical enough (and the research was quite novel) that many potential reviewers did not feel able to review it adequately. (And in case you wonder, it was accepted on the first try, once the editor found reviewers.)
Edit (regarding your comment): my advice is don't retract your submission, unless you think it's the journal's fault (unwilling editor or unknown journal). I know it's tempting! But especially if your work is multidisciplinary, it will take time to find reviewers, even if you submit it to another journal. Thus, better let the current submission process go to its end.
F'x's answer looks right to me, but since this actually happened to me once, I'll add some extra commentary which doesn't sound like it applies to your situation, but might apply to similar ones.
Many journals (at least in math) have a de facto policy that every paper which even plausibly might get published has to go to a referee before a decision can be made. Therefore an article which has problems which make it very unlikely to get accepted---being difficult to read, or highly specialized, or involving multiple areas with few potential referees---but which doesn't have obvious flaws which make it an easy rejection might have trouble getting through the process. The editor keeps asking people to review it, and it looks like a hard job, and the paper isn't so interesting to be worth it, so they keep turning it down. An editor can eventually break this loop by asking a reviewer in a way which implies that the reviewer has permission to give a more cursory rejection for the reasons the editor has identified (perhaps based on comments made by other people when turning down the review).
Once an editor has decided to start nudging potential reviewers in this direction, it becomes faster for everyone if the editor politely suggests that the paper should just be withdrawn.
This (I think---I can't say for sure what the editor was thinking) is what happened to me. I submitted a paper involving two different areas to a journal specializing in one of them. After several months I got back a letter from the editor stating that several people had turned down the paper with negative impressions; the editor offered to look for more referees, but before doing so, wanted to know if I would like to withdraw the paper rather than continue to wait when it wasn't clear the paper would be accepted. I (and people I consulted with) interpreted this as a polite indication that at this point the editor would primarily be looking for an excuse to reject the paper, and therefore that it would speed things up for me if I withdrew it. (And, indeed, the next journal accepted it.)
So a notification that the editor can't find reviewers could be a hint that the editor wants you to withdraw it, but isn't necessarily.
It happened to me once that the editor sent my submitted paper to 5 reviewers (all of them were in my references) and all of them rejected the invitation to review the paper. The editor took this as a sign that my paper was not suitable for publication in that journal and rejected the paper. So, yes, a rejection can actually happen but it is not the rule. As usual, publshing is a combination of good content, a bit of politics, and luck.