How to review a manuscript where the software is difficult to install?

You should ask your editor for advice on this. I doubt, however, that you should reach out to the authors in any way. But the editor can, on your behalf, ask the authors for specific instructions for installing the software on your OS. The editor can make this a condition of getting a review.

Moreover, the editor can communicate your concerns about missing files or whatever else is important to complete a review.

If the protocols/software implementation are a requirement for the journal rather than just the concepts, which sounds like the case, I would say there are two scenarios:

1.) Given a reasonable effort and appropriate background on your part, you were not able to get their software to run. All else in the paper seems fine. In this case, I might suggest "Reject and resubmit". Given that you are not able to test their code, you can't really give it a thumbs up or thumbs down, but it should be a relatively small step for them to meet this requirement and then you can adequately evaluate their software.

2.) You are able to use their software, but it took an unreasonable amount of effort or expertise on your part to get it done. In this case, you can evaluate their software, but ideally they should make it more readily available to the target audience. In this case, under the condition that you liked everything else about the paper, it may be reasonable to suggest "Accept with minor/major revisions", and you ask them to include the necessary instructions that will help guide other users of the code.

It sounds from your description that the software is an essential part of the paper. Well, if the paper was missing one of its normal text sections due to sloppiness of the authors, would you be blaming yourself for that and going on a wild goose chase around the internet to find the missing section? I suspect not. The same should apply here. As referee, it is not your job to become an expert in compiling Linux binaries from source code (or whatever similar tasks you apparently need to do to get a working binary to test).

Similar to what @Buffy suggested, what I would do is write to the editor and inform them that I am unable to finish the refereeing assignment since not all parts of the work that need evaluation were provided to me. I would make clear that this is not a rejection, and that I am happy to finish the job once this situation has been remedied. It’s the editor’s job to communicate with the authors, so leave that part to them.