As a referee, is it okay to ask for references to be removed?
Yes, the spurious references should be removed, as stated by @Solar Mike. There is a diplomatic way to accomplish this without making any assumptions or accusing anyone of anything. Write something along the lines of
The current bibliography contains a number of references which are not explicitly cited in the main text. They are: (insert complete list of numbers). The main text should either be extended to include a discussion of these references or the references should be deleted.
You compile the list by ticking of all explicitly cited references as you read through the main text. Any entries left are spurious.
A tool like LaTeX typically has a command which is used to print the entire bibliography. Spurious references are often generated by forgetting to disable this command when the final draft is compiled. If you can identify the tool, then you can insert a helpful statement to that effect, i.e.
As you are using LaTeX, please check if manuscript contains a \nocite{*} command.
EDIT: I just realized that my original text does not cover the case where the manuscript contains references which are cited in the main text, but appear to be irrelevant. A diplomatic approach can proceed along the lines of:
The author(s) should extend the discussion of the following references: (insert complete list of dubious references). Currently, it is not clear to the reader how these references relate to the subject under discussion.
This allows the authors to either explain why the references are relevant or remove them if they are irrelevant.
Let me suggest, more generally, that as a reviewer you have two tasks. One is to help the editor make a decision about acceptance. But the other is to help the author(s) improve the work. As such, you can suggest anything to the author that you believe will result in an improvement.
Authors don't have to take your advice, necessarily, though they are advised to consider it. Neither do editors have to take your advice.
Be professional and do your best job. In the current instance, as in many, it would be good to state to the author why you believe the references should be removed. Both the editor and the author can take that advice and act on it or not.
When it comes to suggesting changes, it is true that most reviewers will suggest where to add material. As the general purpose of these suggestions for changes is, however, to make the paper "better" (whatever that means in the concrete case), removing information can be fully compliant with that goal.
Now, you list two very different situations:
there are cited references that are not related to the discussed subject
These references do, by their mere presence, reduce the quality of the paper (if we assume a paper should be as concise as possible when it comes to providing information unrelated to the subject of the paper). You cannot truly tell whether such references were added with malicious intentions (for example, to increase someone's citation count) or simply by mistake, so you probably should not make any direct accusations, but a request to remove the respective references is well appropriate.
or there are elements in the bibliography that are not referenced in the manuscript.
This case, on the other hand, depends entirely on the guidelines of the venue you are reviewing for (provided that the references are indeed on topic for the paper):
- If these guidelines require references to all bibliography items within the paper body, go ahead and suggest either their removal or adding a mention in the paper body.
- If, on the other hand, the guidelines do not contain this requirement, keeping these references improves the paper, as it helps readers who browse bibliographies for related work in the subject area by pointing out additional resources.