Should I tell an editor that I believe an article I'm reviewing is not good enough for the journal?

Referees and/or editors should not invite the authors to submit a revised version unless there's a real chance that the revision will be accepted. Also, most things could be made significantly better than they are. If your feeling is that the paper, even though not flawed in any way, does not meet the standards of the journal, then I think you should recommend rejection unless you can identify explicitly which improvements would change your mind. Even then, in a field with a sufficiently large number of reputable/good/great/... journals (my own field, mathematics, is certainly like this), it would be reasonable to suggest improvements that could make the paper suitable for a journal of the caliber submitted to while rejecting the paper. A top journal is going to get more than enough papers that are good in every way than it can publish. And finally, if you revise and improve your paper enough then at a certain point you can justly claim that it's not the same paper. Nothing stops authors from submitting the new, better paper to the same journal. (Once I was added as a coauthor on a paper after the first version was rejected by a strong journal in the field. I saw how to improve the paper so significantly that I thought: why not send it back to the same journal? They accepted it.)

Having said all that: from the comments it doesn't sound like the authors spent much time and effort making significant improvements to their paper. So while there may have been a miscommunication of some sort among you, the editors and the authors, it doesn't seem to have caused much harm to them. In my opinion you should reject the paper for the reasons that you say: there's nothing wrong with it, but it's not good enough for the journal. A top journal must reject many papers for this reason -- that's what makes it a top journal.

(If they had spent a lot of time and effort: well, I would feel bad about it if I were you, but it doesn't really change the decision to reject the paper. Anyway the time they spent improving their paper did after all result in an improved paper, which they can then submit to another journal.)

Maybe the lesson is to be more clear in the future. It feels nice to give people a chance, but in fact you're being asked to apply your professional judgment in a way that's going to have a negative outcome much of the time. When your judgment is negative, the nicest you can be is to deliver the bad news relatively quickly and with sufficient justification.

Finally, the fact that the editors asked you to referee it again might or might not mean that they are more positive about the paper than you -- you have no way of knowing. Anyway, their opinions are their opinions, but they're asking you for yours. Once or twice I have been asked to referee a second draft of a paper that I recommended for rejection the first time. I did look at the revised versions, quickly, but my answer was essentially: "I told you that I didn't think the paper was suitable for your journal. If you disagree, okay: ask someone else to look at it."

Actually, don't assume anything. They sent it to you for your honest evaluation and feedback. Give it to them. But, make it complete. Say what is good, but also where you think it fails to meet the standard, either of scholarship in general or the standards of the journal.

Don't make recommendations to the editor about the alternatives, I think. But if you are "kinder than you should be" then it will, over time, bring down the standards of the journal.

One point I have not seen raised here that I think you should pay attention to in the future: if you think the paper is not up to the journal's standards you should say so in your first referee report. Suggesting changes too would be a courtesy to the authors.

Replying just with the list of changes and saving your reservations for later may cost the editors time and raise false hopes for the authors.

That said, in this case I agree with the other answers that say you should tell the editor you don't think the paper is good/string/important enough for this journal.