Is it ethical for a journal to request substantial payment for publishing a paper?
Well someone's got to pay. If it's the reader, then howls go up about Open Access. So that leaves the authors.
The costs of publishing
A high-quality journal needs high-quality editors. An awful lot of literature still gets printed and distributed around the world and that's expensive. Remember, reading this, we are the ones who are most active online, and so our paper-reading habits are much more likely to be skewed towards predominantly (or even exclusively) online access, so are not fully representative. High-quality journals often also get involved in conference sponsorship, publicity, and so on. The peer-reviews get co-ordinated; there is often some vetting of suitable reviewers; their responses need to be interpreted correctly; special issues get co-ordinated; someone ensures a good balance of articles covering the journal's remit.
So, there's a lot of costs, and someone's got to cover them. If we want to publish at almost-zero overheads, we write a blog. A journal is much much more than a blog - and some of the things that make it much more than a blog, cost money.
Not necessarily an efficient market
That's not to say that in certain sectors, there isn't a market failure: some sectors do have suspiciously high profit margins. And market regulators should be looking at removing barriers to entry. Those barriers certainly aren't insurmountable, as notable new scientific publishers have emerged recently, with new business models: PLOS being an obvious example, founded in 2000 to support Open Access, and becoming a publisher in its own right in 2003; and now a serious player, using the author-pays model.
Author beware
There are also plenty of scammers using the author-pays model. So be careful out there. Talk to colleagues about who's reputable and who's not. Read the journals. And there are lists online of the disreputable publishers. Read more about that on this site at How do you judge the quality of a journal?
A market in which authors pay to publish and their careers depend on the number of publications they have opens new possibilities for unethical practices, and those are being actively exploited by vanity press "gold open access" journals. On the other hand, there are very reputable journals like PlosONE that charge authors but maintain highly ethical standards.
Personally, I think open submission is even more important than open access. For a more thorough criticism of the author-pay model, see these articles from the Notices of the American Mathematical Society.
This is an answer about the cost of publishing science article.
A math paper, where the author gives a pretty good draft (we use LaTeX) still has to manage peer-review (which usually means paying a secretary to contact authors, late referees, and so on) and to typeset articles. I have heard that the overall cost is $50 per page for a cheap journal (and math pages tend to be small since we mostly write with a single column).
It has been computed in the (very interesting) blog SV-POW that subscription earn the publisher more than $5000 on average. But of course, we are cheated a lot with those prices.
PLoS ONE gains money (and reinvest it, since PLoS is non-profit) and charges "only" $1350 per article.
Recently, Cambridge University Press announced the launch of a math journal in gold OA for $750 a paper -- after three years of fee waivers for everyone.
Last, PeerJ claims to be able to publish all the paper one wants for a few hundred dollar, paid once in one's life (plus yearly reviewer duty).
My conclusion: there is a wide range of prices, and a wide range of services a journal can provide, or not.
Concerning the actual question, I would say there is no ethical problem with paying a publisher for the work it does, as soon as the editors are independent from the money flow; but there are some issues with a system where most journals would run such a model.