Can the place of publication be harmful to one's reputation?

Yes, the place of publication can absolutely hurt your reputation. Specifically, I can tell you from having served on many hiring committees (and from conversations with professors at other universities about their hiring committees and tenure processes), that publications in predatory journals can hurt you. I'm talking specifically about journals whose model is to get the author to pay them, and whose peer review standards are a joke. Publications in journals like that can be interpreted as an author trying to side-step the normal process, or unethically inflate their numbers.

It may be hard to break into the absolute top journals with your first few papers (unless you have a famous advisor/coauthor or went to a prestigious school). But there are plenty of good journals around and after a track record of publishing in good journals you will have less difficulty publishing in top journals (of course, it'll always be extremely hard to publish in the Annals and other super elite journals). For people starting out, I recommend at least checking Beall's list of predatory publishers to be sure you don't end up publishing somewhere that might be frowned upon later in your career. Also, don't let fear paralyze you from trying. Lots of editors and referees will go gently on new PhDs. I wish this was even more common, rather than pushing young people out of academia.

If you want to build a reputation as a mathematician, post your preprint on the arXiv (this is a preprint server, not counted as publication, the posts are not refereed, and there are essentially no "barriers").

Then send your paper to a mainstream mathematical journal. Avoid those journals which are not reviewed in Mathscinet. (Recently, many journals proliferated which do not really referee papers, but charge the authors for publication. Avoid them, if you want to build a reputation. The main criterion of a mainstream journal is that all its papers are reviewed in Mathscinet.) Mathscinet also publishes a rating of journals according to their citation rates. Publication in a highly rated journal (say of the first 100) will probably be good for your reputation, but the "barriers" there are also high.

Still I believe that one's reputation is based more on the quality of papers, rather then the rating of the journals where they are published. For example, the reputation of G. Perelman is mainly based on a few preprints which he posted on the arXiv. He did not care to send them to journals.
This is an extreme example, but it is not unique. My 3-d highest cited paper is published in a volume of conference proceedings which is not even a journal (arXiv did not exist yet). My two most cited papers are in the journals which do not enter the list of 100 top journals according to MSN rating (though I also published in top journals). I conclude that there is very little correlation between the rating of an individual paper and the rating of the journal.