Is there a negative stigma associated with choosing a university for non-academic reasons?
Assuming there is a negative stigma, why would you ever need to tell anyone why you picked the university you attended? Having a great passion for Japan and Japanese should be enough for anyone assuming you do an awesome project which you can and should do no matter where you go to school. This is especially true if you're likely to want to stay in Japan after your schooling to be an academic there, and it's a great reason to go to a Japanese university if you'd like to make a life there due to your passion.
As someone on the ground in Japan, I would say that there's more to consider than just the supposed quality of the university if you choose to do a graduate degree in Japan.
First, there's the school year. With the exception of ICU and the University of Tokyo, the school year here starts in April -- meaning if you finish a degree elsewhere in May, you will have an 11 month gap. (the same time gap will hit you on the return but less so. Since programs finish in March, it would not be impossible to start up somewhere else the next fall.)
Second, the gap in quality between institutions in Japan and elsewhere may be larger (or possibly smaller I guess) than people realize. The language barrier in many fields means that it may be hard to grasp the lay of the land within Japan, so the institution you go to here maybe much worse in quality than you thought. On the flip side, when people think about how good your program was, they will have the same information gap. (So they may think "oh, I've heard of X university and professor Q from there, it must be decent).
Even in programs and areas that publish in English or ostensibly hold courses in English, the opacity may still remain.
Based on the second reason, I would say merely doing a degree in Japan leaves no negative mark on your CV. I don't think it's ever hurt to be able to say I got 1.5 years of funding to go study in Japan at a decently ranked university there.
BUT because of the above opacity, it may be hard to tell how good the program you come to really is. If it is poor, then the quality of instruction may genuinely hurt your ability to do further research, i.e. you may not receive the same quality of mentoring and thus not be as prepared for Western-style independent research.
At least in my case, it didn't particularly help much of my research being in Japan, and I didn't learn any new techniques. (But I'm in philosophy where this opacity barrier is huge and I had already started writing my PhD thesis before I came).
If you are considering an MA/MS/PhD or study in Japan, rather than the JET Programme, you should focus on the MEXT Research Scholarship.