Am I punished for playing with the game clock?

You are punished for changing the game time (no matter how much, it seems), but not the system time. Here's what happens:

If you change the time on your 3DS system, Animal Crossing seems to simply think that's the time it is now. Trees grow, events change, the only "bad" things that happen would have happened with the flow of time. Even going back in time via this method seems to work okay, even when going to a day previously played.

If you change the game time, using the option at the main menu of Animal Crossing, you will randomly trip for a while after changing the clock. The duration of tripping seems to last longer than the current play session, possibly the full day? But it eventually goes away. If, however, you were to change the game time this way before every play session you would trip randomly every single session, which is highly frustrating.

Also note that going backwards in time will spoil any turnips you had, since the game knows you went backwards, while it doesn't know you went forwards if you go by the system clock.

tl;dr: Change your 3DS's system time to time travel, do not mess with the in-game clock regularly or will will randomly trip.