Looking for more realistic 3D terrain renderer than ESRI Arcscene
Dynamic or real-time viewing
For real-time viewing, start with the Virtual Terrain Project. Open source from a proficient developer it has native GIS smarts. Even if you opt not to use VTP the website itself is the best curated list of data and white paper resources on terrain modelling I'm aware of -- including short evaluations of other 3D terrain software ;-)
If you're more of a hacker/developer type and comfortable with text files as design and control tools osgEarth might be the ticket. Open source with an active community.
I don't consider Google Earth a viable contender, even though it's performance and smoothness make it a better tool than ArcScene for many circumstances. Why? You can't load your own terrain or base imagery, only work on top of what they already provide.
Static or render-then-view
On the static rendering side best of breed would be Visual Nature Studio and World Construction Set from 3D Nature / AlphaPixel. It has detailed control over atmospherics, vegetation, buildings, water with waves and reflections, and integrates well with GIS data. However that extra control comes with a learning investment.
A couple of notable applications I'm aware of that approach world rendering from the design and artistic creation side are Terragen, already suggested in the comments, and Blender. Both can produce spectacular results, but ingesting GIS is an afterthought so there may be a fair bit of data munging work to get where you want. Of the two Terragen is at least focussed on world building while for Blender it's entirely a sideline.
That said, Blender is an open source project and there is a some activity to improve it's terrain abilities; the generic BlenderGIS and Open Street Map specific Blender-geo data importers for two.
Another worthy of mention is Bryce, also approaching from the artistic side.
2016-Jan-29: added Terragen and Blender
2016-Feb-01: Incorporate Xenon's comment, split into dynamic/static, add Bryce, osgEarth; not-add Google Earth