Programmatically convert a video to FLV
You basically have two choices if you want to host, transcode and stream flv files (and don't want to buy a video transcoding application): you can call out to FFMpeg/MEncoder or you can use an external Web service. You could also sidestep the problem completely by allowing them to embed YouTube videos on your site.
If you go the 'local FFMpeg route' I would suggest simply using ProcessBuilder and constructing a command-line to execute FFMpeg. That way you get full control over what gets executed, you avoid JNI, which is an absolute nightmare to work with, and you keep OS-specific code out of your app. You can find FFMPeg with all the bells and whistles for pretty much any platform. There's a good chance it's already on your server.
The nice thing about the 'Local FFMPeg' route is that you don't have to pay for any extra hosting, and everything is running locally, although your hosting admin might start complaining if you're using a crazy amount of disk and CPU. There are some other StackOverflow questions that talk about some of the gotchas using FFMpeg to create flvs that you can actually play in the flash player.
The Web service route is nice because there is less setup involved. I have not used Hey!Watch but it looks promising. PandaStream is easy to set up and it works well, plus you get all your videos on S3 with no additional effort.
There's a great open source tool call FFmpeg that I use to transcode my videos. I use PHP making shell calls to make it come to life, but I can't imagine that it would be too hard to get it to play nice with Java. (Maybe this could be a good starting point for you.)
I feed my installation 30+ gig batches on a weekly basis and it always comes out as quality material. The only tricky part for me has been getting it compiled to handle a wide variety of video formats. On the bright side, this has provided me with heavy lifting I need.
You can encode video in Java using Xuggler, which is a Java API that natively uses FFmpeg's C code behind the scenes.