How to cartoon-ify an image programmatically?

You could try rotoscopy, like toonyphotos.com does:

Here's some algorithms to play with:

• Median or repeated box blur filter to obtain cartoonish color palette
  • Edit: Bilateral filtering should suit your needs even better
• Min filter (zeroth percentile) to enhance some types of edges
• Color image segmentation using either small subcube or sphere in the RGB color cube
• Generic edge enhancement on segmented image using edge detection such as Sobel kernels or 8-way edge tracing
• Composit blurred/median-filtered image with enhanced edges

These are fairly basic and all very easy to implement. Keep in mind that median and box blur filters can be implemented with linear time complexity w.r.t. the kernel radius.

More edits:

Once you get the idea of Huang's algorithm, implementing a box blur filter is a delicious piece of cake.

Reading material:

• Fast Median and Bilateral Filtering (get the PDF)
• Median Filtering Constant time (get the PDF) Note: I have an implementation of this in C# using Mono/SIMD to accelerate histogram coalescence, however it only seems better than the O(r) algorithm when the diameter exceeds ~60 pixels due to the comparable number of add/sub instructions (the break-even point), a C++ implementation is probably much better suited to harness SIMD.

Other reading materials include Gonzalez & Woods' Digital Image Processing (seems to be an older edition) for segmentation and edge tracing. 8-way edge tracing can be really hard to bend your head around (choosing between on-pixel or between-pixel edges and how to latch onto edges). I'd be happy to share some code, but the hundred-liners don't exactly fit smoothly in here.

You might want to check out Freestyle, an open-source (Google Summer of Code, even) project to implement a non-photorealistic renderer for Blender. Here's an example of its output, in cartoon-mode:
_{(source: sourceforge.net)}