Rounded corners in XSL-FO
You can create the table as a Scalable Vector Graphics(SVG) object, and include that as an external-image in your XSL-FO document. SVG supports rounded corners, and FOP supports SVG.
I believe you can also just create a rounded rectangle SVG and use that as a background to your content, and put the table in front of it. I think I did this once, but I cannot seem to locate the code...