Rounding the Corners of a Shape

UPDATE:

The previous version of my answer worked, but did not give control on the rounding radius, nor did it fully work with as a starting point for a geometric region for further calculations. Here is a version that is still based on spline curves, but it gives full control over the corner rounding radius. It also returns a FilledCurve object that in my opinion is easier to style and can also be discretized reliably to use in further calculations.

Clear[splineRoundedNgon]
splineRoundedNgon[n_Integer /; n >= 3, roundingRadius_?(0 <= # <= 1 &)] :=
  Module[{vertices, circleCenters, tangentPoints, splineControlPoints},
   vertices = CirclePoints[n];
   circleCenters = CirclePoints[1 - Sec[Pi/n] roundingRadius, n];
   tangentPoints =
   {
    Table[RotationMatrix[2 i Pi/n].{circleCenters[[1, 1]], vertices[[1, 2]]}, {i, 0, n - 1}],
    Table[RotationMatrix[2 i Pi/n].{circleCenters[[-1, 1]], vertices[[-1, 2]]}, {i, 1, n}]
   };
   splineControlPoints = Flatten[Transpose[Insert[tangentPoints, vertices, 2]], 1];
   FilledCurve@BSplineCurve[splineControlPoints, SplineClosed -> True]
]

Here's the obligatory animation :-)

Animate[
 Graphics[
  {EdgeForm[{Thickness[0.01], Black}], FaceForm[Darker@Green], 
   splineRoundedNgon[5, radius]}
 ],
 {{radius, 0, "Rounding\nradius"}, 0, 1}
]

And here is an example of a discretized region obtained from it:

DiscretizeGraphics[splineRoundedNgon[5, 0.3], MaxCellMeasure -> 0.001]

Such regions can be used e.g. as domains for plotting and in NDSolve calculations. For instance:

Plot3D[
  y Sin[5 x] + x Cos[7 y], {x, y} ∈ DiscretizeGraphics@splineRoundedNgon[5, 0.4]
]

---

You can also create a spline curve to get a bit more roundness in the corners than allowed by JoinedForm. You need to double each control point in your spline definition to have the spline "hug" the points more closely. This is conveniently wrapped up in the roundRegPoly helper function below:

Clear[roundRegPoly]
roundRegPoly[n_Integer /; n >= 3] :=
 FilledCurve@BSplineCurve[
   Flatten[#, 1] &@Transpose[{#, #}] &@CirclePoints[n],
   SplineClosed -> True
 ]

Graphics[
  {Darker@Green, EdgeForm[{Thickness[0.01], Black}], roundRegPoly[5]},
  PlotRangePadding -> Scaled[.1]
]

Just wanted to add purely mathematical approach using complex mapping technique.

 PolyMap[n_, z_] := z Hypergeometric2F1[1/n, 2/n, (n + 1)/n, z^n]
(* Integrate[1/(1 - ξ^n)^(2/n), {ξ, 0, z}] *) 

g = GraphicsGrid[
Table[
 ParametricPlot[
  z = PolyMap[n, r (Cos[t] + I Sin[t])]; {Re[z], Im[z]}, 
   {t, 0, 2 π}, PlotRange -> All, Axes -> False] /. 
   Line[l_List] :> {{Lighter[ColorData[3, "ColorList"][[n]]], Polygon[l]}, {Red, Thick, Line[l]}}, 
 {n, 3, 8}, {r, 0.799, 1., 0.1}], 
ImageSize -> 400]

Since you mention that you want to use the rounded polygon in NDSolve[] as a region, you might want to look at the following construction:

With[{r = 1/5 (* rounding radius *)}, 
     rp = DiscretizeRegion[
          ImplicitRegion[RegionDistance[
          Polygon[CirclePoints[{1 - 2 Sqrt[5 - 2 Sqrt[5]] r, π/10}, 5]], {x, y}] <=
          r Sqrt[(5 - Sqrt[5])/2], {x, y}], MaxCellMeasure -> 1/200]];

Graphics[{{Yellow, Polygon[CirclePoints[{1, π/10}, 5]]},
          {Opacity[2/3, Blue], MeshPrimitives[rp, 2]}}]

Rescale/rotate/translate as needed.