PCB cutout specification

One way to handle this is to place a drilled out hole at each corner. Then the milling corner is not a problem. Your cut out corner then ends up looking like the image below. Another alternative is to have the route go past the corner as shown in the second image.

I used a cutout that looked like this for a snap in plastic switch component:

The above style of the cutout was also used on another board design to accommodate a plastic screw standoff that had a rectangular snap-in profile so it would not spin as a security screw was installed.

You can define a hole in each corner (hole must be bigger radius than the routing tool) or modify the routing at the corners to remove all the material (extend the routes in one axis or add a stub route that cleans out the corner at a 45° angle). Called a 'dog bone' for obvious reasons.

For example (from here):

If you use the hole method you probably want to specify an unplated hole. The above diagram also shows you a possible size of hole and position.

Edit: The above page is sub-optimal as is extending the routed line if you want to minimize the volume of material removed beyond the required amount.

I think the minimum material removed would be a hole (must be drilled since it is of radius less than R) with center at -R/2, -R/2 from the edge and diameter \$\sqrt{2}R\$

Comparing the figure of merit of area of unnecessary material/radius squared (lower numbers are better):

Extend routed line: 1.57

Circle of radius R: 0.571

Pefhany method: 0.285

It's also possible (from some vendors- you can ask) to do drilled slots which can be very small radius.