What to do about very hot planar inductor?

The problem is that the cores being used have no gap, so the inductor saturates. A topology with Flyback type operation stores energy in the core when the switch is on, moving the core up the BH curve. But, for the simple example of discontinuous conduction (DCM), when the switch turns off and current falls to zero, B does not return to zero but to an elevated residual point. So, the usable \$\Delta B\$ is very small and the core is walked into saturation. Continuous conduction (CCM) is even worse in this regard.

Adding a gap moves the residual point nearer to zero, giving a usable \$\Delta B\$. In the case of a gap, the inductance will be determined by the gap, not so much the core \$\mu\$. Consider the inductance of a gapped core inductor; with core cross section \$A_c\$ and gap length \$l_g\$ and winding turns \$n\$:

\$L_g\$ = \$\frac{n^2 A_c \mu _o}{l_g}\$

also relating maximum winding current (\$I_{\text{max}}\$) to gap flux (\$B_{\text{max}}\$):

\$n I_{\text{max}}\$ = \$\frac{B_{\max } l_g}{\mu _o}\$

By starting with a value for \$L_g\$, \$B_{\text{max}}\$, \$A_c\$, and \$I_{\text{max}}\$, it's possible to get an idea what \$l_g\$ and \$n\$ for the inductor would need to be. Let \$L_g\$=100\$\mu H\$, \$B_{\text{max}}\$=0.2T, \$A_c\$=20\$mm^2\$

\$l_g\$ = \$\frac{I_{\max }^2 L_g \mu _o}{A_c B_{\max }^2}\$ = \$\frac{1 Amp^2 100\mu H \mu _o}{20 mm^2 0.2T^2}\$ ~ \$0.16 mm\$

and

\$n\$ = \$\frac{i_{\max } L_g}{A_c B_{\max }}\$ = \$\frac{1Amp 100\mu H}{20 mm^2 0.2T}\$ = \$25 turns\$

This analysis is pretty simplified, leaving a lot out, but gives an idea of what to expect. Designing these types of inductors gets very involved. You might look at "Inductor and Flyback Transformer Design" as a reference.

I think you are using N87 material so I'm going to do a quick calculation of stuff. At 500 kHz the inductor current can rise to a certain value in 1 microsecond (50:50 duty cycle). You say it has an inductance of 244 uH so with 18V applied I expect the current to rise up to: -

18V x 1 us / 244 uH = 74mA - this is the magnetization current (it stores the enrgy that is released in the next half cycle) but it sounds really, really low. The energy stored up in the main winding has to transfer to the output and this energy is 0.66 uJ (still sounding very low). The power that can be transferred to a load is therefore 0.66 uJ x 500 kHz = 0.33 watts.

I think you need to look at other examples in that data sheet you linked. I see one that can work with voltages as high as 30V and operating at 300 kHz using an inductor of 150 uH so I think your main losses are copper losses in the windings - how did you fabricate these?

I'll also point out that N87 material is going to give you about 5 to 10% losses at 500 kHz too so it's probably not the best choice.

Added to this make sure that the output winding produces a negative voltage when positive is applied to the primary. In other words the phasing of the windings is fundamental to this type of flyback circuit.

My reasoning about this discontinuous mode assessment is that although you may be expecting to run in continuous conduction mode you can get a reasonable idea by looking at it in DCM and trying to work out whether DCM is in the right ballpark.

The hole for the center leg of the core on the PCB looks plated in the figure. Is it plated in the actual PCB? If it is, that explains why you might have large currents. You have a shorted turn that gets coupled thru the core.