What do the "optional" resistor and capacitor do in this circuit?

The pin to which that R-C circuit connects to is the LX pin and that pin is connected to the power switches (an NMOS and a PMOS) of this converter, see the block diagram, figure 3 in the datasheet.

To keep the DCDC conversion efficient, these switches switch on/off quite rapidly. That causes the voltage on the LX pin to go up and down with high speed. This steep slopes cause EMI (Electro Magnetic Interference) emissions. So the circuit will radiate RF signals.

That is normal and to be expected and does not have to be an issue depending on your application. If it is an issue a possible solution is to make these steep slopes a bit slower, that's what this RC snubber network does. It might cost some power efficiency though so that's why the circuit is optional.

Another solution could be to place this DCDC converter in a shielded cage (Faraday cage), this can be a small metal cover on the PCB. This is used in nearly all smartphones as the DCDC converters should not disturb the phone's reception.

In this particular application, the use of an RC snubber on the SW node is to prevent Electrical Over Stress (EOS) of the LX pin / SW node.

The FP6717 boost switching regulator employs a synchronous rectifier to achieve a high DC-DC conversion efficiency. A caveat of a synchronous rectifier (logic driven pass FET) is typically an even slower rectifier turn-on time as compared to a high-speed rectifier diode.

Note the following absolute maximum voltage specification for the LX pin of the FP6717 in the datasheet:

Now, note the following scope-shot of the FP6717 operating in a 5 V demo circuit:

Notice that, for a brief period the SW node (LX Pin) rises to within 200 mV of the absolute maximum voltage rating of the converter.

Since the high-side synchronous rectifier must include a finite dead-dead time in order to avoid inadvertently crowbarring the output filter capacitors with the low side NMOS switch. For a brief period the inductor is allowed to kick-back on the switch node un-clamped (or marginally clamped via the body diode of the converter) resulting in EOS of the converter IC.

The late Jim Williams authored a nice application note on a very similar topic which applies equally well here titled: Diode Turn-On Time Induced Failures in Switching Regulators

The RC snubber also aids in EMC as others have outlined, but I believe EOS to be the #1 motivation in this application.

I have worked with large Thyristor power supplies before. Another reason for the snubber circuit to limit the rate of voltage change, is that some components are sensitive to high dV/dt. Not that this is the reason in this particular application though. As others have said, it is more for EMI and to protect against transient spikes.