Is it okay to keep a USB charger plugged into the cigar lighter?

A good quality adapter will probably draw some current - but not much.
Strangely - a poor quality adapter MAY draw none.

A good quality adapter will use a switching regulator to step the voltage down - probably a "buck regulator" The quiescent (no load) current draw will vary depending on the design but I'd guesstimate it could be as low as 10's of microamps and would hopefully not be more than say 5 mA.

A load of 5 mA will take 200 hours or about 8 days to drain 1 Ah from the battery. That's about 2 to 5% of a typical car batteries maximum capacity. So even if you left that connected for a year it would probably take not more than about 1/2 of the battery's capacity. as you start a car far more frequently than that it should not be a problem.

A low quality adaptor may use a zener diode dropper - I've seen it done. This can draw no current at all when there is no load, but the output varies badly with load and it wastes more energy that it outputs.

Most adapters will be active switching regulator types.

You can easily test the quiescent current draw.
Using a 12V power supply or a car battery and a multimeter with mA ranges.
Set meter to low current range. (You may want to start on a higher current range to protect the meter from violence or stupidity. Operate adapter with no load powered by 12V and with meter is series with the battery leads so you can measure current.

Does your car switch the power to the lighter socket?

I know in my car, when I turn off the ignition, it disconnects power from the lighter socket.

If your car does as well, it would be fine to leave in for an indefinite period of time.

It's simple enough to test, simply plug your device into the USB charger, and see if it stops charging when you turn off the ignition.

Yes, it draws some current when plugged in. Typically <5 mA.