How to make 1 bit permanent memory circuit?

The original electronic nonvolatile memory is based on ferrite cores. While it's relatively easy to magnetize such a core in one direction or the other to store a one or a zero, it takes some fairly sophisticated circuitry to read it back reliably.

Modern nonvolatile chips rely on charge storage, but in order to make this work, you need to be able to create a capacitor that has essentially zero leakage, and a way to read out that charge. This can only be done in the context of microelectronics, where the capacitor is a tiny piece of metal (the "floating gate") that's completely encased in glass (silicon dioxide), and is read out by means of its influence on a nearby transistor.

Another choice is ferrorelectric RAM (FRAM), which uses a special dielectric material that has two distinct, stable polarization states. Again, this only works in microelectronics.

Therefore, you need to pick some other physical phenomenon to store your bit of information. One obvious choice is the latching relay, which stores information in the physical position of its armature, which is held in either of two stable positions by a permanent magnet or a spring. The position can be changed by applying a relatively short pulse of current, and the readout is accomplished by attaching electrical contacts to the armature.

Make a circuit that flicks a mechanical switch eg. useless box. The circuit would need to be powered up to change/read state but it would keep it in between.

A simple solution would be a micro controller such as a PIC12F635 which is available in an 8-pin DIP or smaller, and has a built-in clock and brown-out reset circuit (the latter is important to maintain the integrity of the EEPROM nonvolatile storage).

The code required is not much, a good starter project.

The only external parts required would be a bypass capacitor and a current-limiting resistor for the LED.

The very simplest solution is probably a 2-coil latching signal relay.