How can I control 8 LEDs using only 5 pins of a microcontroller?

8 leds on/off have 2^8 = 256 possible states

5 control pins on/off have 2^5 = 32 possible states

Since 32 < 256, and you can't use shift registers or i/o expanders, it would be impossible...

But if the 5 control pins are the pins of a MCU, you need to use only one of them for controling the 4 * 2 multiplexing process. One of the pins can be 0V or +Vcc, each of these two states enables a different set of 4 LEDs attached to the other 4 pins.

"he gave us a hint : use a wire connection combination if a problem appears try solving it in your code"

As this is a theoretical question in a curse, I think that this would be a possible correct answer. This is a kind of 2*4 multiplexing, using only 5 pins.

If you need to control a higher number of LED's, charlieplexing would allow to control untill 20 LED's.

^{simulate this circuit – Schematic created using CircuitLab}

Use addressable LEDs like WS2812B or similar.

You’d need only 1 pin (plus power and ground) to control hundreds of them. They are available individually or as part of an LED strip (which can usually be cut into smaller strips or individual LEDs).

There are these things that are known as "movies", where an image is switched on and off rapidly, giving the feeble human mind the illusion that the image is continuous.

You can accomplish something similar by having a multiplexer (3 selector signals and a strobe signal) turn the 8 LEDs on (or not) one at a time.

And if you really want to get fancy you can add a capacitor to each LED to keep it lit for a period of time after it's signal is turned off.