What actually happens when we set direction on GPIOs in controllers ?

NXP doesn't seem to provide this, but STM32F4 user manual (page 270) shows a block diagram of a typical GPIO pin.

As you can see, the pins are connected to all blocks: Analog/Digital Input as well as the Output sections. The control registers enable/disable and configure these blocks so they don't interfere with each other. There are also separate enable signals for internal pull-ups/pull-downs.

The exact details of how this is implemented in silicon, obviously, differs between manufacturers, architectures and processes.

I couldn't find the specific internal schematic for NXP's part in the User Manual, so I will explain using the one found in a ATMEL ATMega328, but they should look similar.

The picture shows the internal schematic for a single pin. The highlight part is the port direction pin part. When The port is set to output (via the databus, where one can set each pin individually as Input or Output; and via the Internal control signal WDx, which "strobes" the data on the data bus to the flip-flop), the buffer is activated and begins to drive the pin to the logic levels (voltages) on the output pin*.

When the pin is set as input, the enable pin on that buffer is configured so the buffer is a High Impedance output, thus not driving the pin to any specific voltage level. That becomes a task for the circuitry connected to that pin in the circuit outside the microcontroller. Note, in the Atmel's case, that reading the logic state of a pin (bottom part of schematic) always reads the actual state of that pin, doesn't matter if it is a input or output pin. Configuring a pin as input also enables the user to enable the pull-up resistor inside the microcontroller.

*: I am ignoring open colector and other types of pins because I think it is not relevant to this question. The workings should be similar, though.

How setting/clearing the pin makes it as output/input. I think it is due to pull-up and pull-down transistors but I couldn't conclude a specific reason.

Note that you set the direction register, not the pin state itself. They are different things. Setting/clearing the pins tells the hardware which voltage is to be put on the output of the output buffer (when it is enabled by the enable pin), while setting/clearing the direction of the pin tells the micro if the output buffer will be enabled or disabled at all.

To add to the answers above (which are good), do note that the IODIR register (or whatever it's called on your processor) only controls whether the output driver is enabled/disabled. The output driver and the input driver are independent.

Where this gets interesting is when you read back the state of the pin. Usually you find that writing to the I/O register sets the output state, but reading from the I/O register reads the input state. Aren't the two the same? Actually not always, if there's something else also connected to that pin which might drive the pin more strongly. In particular, if the output is open-collector/open-drain then the output driver goes high-impedance when you set the output to 1, so that pin's state is not controlled by the processor.

On most processors, this means that if you want to keep track of the state of the output driver then you need a variable storing that value, because the processor does not give you a way of reading it back.

Also note that the pull-up/down is completely independent of whether the output is driven. Internal pull-up/down resistors are often convenient when you don't care exactly what the resistance is, because it ensures those pins default to known level. Leaving inputs unconnected is almost always bad practise, because it allows for build-up of static charge which can damage the device. Pull-up/down resistors ensure this doesn't happen. They can also be used for applications where you would usually have an pull-up resistor on an input, such as a pushbutton driving the input to 0V; or on applications where you would usually have a pull-up resistor on an output, such as open-collector/open-drain outputs. This reduces the component count in your schematic. If you need a specific value of resistor though (for instance if you add a capacitor for filtering) then an internal pull-up is typically not accurate enough and you would need an external resistor.