Why is the brightness of my red LED slowly decreasing?

Just because the forward voltage is in spec it doesn't automatically mean the forward current is too. 2.6V across the LED leaves 2.4V across the resistor/MOSFET which could mean 96mA through the LED and hence 250mW dissipated, which is well above the absolute maximum of 150mW.

I think you're killing your red LED.

The forward voltage in the datasheet simply gives the value you would expect to see when passing a given current through it, I presume this is the 50mA it gives in the column header but it will say somewhere.

Your problem appears to be the variation in the RDS(on) for the device you are using. The datasheet for the BSS84 shows that the normalized RDS(on) will increase as the FET gets warmer.

The RDS(on) for the device could be quite high (10 Ohms) compared to your series resistor.
Since in your comments you say the voltage across the 24.9 Ohm is 1.946 V ....added to the Vf of the LED this leaves about 494 mV across the FET (depending how accurate your 5 V supply is), so it's not an ideal device (RDS(on) is about 6.3 Ohms) for your application.