Why does the Sun feel hotter through a window?
You were right with your original though. This is an effect purely due to a lack of convective cooling. When you are inside, there is normally very little to no airflow, which means when you stand in the sunlight and heat up due to it, the air around you will be warmed by you but isn't moving enough to be replaced with cool air and carry the heat away. Don't misunderstand, the air is moving, which is why you won't continue to build up more and more heat, your temperature will plateau. It's just that the air isn't moving enough to prevent an increase in skin temperature. When you step out of the direct sunlight, it feels cool again because now you are not being heated as much and what convective flow exists is enough to cool you to normal levels. Additionally, there is a component that is attributed to the air being directly heated by the sunlight as well as re-radiation from heated objects (like a table or chair), but most is due to a lack of convective cooling.
As for your anecdotal evidence, I cannot comment on why it might have felt just as hot with a strong fan on you, but I can try to convince you that my answer is true. If you go outside one morning when there is little to no wind/airflow and stand in the sunlight, it will feel just as hot. Even in the winter, the sunlight will feel hot, however the air around you will have slightly more cooling power so it may still feel cold.