Is "dark clothes for winter, light for summer" relevant?
This article has some relevant results based on a study of bird plumage (it also happens to be cited in the abstract of the Nature paper mentioned in one of the other answers), and is summarized in simpler terms here.
I'll attempt to summarize the summary.
Black and fluffy/loose fitting clothing is best if it is hot out and there is any ($>3 \mathrm{m}/\mathrm{s}$) wind. The black clothing absorbs both solar radiation and radiation from the body. The air in the immediate vicinity is heated, then efficiently transported away by the wind. This is slightly better than white fluffy/loose fitting clothing, which reflects more sunlight and radiation from the body. The emission from the body is reflected, so it cannot heat the air near the clothing as efficiently and have a chance to be transported away.
Tight black clothing is a terrible idea if trying to stay cool, regardless of windspeed.
If there is no wind ($<3 \mathrm{m}/\mathrm{s}$), white clothing is better since the most important thing in these conditions is to reflect as much incoming sunlight as possible.
I also have another possibility to think about. My recollection regarding loose fitting black robes in the desert is that - given a garment that is open at the bottom (robe) and top (not too tight fitting) - heating the air inside is actually advantageous to keeping cool since this drives a convection flow upward through the garment. This airflow makes cooling via sweating efficient, enough that the person wearing the garment doesn't feel as hot. Unfortunately I can't find any experimental results to validate this picture, but it seems more or less in line with the results above, at least in as much as airflow seems to be key to answering the question.