Does entropy depend on the observer?

E.T. Jaynes agrees with you, and luckily he is a good guy to have on your side:

From this we see that entropy is an anthropomorphic concept, not only in the well-known statistical sense that it measures the extent of human ignorance as to the microstate. Even at the purely phenomenological level, entropy is an anthropomorphic concept. For it is a property, not of the physical system, but of the particular experiments you or I choose to perform on it.

This is a quote from his short article ``Gibbs vs Boltzmann Entropies'' (1965), which is a great article on the concept of entropy in general, but for this discussion in specific you can turn to section VI. The "Anthropomorphic" Nature of Entropy. I will not try to paraphrase him here, because I believe he already described himself there as succinctly and clearly as possible. (Note it's only one page).

I was trying to find another article of him, but I couldn't trace it at the moment. [EDIT: thanks to Nathaniel for finding it]. There he gave a nice example which I can try to paraphrase here:

Imagine having a box which is partitioned in two equally large sections. Suppose each half has the same number of balls, and they all look a dull grey to you, all bouncing around at the same velocity. If you now remove the partition, you don't see much happen actually. Indeed: if you re-insert the partition, it pretty much looks like the same system you started with. You would say: there has been no entropy increase.

However, imagine it turns out you were color blind, and a friend of yours could actually see that in the original situation, the left half of the box had only blue balls, and the right half only red balls. Upon removing the partition, he would see the colors mix irreversibly. Upon re-inserting the partition, the system certainly is not back to its original configuration. He would say the entropy has increased. (Indeed, he would count a $\log 2$ for every ball.)

Who is right? Did entropy increase or not? Both are right. As Jaynes nicely argues in the above reference, entropy is not a mechanical property, it is only a thermodynamic property. And a given mechanical system can have many different thermodynamic descriptions. These depend on what one can --or chooses to-- measure. Indeed: if you live in a universe where there are no people and/or machines that can distinguish red from blue, there would really be no sense in saying the entropy has increased in the above process. Moreover, suppose you were color blind, arrive at the conclusion that the entropy did not increase, and then someone came along with a machine that was able to tell apart red and blue, then this person could extract work from the initial configuration, which you thought had maximal entropy, and hence you would conclude that this machine can extract work from a maximal entropy system, violating the second law. The conclusion would just be that your assumption was wrong: in your calculation of the entropy, you presumed that whatever you did you could not tell apart red and blue on a macroscopic level. This machine then violated your assumption. Hence using the 'correct' entropy is a matter of context, and it depends on what kind of operations you can perform. There is nothing problematic with this. In fact, it is the only consistent approach.