Why is "P vs. NP" necessarily relevant?
Whether $P = NP$ is in the end only a single bit of information; moreover, people expect that they are not equal. The same can be said of any of the Clay Prize problems. So what's the point? Since it is a very common assumption that $P \ne NP$, in many very important papers in computer science and mathematics, people expect that the proof will also be very interesting and that there will be a lot to learn from it. After all, every time any version of the Poincaré conjecture has been proven (by Smale, Freedman, Perelman, or even in 2D by Riemann et al) or disproven (by Milnor in the smooth case in high dimensions), the proof has led to all kinds of interesting other mathematics. Every once in a blue moon, mathematics is a little bit cruel and a long awaited proof is not as interesting as people expected. Possibly the first proof of the transcendence of $\pi$ was an example of that. But that's not what usually happens.
In the specific case of the $P$ vs $NP$ problem, it is easy to see some of the related questions that would hopefully be affected. You can go down the list of NP-hard problems and ask, more precisely, how hard each one is. The reductions among these problems tell you that they are roughly as difficult as each other, but that's about it. Moreover, there are many other conjectured separations between complexity classes that look similar to $P$ vs $NP$. $P$ vs $PSPACE$ looks similar, but these two should be much farther apart. $P$ vs $BQP$ is a particularly juicy question, since it isn't even obvious to everyone (although I think it's a safe bet) that it's worthwhile to build a quantum computer.
Even better, there are table-turning results. Given an assumption that's not so much stronger than $P \ne NP$, it follows that $P = BPP$, i.e., every randomized polynomial time algorithm can be derandomized in polynomial time.
Yes, conceivably mathematics is cruel and it will turn out that 3-SAT can be solved in time $O(n^G)$, where $G$ is Graham's number. But why should it be? Let's say that our aesthetic and our intuition for expecting a very influential proof is good 90% of the time. Recently our batting average has been something like that. That is reason enough to explore the problem.
The $P \ne NP$ problem is the best way we know to formulate the belief (which was expressed even before the problem was formally stated) that certain specific algorithmic problems (such as finding a Hamiltonian cycle in a graph) requires exponential number of steps as a function of their description. The formulation is based on the important notion of a nondeterministic algorithm. The conjecture that P is not equal to NP is the basis of a mathematical theory of complexity which is remarkably beautiful. It is likely that a proof for the conjecture will lead to better understanding of the limitations of computation which will go beyond the conjecture itself. (Unfortunately we are very far from such a proof.) A counterexample (which is not expected) may lead to a major change of our reality and not just our understanding of it. (I suppose this is one of the main reasons in believing the conjecture.) Your concern is that maybe, because of the asymptotic nature of the conjecture, we can question its real relevance. Namely problems we regard as intractable may still be even if P=NP because of huge constants in the polynomial involved, or perhaps even if NP not equal P there can be efficient algorithm to relevant cases of intractable problems. These are serious concerns that are often raised and sometimes there are efforts to study them scientifically.
Overall, it looks that these asymptotic concerns are secondary compared to the problem itself. Moreover, there are many examples that the asymptotic behavior and practical behavior are similar beyond what the theory dictates (but a few examples also in the other direction). There are other related concerns regarding the relevance for the $P \ne NP$ problem. Is the worst case analysis relevant to practical problems? Is the hardness apply when we are interested in approximate solution rather than in exact solution? Both these questions (like the asymptotic issue) can be regarded as secondary in importance compared to the $NP \ne P$ problem itself but otherwise very important. Indeed both hardness of approximation and average case analysis are very central research topics.
(A side remark, there is something unnatural about the way Graham's numbers are used in the question. You are dealing with a decision problem where the answer is always yes when n is sufficiently large. And there is a simple polynomial time algorithm as a function of n to decide the answer for a every given n. So the relevance of this particular example (which is interesting) to the question asked about NP complete problems (which is also interesting) is not so strong.)
Finally, regarding relevance. Andreas raised the interesting possibility that NP=P but the constant involved are so huge that the polynomial algorithm for solving NP complete problems is utterly not practical. A related interesting possibility is that there even exists a practical polynomial time algorithm for NP complete problems but that finding this algorithm itself is computationally intractable. As I said, both these possibilities are considered unplausible. Even if true they will not harm the relevance of the NP=P problem in making the central distinction between intractable and tractable problems. In order to make these concerns interesting one should come up with a theoretical framework to study them, and then study them fruitfully. (As I mentioned above, this was done for related concerns regarding approximation and regarding average case behavior.)
It is clearly relevant whether there exist feasible algorithms for NP-complete problems, where I am using the term "feasible algorithm" in the informal sense of an algorithm that we could actually run quickly in practice on problems of practical interest. Call this the Main Question.
The relevance of the $P = NP$ question lies in its perceived relevance to the Main Question. Clearly, to prove that $P = NP$ is a relevant question, it would suffice to prove the lemma that $P$ is precisely the class of problems with feasible algorithms.
Of course, the lemma is obviously false, as you point out and as others have pointed out (e.g., Yuri Gurevich). Still, there is enough overlap between $P$ and the class of problems with feasible solutions that settling the $P=NP$ question would be a major partial step towards answering the Main Question. It is in this sense that $P=NP$ is a relevant question.