What are the main skills to be a performance consultant?

I think the most important skill is an analytical mind able to follow the evidence and not the "gut feel" and really break down and measure the issues.

After that, it would take some creative ways to understand measuring. When someone hires a performance consultant they are often stuck, which most likely means they have real users doing real things, which means Java won't be the only thing in the stack (as you noted databases, but there may be JavaScript and networking issues and who knows what else as well).

In addition to all of that, in terms of the laundry list, the big thing missing is garbage collection. Understanding how that works and how to tweak that is critical. Many cases will involve problems with stop-the-world garbage collection just stopping an important process for too long.

You really need to have an extensive toolkit of potential solutions, as each client will need different things. Some will need an object pool for their immutable objects, some will need to introduce immutable objects to reduce synchronization, some will need to introduce mutable objects to prevent excessive object creation, etc.. Performance is really a case-by-case thing, and you need to have a range of experience and knowledge to pull from to help with each case.

I think hardware is also important to understand (at least it's important in designing high performing databases). When do you need to throw more hardware at the issue and when do you need to fix the code and when do you need to do both.

As a consultant on these sorts of things you are going to need serious people/political skills. You will be interviewing and collecting data from the people who already tried to fix the performance problem and couldn't. Many of them will be unhappy a consultant was called in. You'll have to deal with their resistance. IF you are going to do this as a business of your own, you will also need good accounting knowledge (and a tax consultant) and sales skills. YOu will need speaking skills to presnt your service to potential customers.

Make sure not only to measure but to document what was tried and the difference in time. Keep your own records of such things in a database and pretty soon, you will have a way to see the most likely performance tunings to try based on hard data over many clients. Developing a knowledgebase program will help you immensely as time goes on.

I'd also invest in a set of books on database performance tuning (At least one for each of the major database backends as tuning is very database specific) and database design. I think you will be able to trace many, many performance problems to bad database design and lack of knowledge of how to write good SQL code that will perform well. Database performance tuning is way more extensive than knowing what indexes to create.

I think that depending upon the particular client and problem at hand, you'll need to have excellent skills in algorithmic analysis and optimization. Is your problem at the math-algorithmic level, or is it at the Java-algorithmic level?

You'll also need to be very familiar with the implementations of JVM you'll be working with. Knowing the language and API isn't good enough: you'll have to know in intimate detail which language constructs to use in certain situations, and this can vary depending upon the specific JVM you are using. A profiler won't always reveal JVM-specific performance issues. As others have noted, the same JVM can behave differently on disparate harware.