Are all of the C++ headers, a class?

Header files doesn't on their own introduce a new scope or namespace, or indeed any class.

Header files included with the preprocessor #include directive are basically copy-pasted as-is into the translation unit for the later stages of the compiler to parse.

In short, there's really no difference between source and header files. Both can contain function or class declarations or definitions.

---

A very simplified example.

Lets say you have the header file a.h which contains a single function declaration:

void a_function();

Then you have a source file which include the header file:

#include "a.h"

int main()
{
    a_function();
}

After preprocessing the translation unit would look something like this:

void a_function();

int main()
{
    a_function();
}

The function a_function isn't part of a class, it's a global function.

A header is a file, a class is a type introduced by class or struct. While people often put each class in a dedicated header file for organization, there is no specific relationship between them. For example, <cmath> is just a bag of global (or, more precisely, namespace-scope) math-related functions, no "math" class needed.

We can use headers to include our classes. I mean one of their use is for including classes, so what are the other ones?

We use headers to include various declarations (nitpick shield: definitions as well), not just class declarations. These can be classes, typedefs, functions, variables, named constants, and so on.

In addition to all the other answers you have gotten, I would like to show you a few examples that should drive home that #include ... is just a copy-paste mechanism.

I have three files:

CodeBody.h

std::cout << a << std::endl;

Numbers.h

1, 2, 3, 4, 5, 6, 7, 8

and main.cpp

#include <iostream>
#include <vector>

int main()
{
    std::vector<int> bla = {
#include "Numbers.h"
    };

    for (auto a : bla) {
#include "CodeBody.h"
    }

    return 0;
}

This compiles and outputs:

1
2
3
4
5
6
7
8

Now, I would say that pasting code in like I did here with CodeBody.h is somewhat questionable, and you should not do that. But using an include file to populate an array or other data structure is used reasonably often, for example in digital signal processing where you use a dedicated program to calculate the coefficients for a filter, save them to a file, and then import them into your program with an include.