Can I use a slim version of my header to be included with the library?

This is a One Definition Rule violation. The moment you deviate by a single token.

[basic.def.odr]/6

There can be more than one definition of a class type, [...] in a program provided that each definition appears in a different translation unit, and provided the definitions satisfy the following requirements. Given such an entity named D defined in more than one translation unit, then

• each definition of D shall consist of the same sequence of tokens; and

Your program may easily break if you violate the ODR like that. And your build system isn't at all obligated to even warn you about it.

You cannot define a class twice. It breaks the One Definition Rule (ODT). MyLibrary does that, unfortunately.

they're never passed to the compiler

They will. Members of a class must be known at compile time, so that the compiler can determine the class's size.

Header are just declarations anyway right? they're never passed to the compiler. And I've declared what the user will be using.

No. Headers are part of source code and are compiled together with source files. They contain the information necessary for a compiler to understand how to work with code (in your case, with class MyLibrary).

As an example, you want library users to be able to create objects of class MyLibrary, so you export the constructor. However, this is not sufficient: the compiler needs to know the size of the object to be created, which is impossible unless you specify all the fields.

In practice, deciding what to expose to library users and what to hide as implementation details is a hard question, which requires detailed inspection of the library usage and semantics. If you really want to hide the class internals as implementation detail, here are some common options:

• The pimpl idiom is a common solution. It enables you to work with the class as it is usually done, but the implementation details are nicely hidden.
• Extract the interface into an abstract class with virtual functions, and use pointers (preferably smart pointers) to work with the objects.