gitignore directory exception not working

It's usually simplest to put just a .gitignore at the level where it starts to matter. (This also helps if you ever split a repository or move directories around.) In this case you need to ignore everything except catalog and private in the public/media folder so in public/media/.gitignore put:

/*
!/catalog/
!/private/

and in public/media/catalog/.gitignore put:

/*
!/category/

It's important (and the reason that your rules are not working) not to ignore the public/media/catalog directory itself, as otherwise everything in it will be ignored, even if you didn't want to ignore a specific part of its contents.

Of course, you can combine this into a single ignore at the public/media level if you like:

/*
!/catalog/
!/private/
/catalog/*
!/catalog/category/

Solution with a single .gitignore file in the repo root directory

You tried to use one .gitignore file at the repository's root level. Like you, I usually also centralize all ignores like this to keep things tidy. In this case it is difficult though, because "It is not possible to re-include a file if a parent directory of that file is excluded" [source].

My ugly solution for this restriction, which I also explore in my related answers here and here: re-include all parent directories along the path before re-including your desired file, while keeping other files in these parent directories out by re-excluding them.

With that, your .gitignore patterns would be:

public/media/**
!public/media/private
!public/media/catalog/

public/media/catalog/**
!public/media/catalog/category/