file(1) and magic(5) : describing other formats
You can use the -m option to specify an alternate list of magic files, and if you include your own before the compiled magic file (/usr/share/file/magic.mgc on my system) in that list, those patterns will be tested before the "global" ones. You can create a function, or an alias, to transparently always transparently use that option by just issuing the file command.
The language used in magic file is quite powerful, so there is seldom a need to revert to custom C coding. The only time I felt inclined to do so was in the 90's when matching HTML and XML files was difficult because there was no way (at that time) to have the flexible casing and offset matching necessary to be able to parse <HTML and < Html and < html with one pattern. I implemented that in C as modifier to the 'string' pattern, allowing the ignoring of case and compacting of (optional) blanks.
These changes in C required adaptation of the magic files as well. And unless the file source code has significantly changed since then, you will always need to modify (or provide extra) rules in magic files that match those C code changes. So you might as well start out trying to do it with changes to the magic files only, and fall back to changing the C code if that really doesn't work out.