What kind of design freedom can you permit yourself in academic writing?
I read papers, I don't hang them on my wall. That is the main objective, and your design should subdue to it. I enjoy a well crafted book, but I get extremely annoyed when some unnecessary decoration gets in the way of usage.
In your case, the font of the titles requires a mental effort from my part to read them, which I find unacceptable, and thus, annoying. When reading a paper, I usually skim through the sections to get a broad picture of it; with Pacifico, I need to waste neurons in deciphering it while constructing my mental idea. I also find that the numbers in the text stand out a lot, probably more than they should, as they are not information I would like to get while skimming.
On the other hand, I have encountered a few unique and very nicely crafted documents, and those made a more long lasting impression. If not for anything else, I remember the general image of the article.
Bottomline is, if your design is good, it will be welcomed; but if it has flaws, some picky people like me* may get displeased and cause a worse first impression. A standard template is a safer option; but more difficult to stand out.
*To give you an idea of how much, I find several citation styles to be rather annoying because they are clearly inferior to others. For example, anything without DOI in the modern times.
As March Ho mentioned in the comments, one important factor is legibility. This is true both at the level of fonts and more broadly in design/structure: if your most important goal is to communicate your ideas, then it's worth optimizing for ease of reading and comprehension. Unfamiliarity is itself an obstacle, so even if your design would be superior if widely adopted, it might prove inferior in practice.
Another issue is demonstrating membership in the community of scholars. Choosing an unconventional design amounts to announcing "I am an outsider," and although being an outsider is not a bad thing in itself, it can come across to readers as a terrible sign. For example, when I see mathematics papers with highly nonstandard formatting, they are almost always crackpot papers of no value whatsoever. That's not company you should voluntarily join. I wouldn't consciously ignore a paper just because of the formatting, but I can't help approaching it with a strong prejudice. Unless your papers are really exceptional, unconventional design is likely to decrease your readership.
The fundamental issue here is whether you are writing for yourself or others, which is a common tension within academia. You can take the position that you are creating a work of art and aren't willing to compromise on your vision even if it will hurt the work's reception, or you can try to do what's necessary to increase your work's impact on the scholarly community. Different authors end up balancing these concerns in different ways.
If you are self-publishing, then you are free to format your work however you please. However, as the others have already pointed out, the harder your work is to read, the less it's going to be read, and the more trouble you will have getting others to take your work seriously.
As you mentioned in your question, you yourself find the font "borderline legible." That's already a huge strike against its use: why would anyone else want to read it if you, the designer, find it only marginally legible?
As someone who greatly appreciates good page design, I have to admit that I find it too "cute" for its own good. There are many fonts that could serve your design purposes of being "original" without sacrificing legibility. Pacifico is not one of them. Also, the bold numbering in the text is probably not good unless those are going to be hyperlinks.