Questions about BRST formalism and BV formalism
I) On one hand, the Faddeev-Popov (FP) formalism assumes that
The gauge algebra is "irreducible", meaning that there are not higher levels of gauge-symmetries among the gauge generators. This is aka. gauge-for-gauge symmetry.
The gauge algebra closes off-shell.
If the gauge-fixing conditions do not depend on ghosts, then the FP action is quadratic in the ghosts $c$ & $\bar{c}$.
II) On the other hand, the Batalin-Vilkovisky (BV) formalism [1] also works for reducible & open gauge algebras:
Reducible gauge algebra typically leads to multiple FP determinants. BF theories & abelian $p$-form theories are typical examples.
The hallmark of an open gauge algebra is a term in the BV action of the form $$\int\!d^dx~\varphi^{\ast}_i\varphi^{\ast}_j ~E^{ji}_{ba}(\varphi)~c^ac^b,$$ which in its gauge-fixed form becomes quartic in the ghosts $c$ & $\bar{c}$. SUGRA, Green-Schwarz superstring & the superparticle are examples of an open gauge algebra [3].
References:
I.A. Batalin & G.A. Vilkovisky, Gauge Algebra and Quantization, Phys. Lett. B 102 (1981) 27–31.
M. Henneaux & C. Teitelboim, Quantization of Gauge Systems, 1994.
M. Henneaux, Lectures on the antifield-BRST formalism for gauge theories, Nucl. Phys. B Proc. Suppl. 18 (1990) 47.
J. Gomis, J. Paris & S. Samuel, Antibracket, Antifields and Gauge-Theory Quantization, arXiv:hep-th/9412228.
Question 1:
Why it says "quartic terms are not allowed" in Faddeev-Popov formalism?
Well it's not that they are not allowed, but rather that the FP procedure will not generate such terms, and furthermore that there exist theories where such terms are warranted (see below). Your observation is correct.
Question 2:
when working in a theory with open symmetries, we might end up with quartic ghost terms in the gauge-fixed Lagrangian?
Yes, there exist such examples. Most famously closed bosonic string field theory is of this form. I recommend the freely available review by Gomis et al. which discusses string field theory in this context. There is also a canonical textbook
"Quantization of Gauge Systems" by Marc Henneaux and Claudio Teitelboim.
Both of these should have other examples as well.