Why do we not observe a greater Casimir force than we do?
The answer by G.Smith is correct and concise. For whatever it's worth, I'll give a longer answer.
Sometimes authors use terms like zero point energy or vacuum energy for marketing purposes, because it sounds exotic. But sometimes authors use those terms for a different reason: they're describing a shortcut for doing what would otherwise be a more difficult calculation.
Calculations of the Casimir effect typically use a shortcut in which material plates (which would be made of some complicated arrangement of electrons and nuclei) are replaced with idealized boundary conditions on space itself. In that shortcut, the "force" between the boundaries of space is defined in terms of $dE/dx$, where $E$ is the energy of the ground state (with the given boundary conditions) and $x$ is the distance between the boundaries. This is a standard shortcut for calculating the force between two nearly-static objects: calculate the lowest-energy configuration as a function of the distance between them, and then take the derivative of that lowest energy with respect to the distance. When we idealize the material plates as boundaries of space, the lowest-energy configuration is called the vacuum, hence the vacuum energy language.
The important point is that this is only a shortcut for the calculation that we wish we could do, namely one that explicitly includes the molecules that make up material plates, with all of the complicated time-dependent interactions between those molecules. The only known long-range interactions are the electromagnetic interaction and the gravitational interaction, and gravity is extremely weak, so that leaves electromagnetism.
What about all of the other quantum fields in the standard model(s)? Why don't they also contribute to the Casimir effect? Well, they would if we really were dealing with the force between two movable boundaries of space itself, because then the same boundary conditions would apply to all of the fields. But again, the boundaries-of-space thing is just an idealization of plates made of matter, so the only relevant fields are the ones that mediate macroscopic interactions between matter.
Okay, but isn't the usual formula for the Casimir effect independent of the strength of the interaction? Not really. That's another artifact of the idealization. The paper https://arxiv.org/abs/hep-th/0503158 says it like this:
The Casimir force (per unit area) between parallel plates... the standard result [which I called the shortcut], which appears to be independent of [the fine structure constant] $\alpha$, corresponds to the $\alpha\to\infty$ limit. ... The Casimir force is simply the (relativistic, retarded) van der Waals force between the metal plates.
For perspective, the electromagnetic Casimir effect typically refers to an attractive interaction between closely-spaced plates, and van der Waals force typically refers to an attractive interaction between neutral molecules, but they're basically the same thing: interactions between objects, mediated by the (quantum) electromagnetic field. The related post Van der Waals and Casimir forces emphasizes the same point.
Metal plates impose a boundary condition on the electromagnetic field, because metal is made of charged particles which interact with an electromagnetic field. But those metal plates do not impose a boundary condition on the Higgs field, which extends through conductors.