If all the Structure in our visible Universe originated from Quantum Fluctuations, why isn't there as much Anti-Matter as Matter?

The matter-antimatter asymmetry requires the three Sakharov conditions to be satisfied. I'll summarise that link's explanation. Unfortunately, your question isn't completely solved.

The first condition is that some interactions don't conserve baryon number (I.e. baryons minus antibaryons, baryons being three-quark hadrons such as protons and neutrons). How much asymmetry exists is an open question, as it depends on unverified conditions such as supersymmetry and grand unification.

The second condition, CP violation, has been observed since 1964. However, it's still not known how there's enough of it to explain the observed asymmetry. As with the previous condition, the Standard Model doesn't explain all of the effect, but hopefully new physics soon will.

The third condition is out-of-thermal-equilibrium thermal conditions. Early inflation would ensure this.

Adding to J.G. answer, it should be noted that Baryon number violation is possible within the Standard Model. This has been shown by t'Hooft in the 70's and this involves non-perturbative effects. Necessarily so since the Electroweak Lagrangian has a global B symmetry and therefore this carries to any Feynman diagram. But this does not prevent the dynamic of the model to break B. Actually t'Hooft processes break B+L but not B-L, where L is the lepton number. The issue is then quantitative: do we get enough of B-violation? As J.G. explained, this must happen out of thermal equilibrium, so there is the added difficulty of correctly computing the dynamic. Still very much an open question.