How was higher education done before the university system?

As with many things the history of higher education is not well known since accounts may be scarce or missing altogether. Early evidence indicate that schools existed in Egypt where, primarily, boys would learn to read and write etc. One has to remember that such skills were not for the masses. Religiously connected schools also emerged where religious texts were handled and copied. Theological and medical teaching was also done but very little else. evidence of mathematics have been found so there were also schools, perhaps aiming at architecture, astronomy , etc. relevant for the culture. None of these has a degree at the end but was likely based on apprenticeship and mastery evaluated by the teachers. Similar evidence for teaching exists in all older cultures and seem to focus on maintaining order in the social and cultural basis for society.

There seems to be a big step when considering Greek education which was far more comprehensive than the pragmatic education earlier. In the city states of ancient Greece specialized schools emerged where teaching circled around very specific topics such as the Hippocratic school of medicine on Cos. This specialization seems to have continued with the well known Greek philosophers/scientists such as Aristotle, Plato etc. and teaching progressed in directions envisioned by these founders. In other words, there was no single systematic way for schools to teach and operate.

In the Roman world subjects were ordered in groups that we can recognize today: I grammar rhetoric, dialectic; II geometry, arithmetic, astronomy, music; III medicine, architecture. This was how the liberal arts (defined as theoretical and intellectual activities by the Greeks) were seen. The subjects have of course survived to modern times although organisation has changed. So education became more organised but the education was not open and still served a purpose for maintaining government rule. Out of the post-Roman world came the first universities as stated in the question where education became even more organized and eventually including fixed degrees.

This answer is loosely based on the excellent book: The first universities by Olaf Pedersen, Cambridge UP, 1997. I strongly recommend it!

It depends on what you mean by higher education.

Degrees are highly inflated nowadays, now practically everyone and his dog has a PhD, or at least a BSc.

100 years ago passing final exams in high-school practically was the terminal degree for most administrative positions. Much fewer people went on universities, mostly engineering, doctors and such.

If you go back a couple of hundred years you realize that very few profession actually needed university degree, and universities were mostly scholarly centers for philosophy and religious studies. Most engineering didn't even exist 500 years ago, and what existed was taught by professional communities through apprenticeship, not by universities. So what you call higher education is defined very much by the post-industrial area.