Why do we need to convert from SFDX to MDAPI format in packaging org
The dx format isn't understood by the mdapi, and non-scratch orgs do not support the dx format because they're missing key data tables under the hood. The net result is that conversion to and from dx format will be necessary until Next Generation Packaging reaches parity with Classic Packaging.
In the future, your code repository will be broken up in to packages that can be independently installed and upgraded. This will ultimately minimize deployment times and allow Salesforce to automatically "diff" upgrades for you in the same manner that happens today with managed packages, but without the namespaces.
Next Generation Packaging will even be intelligent enough to allow you to automatically delete classes, components, pages, and so on that are no longer needed, will validate that your code base is in a consistent state (because packages must be able to validate on their own or as a dependency), and are said to be supported even within managed packages, which will make migrating code and features into and out of managed packages a much simpler experience.
In other words, it's something that we have to do today in order to migrate towards a better experience, but this will likely not be needed in the coming releases as we're given newer and even better tools to deploy code and configuration from any particular organization to any other organization.
sfdx force:source:push combines the split up metadata in the DX format back into the standard metadata format before deploying to a scratch org, and it uses the normal SOAP metadata API to deploy it.
But the reason that you can't use force:source:push to deploy to a non-scratch org is that before the deploy, sfdx queries the scratch org for SourceMember records to determine what metadata to send to the org. The SourceMember object is used to track a per-org revision number for each metadata object. This doesn't work for a non-scratch org because the SourceMember object doesn't exist in non-scratch orgs.
If you want to overwrite the metadata in a non-scratch org, you can use sfdx force:source:push --forceoverwrite, which will skip the step of querying SourceMember, and simply bundle everything up and deploy it to your org.
Salesforce DX is best thought of as a set of projects and features that combined will lead to a better developer experience.
One of these features is a scratch org. The disposable org environment that a developer would use to build new features.
One feature that scratch orgs have been given is a source change tracking feature, such that every time a change is made, there is a metadata object that tracks the change.
The changes are also tracked locally in your DX project (poke around the .sfdx hidden directory and you will soon find where.)
The two together make the sfdx force:source:??? commands possible.
The architecture is spoken of in this talk from Dreamforce 16 although it went through some transformations since then coming up to the core DX features going GA.
So long story short, only scratch orgs have the tracking enabled. And thus your non-scratch org needs to speak the API that it knows...the Metadata API.