How to remove a Package from my DevHub org?

It cannot currently be done by you. Perhaps Salesforce support could do it if you contacted them.

You can uninstall the package from various orgs as suggested by the previous answer. That does not eliminate the package from your package list.

sfdx force:package:list

... will still show the package even if it is uninstalled from all orgs.

I have created an idea to add this capability to the SFDX cli on the idea exchange. Go vote for it. https://success.salesforce.com/ideaView?id=0873A000000lMYmQAM