Why Powershell's New-WebBinding commandlet creates incorrect HostHeader?

Looking at the decompiled code of the cmdlet, looks like it adding the IPAddress and Port information in the binding and there is no workaround to it.

Relevant sections from the code:

private string ipAddress = "*";
...
builder.Append(this.ipAddress);
...
builder.Append(":" + this.sitePort.ToString(CultureInfo.InvariantCulture) + ":");

But you can do what the cmdlet actually does ( below code from cmdlet):

new-itemproperty -path "IIS:\sites\test" -name bindings -value @{protocol="net.msmq"; bindingInformation="localhost"}

Give this a try:

New-ItemProperty "IIS:\sites\NameOfYourSite" -name bindings -value @{protocol="net.msmq";bindingInformation="localhost"}