USB device enumeration fails: "device descriptor read/64, error -32"

This error can be caused by clocking the on-chip USB device at the wrong frequency. Check your clock tree configuration. The frequency should be 48MHz. If it's slightly off, it's possible that enumeration would sometimes be successful and sometimes fail. If it's way off, enumeration will always fail. Various errors are possible including the ones you listed.

AFAIK status -32 means "Broken pipe" (EPIPE). It means that there are problems with usb communication (protocol). For example usb-device doesn't answer correctly for usb-request, and sending some data that are not expected by host or sends not enough data. There may be also other reasons.

The first message tells that there are problems with "get device descriptor" and "set address" requests from host. Those are basic requests sending by host at the very beginning of enumeration process. You can't go further if those requests cannot be succesfully serviced by usb-device.