Why can my custom-build iso not boot in EFI?

Thank you for flying xorriso.

The problem is in the image file which you let xorriso mark as EFI System Partition. It is supposed to be a FAT filesystem image which contains a binary file named /EFI/BOOT/BOOTX64.EFI (or .../BOOTIA32.EFI for 32 bit x86) plus possibly other files. Mount the file /boot/grub/efi.img from the Ubuntu ISO for learning about its content.

The FAT filesystems used by Ubuntu and others are probably produced by GRUB2 program grub-mkimage. ISOLINUX/SYSLINUX EFI software cannot be used because it bails out when seeing a CD-ROM device.

A quite convenient way to get a bootable ISO is program grub-mkrescue. If GRUB2 is configured for BIOS and EFI (e.g. by installing binary packages grub-pc, grub-efi-amd64, and grub-efi-ia32) then grub-mkrescue will produce an ISO which boots on BIOS and EFI of 32 bit and 64 bit x86 machines: El Torito boot images for BIOS and EFI, EFI image having BOOTX64.EFI and BOOTIA32.EFI, MBR for BIOS from hard disk, GPT for EFI from hard disk.