How to set device-specific mouse settings in Wayland under Libinput (Debian Gnome)?

Wayland expects all mice motion to have been normalised, so there is only one global changeable configuration. You may have to edit your hwdb entry for one of your devices to correct it if it is wrong, or just make it fit in with your preferences.

Alternatively, you may be able to use libevdev-tweak-device from the package libevdev-tools (or libevdev-utils). It says it can alter the definition of an evdev device dynamically. You would do something like

sudo libevdev-tweak-device --abs ABS_X --res 99 /dev/input/event99
sudo libevdev-tweak-device --abs ABS_Y --res 99 /dev/input/event99

where you need to replace the 99 by the resolution you want, and event99 by the input device. You can find the input device from, eg:

$ ls -l /dev/input/by-id/
lrwxrwxrwx ... usb-Logitech_USB_Optical_Mouse-event-mouse -> ../event5

To find the current resolution try sudo evemu-describe from the evemu-tools package, or use mouse-dpi-tool to try to choose a good value.