How does the xdg-open command know which application to use to open a file?

xdg-open is a desktop-independent tool for configuring the default applications of a user. Many applications invoke the xdg-open command internally. Inside a desktop environment (like GNOME, KDE, or Xfce), xdg-open simply passes the arguments to those desktop environment's file-opener application (eg. gvfs-open, kde-open, or exo-open). which means that the associations are left up to the desktop environment. When no desktop environment is detected (for example when one runs a standalone window manager like eg. Openbox), xdg-open will use its own configuration files.

from archwiki

specific to your question, you could try this to set the default application associated with the png file:

xdg-mime default <ristretto.desktop> image/png

you need find out what exactly the desktop file name of Ristretto. afterwards, you could check it with this:

xdg-mime query default image/png

@Ivan That happens probably, because you didn't provide the right filetype when asking for the default application. Important is how xdg calls that specific file-type. This can easily be found out by running

xdg-mime query filetype example.png

As an example, if I do this with a specific png-file on my computer it returns image/x-apple-ios-png. Once you have the actual file type figured out, you can run

xdg-mime query default <new filetype>

and get the application that is used by xdg for this specific file type. In case no application is defined for this filetype, xdg-open checks if the file /etc/debian_version is available, if this is the case, it will launch run-mailcap, which is very similar to xdg-open and tries to find a program to open the file as well.

Only if this fails too, xdg-open will ask the user to define a program to open the file.