How to use Wayland in Fedora?

As far as I know, the GNOME option does already use Wayland. So, if you want to use Wayland, just choose GNOME.

If not, maybe try troubleshooting your HW? Is it outdated/old? Maybe try to reinstall the system?

Recent versions of Fedora uses Wayland by default. However, things can prevent Fedora from using it, mainly drivers. Most likely, it is only one thing: NVIDIA proprietary drivers.

To enable Wayland for Gnome in Fedora, two main actions are needed, as explained below in details:

1. TL;DR: Enable Wayland in the GDM configurations in /etc/gdm/custom.conf

   1. Open /etc/gdm/custom.conf for editing (as you know, you must be root).

   2. Go to the line which looks like the following:

      [daemon]
      WaylandEnable=false
      

   3. Change it to (even if the line was commented):

      [daemon]
      WaylandEnable=true
      

   4. Save the file and exit.

2. TL;DR: Comment out all Wayland-disabling lines in /usr/lib/udev/rules.d/61-gdm.rules (thanks to this forum thread)

   1. Open /usr/lib/udev/rules.d/61-gdm.rules for editing (root, again).

   2. Comment lines which causes Wayland not to start; usually, it's related to proprietary NVIDIA drivers. The comments before each line should help you on that. For example, I commented the following line:

      # disable Wayland when using the proprietary nvidia driver
      DRIVER=="nvidia", RUN+="/usr/libexec/gdm-disable-wayland"
      

   3. Save the file and exit.

      Note: As you are probably using NVIDIA proprietary drivers, this step should make you able to use Wayland alongside these drivers. See egl-wayland and this.

      However, you may not experience the best performance, as well as having other problems. For example, nvidia-settings only work in X11, Xwayland has problems with 3D hardware accelerations, and as a result, glxinfo shows llvmpipe as the renderer (also the About section on recent version of Gnome Settings). Nonetheless, you can verify the running driver by lspci -vnn, if you did the steps correctly. See this, also.

      Duh. It's NVIDIA, BTW.

Restart your computer (a logout and login should not be enough). Before logging in, inside logging options, you should now see both "Gnome" and "Gnome on Xorg", which the first one is the Wayland option. Using it, you should able to use Wayland on Gnome. Enjoy it!