Wordpress - Why is WordPress Multisite redirecting to wp-signup.php and how to fix it?

After many hours of debugging and despair, the problem is now solved. It turned out to be a very obscure thing.

The redirect is issued by the function ms_load_current_site_and_network() inside /wp-includes/ms-load.php. It was issuing the redirect because /wp-includes/ms-settings.php was not able to set a domain. The reason it was not able to set a domain is because $_SERVER['HTTP_HOST'] was not set.

$_SERVER['HTTP_HOST'] not being set had to do with the PHP setting auto_globals_jit. It was set to On, resulting in the $_SERVER array not being set (as explained in this comment). Even though $_SERVER['HTTP_HOST'] and the rest were defined when loading a single php file with phpinfo(); in it, they were not defined when loading Wordpress's php files. And that led to the redirect.

Setting auto_globals_jit to Off in php.ini resolved it.

Bonus info:

The issue was additionally made more complicated by some form of caching (which I couldn't pin down or clear). So I had to rely on behavior that I saw occur only once and couldn't replicate afterwards. For example, I created a separate php file to just check if $_SERVER['HTTP_HOST'] is set - and initially it wasn't. Following the suggestion from the comment, I used $_SERVER['HTTP_HOST'] twice in the php file - and immediately saw that it was now set. However, when reverting the change and having $_SERVER['HTTP_HOST'] only once in the php file, it still remained set - suggesting I was seeing a cached response. I coulnd't get it to break again after I fixed it once.

Just documenting this here in case someone encounters similar behavior.

I can't comment so I'm leaving this answer. As a follow up to the fantastic research done by @Borislav there is another SE thread that is related, with several potential fixes:

Site Redirecting to wp-signup.php

Ultimately, all the redirect issues with WordPress Multisite seem related to siteurl and home which even if properly hardcoded (e.g. using WP_HOME and WP_SITEURL in the wp-config.php file) will still result in redirect issues if the accompanying fields in wp_options table are not also correct. This is due to the way that WordPress Multisite processes the initial setup routine; while I haven't found any documentation on the reason for this, I believe it was probably done on purpose by a Core dev to ensure no conflicts arise later on in the functionality of the Multisite installation, which is more prone to issues than single sites.

Some have suggested implementing NOBLOGREDIRECT as a fix but this does not seem to work, and does not address the underlying issue. That setting is only meant to redirect visitors who land on non-existent sub-sites of your Multisite network (e.g. if registration is disabled for new blogs).

Ref: https://www.sarahgebauer.com/today-i-learned-multisite-mystery-signup/

Ref: https://gist.github.com/dejanmarkovic/8323792

In conclusion:

Always ensure that your wp_options table has the siteurl and home settings updated to accurately reflect the main domain of your Multisite installation. In certain cases, running a search/replace across the entire MySQL database may be required, depending on the history of your domain (and database), in particular to target incorrect entries in the wp_blogs and wp_site tables...