404 Exception not handled in Spring ControllerAdvice

To make it work, you need to set throwExceptionIfNoHandlerFound property on DispecherServlet. You can do that with:

spring.mvc.throwExceptionIfNoHandlerFound=true

in application.properties file, otherwise the requests will always be forwarded to the default servlet and NoHandlerFoundException would ever be thrown.

The problem is, even with this configuration, it doesn't work. From the documentation:

Note that if org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler is used, then requests will always be forwarded to the default servlet and NoHandlerFoundException would never be thrown in that case.

Because Spring Boot uses by default the org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler you'll have to override this using your own WebMvcConfigurer:

import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.DefaultServletHandlerConfigurer;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;

@EnableWebMvc
@Configuration
public class WebConfig implements WebMvcConfigurer {
    @Override
    public void configureDefaultServletHandling(DefaultServletHandlerConfigurer configurer) {
        // Do nothing instead of configurer.enable();
    }
} 

Of course, the above class might be more complicated in your case.