Why is custom system classloader not working?

Your problem is that your custom class loader is being used to load Main, but its loadClass simply delegates to the parent class loader to load Main. Therefore. within Main, if you called Main.class.getClassLoader(), it would return the sun.misc.Launcher$AppClassLoader, not MyLoader.

To see what class loader will be used for Class.forName(String) calls and symbolic references from your own class, you should print getClass().getClassLoader() (or MyClass.class.getClassLoader() from a static method). The class loader that is used is the one that defined the class whose code is currently being executed. This is the rule everywhere except when using reflection (Class.forName(String, boolean, ClassLoader)).

Once a class is loaded from the parent class loader, any classes that it loads will also use that primitive class loader. So once Main is loaded from the sun.misc.Launcher$AppClassLoader class loader, all the classes that it calls will come from that same class loader, not from your own MyLoader. Similarly, once the javax.crypto.Cypher class is loaded from the null (aka Bootstrap) class loader, any classes that it mentions will also come from the bootstrap class loader except the classes it loads using reflection (SPI).

To stop loading classes from the sun.misc.Launcher$AppClassLoader class loader, set MyLoader's CLASSPATH to AppClassLoader's CLASSPATH and don't delegate classloading to AppClassLoader. Note that this will cause all the CLASSPATH classes to be loaded from MyLoader, but the classes from JDK will in general still be loaded from the null (Bootstrap) class loader.

To stop loading JDK classes from the bootstrap class loader, you must explicitly put the JDK into the classpath and modify loadClass to not check the parent first for some classes. Loading JDK classes from your own class loader is delicate; some classes (e.g. java.lang.String) must be loaded from the boostrap class loader. This is not something I have tried myself, but I have read that OSGi loads java.* from the bootstrap class loader but loads other JDK classes (e.g. sun.* and javax.*) from its own graph of class loaders.

/** Run with -Djava.system.class.loader=MyLoader to use this class loader. */
public static class MyLoader extends URLClassLoader {
    public MyLoader(ClassLoader launcherClassLoader) {
        super(getUrls(launcherClassLoader), launcherClassLoader.getParent());
    }

    private static URL[] getUrls(ClassLoader cl) {
        System.out.println("---MyLoader--- inside #constructor(" + cl + ")...");
        return ((URLClassLoader) cl).getURLs();
    }

    @Override public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
        System.out.println("---MyLoader--- inside loadClass(" + name + ", " + resolve + ")...");
        return super.loadClass(name, resolve);
    }
}

As for the SPI factories within JDK (think XML parsers and crypto implementations), they use reflection to load named classes from either the ContextClassLoader or the SystemClassLoader or one after the other, because they want you to be able to define your own implementation, and the bootstrap class loader does not load user-defined classes. There seems to be no consistency in which one of the two they used, and I wish they just took a ClassLoader parameter instead of guessing.

The Javadoc of Class.forName(String) states (emphasis mine):

Returns the Class object associated with the class or interface with the given string name. Invoking this method is equivalent to: Class.forName(className, true, currentLoader) where currentLoader denotes the defining class loader of the current class.

In other words, the method doesn't automatically use the system classloader - it uses the loader that physically defined the class from which it's called. From the Java language spec, section 5.3:

A class loader L may create C by defining it directly or by delegating to another class loader. If L creates C directly, we say that L defines C or, equivalently, that L is the defining loader of C.

Your custom classloader doesn't create the Main class directly - it delegates to the parent loader to create the class, so it's that parent classloader that will be used when you call Class.forName(String) in a method of Main. If you want to use your custom classloader directly, you'll need to either explicitly specify it using the 3-arg variant of Class.forName or alter your custom classloader implementation so that it actually loads the class in question (typically by extending URLClassLoader).