Custom Setting proxy class, how to make its operation transparent outside of tests?

You can use a class to handle the access to the custom settings and have it implement a custom logic using Test.isRunningTest(). The idea would be that when not running a test, it would return the correct type of the custom setting the caller wants to access. When running a test though, it could search for the value in a private, test visible, map of values.

private static Map<String, Object> values;

public static Object getValue(String csName) {
    if (Test.isRunningTest()) {
        return values.get(csName);
    }
    return CustomSetting__c.getInstance().Value__c;
}

In this scenario, in your test, you would set the values of values before executing the unit test:

CustomSettingProxy.values.put('DoTheThing__c', true);

Test.startTest();

// ... it should do the thing when the code tries to access
// the custom setting value

Test.stopTest();

Another approach, elaborating a little more on the use of Test.isRunningTest() method, is to create a class that has a mapping to the implementation of a class that handles the call to the custom setting. This way you would be effectively creating a class for each custom setting, implementing a common interface. In this approach the proxy class has a mapping to which implementation it should call using Type.forName, for example.

Then you would have each custom setting pointing to each respective class implementing such interface to get the values from the SObject. The same principle of substitution applies here: before the unit test is actually run you would replace the mapping with a mock class (which I believe could be a subclass of your test class).

These solutions look more like the third approach mentioned: Create and maintain separate proxy classes for each custom setting.

Our solution was to introduce a "settings repository" class. This didn't try to be "all things to all men" and instead exposed custom settings relevant to the app. I'm not even sure it makes sense to try to do that anyway as an app will have certain expectations for the various custom settings types it uses.

The class's job is to in-memory cache specific settings instances. Doing this means that, as long as all code (production and unit tests) use this "settings repository" class, unit tests can invoke the class to get the required settings instance and set the fields it wants to set. As long as this happens before the tested production code looks at the field values, the tested code sees the correct settings data.

We don't even care that the unit tests will actually load the in-org settings data because the unit tests are written to explicitly set those fields needed by the tested code. This means the "settings repository" simply invokes the normal custom settings method, but only if it doesn't already have a cached value.

Our repository class has type-specific custom settings methods that support the settings in a manner appropriate to how we expect them to be used (e.g. for some hierarchy settings we get the user-specific instance but for others we always use the org default since we don't want it user-specific).

For example:

public class Settings {
    private static ExampleSettings__c example = null;

    public ExampleSettings__c getExample() {
        if (example == null) {
            example = ExampleSettings__c.getInstance();
        }

        return example;
    }
}

Each time we have another type of settings we add another method for accessing it in an app-appropriate manner.

Production code simply accesses the settings as needed, e.g.:

if (Settings.getExample().SomeFlag__c) {
    // Do something appropriate
}

A unit test can include methods that test the settings-based scenarios very simply too:

Settings.getExample().SomeFlag__c = false;

// Now call the production code that uses SomeFlag__c. Of course, if there are
// multiple fields to be set the ExampleSettings__c instance can be held in a
// local variable for improved readability

Or:

Settings.getExample().SomeFlag__c = true;

// Again, call the production code

This is a cheap way of addressing the production vs test case with minimal overhead and still leveraging the underlying custom settings caching that Salesforce provides.

I know this doesn't do quite what you discussed, but like I said - it makes sense for the app code to constrain how settings are accessed across the entire code-base.