How to create nested JSON object from custom sObject?

I suggest building and serializing nested maps for this:

Map<String, Object> m = new Map<String, Object>();
for (options__C opt:options) {
    m.put(opt.Name__c, new Map<String, Boolean>{
            'isActive' => opt.isActive__c,
            'isRequired' => opt.isRequired__c
            });
}
return JSON.serialize(m);

Desired format cannot be achieved with standard wrapper structure.

Usually I use maps to serialize for unsupported structure. An example below:

Map<String, String> mapOption = new Map<String, String>();
Map<String, Boolean> mapProperty = new Map<String, Boolean>();

for(options__C opt: [SELECT Name__c, isActive__c, isRequired__c FROM 
    options__C WHERE 
    group__c = 'NEW_SUB_CONF']){

    // collecting property as JSON string
    String strJsonProperty = JSON.serialize(mapProperty = new Map<String, Boolean>{
        isActive => opt.isActive__c,
        isRequired => opt.isRequired__c
    });

    // form map of options
    mapOption.put(opt.Name__c, strJsonProperty);
}

// JSON in your format
return JSON.serialize(mapOption);

---

Here is an example to illustrate:

Map<String, String> mapOption = new Map<String, String>();
Map<String, Boolean> mapProperty = new Map<String, Boolean>();

// collecting property as JSON string
String strJsonProperty = JSON.serialize(mapProperty = new Map<String, Boolean>{
    'isActive' => true,
    'isRequired' => true
});

// form map of options
mapOption.put('option1', strJsonProperty);
mapOption.put('option2', strJsonProperty);

String finalOutput = JSON.serialize(mapOption);

System.debug(finalOutput);

Output:

{"option2":"{\"isRequired\":true,\"isActive\":true}","option1":"{\"isRequired\":true,\"isActive\":true}"}