Wordpress - Deactivate plugin for a specific user group

I think the answer to this Disable plugin / plugin action via theme is good for base knowledge on how to disable plugins from code.

Adapting that knowledge to your needs will leave us with this:

add_action('admin_init', 'my_filter_the_plugins');    
function my_filter_the_plugins()
{
    global $current_user;
    if (in_array('media_manager', $current_user->roles)) {
        deactivate_plugins( // deactivate for media_manager
            array(
                '/advanced-page-manager/advanced_page_manager.php'
            ),
            true, // silent mode (no deactivation hooks fired)
            false // network wide
        );
    } else { // activate for those than can use it
        activate_plugins(
            array(
                '/advanced-page-manager/advanced_page_manager.php'
            ),
            '', // redirect url, does not matter (default is '')
            false, // network wise
            true // silent mode (no activation hooks fired)
        );
    }
}

Basically this happens:

For the media_manager user group the my_filter_the_plugins disables(silently) the advanced-page-manager plugin. We then need to reactivate the plugin(silently, again) for those that aren't in the media_manager user group.