Wordpress - Updates for a private plugin?

Looks like the applicable code is in wp-includes/update.php, wp_update_plugins():

$to_send = (object) compact('plugins', 'active');

$options = array(
    'timeout' => ( ( defined('DOING_CRON') && DOING_CRON ) ? 30 : 3), 
    'body' => array( 'plugins' => serialize( $to_send ) ),
    'user-agent' => 'WordPress/' . $wp_version . '; ' . get_bloginfo( 'url' )
);  

$raw_response = wp_remote_post('http://api.wordpress.org/plugins/update-check/1.0/', $options);

It specifically checks api.wordpress.org. Technically speaking it would be possible to pass a key inside $to_send to delegate the check, but to my knowledge that is not a supported feature.

If you hook into set_site_transient_update_plugins you could add your own package details into this variable. It looks like those values will be trusted when you run the plugin updater. See wp-admin/update.php and wp-admin/includes/class-wp-upgrader.php. Given the code in these two functions, I think it would be possible to inject your own update server, you just need to look at how the package details are formatted and match that.

This plugin does it for you:

Automatic Updates For Private And Commercial Plugins

Since time immemorial, only plugins hosted in the official WordPress.org plugin directory have supported automatic updates. Now, I’ve written a PHP library that you can use to add automatic update capabilities to any plugin. Public, private and commercial plugins alike – all can now enjoy the benefits of automatic update notifications and one-click upgrades.

_{GitHub repository}

If you like free ... and want to use something like GitHub, GitLab, or BitBucket, this is a good quality and supported plugin for handling it (supports private and enterprise):

https://github.com/afragen/github-updater