How to manage installation from Unknown Sources in Android Oreo?

For starters, your application needs to declare a targetSdkVersion of 26 (API level of Android Oreo) or higher in your build.gradle or AndroidManifest.xml for all this to work.

Then on to answer the questions above:

1. How to check whether I'm allowed to request a package install?

You can check this using getPackageManager().canRequestPackageInstalls() anywhere in your Activity's code. Note that this method always returns false, if you don't declare the appropriate permission or target the wrong SDK version.

2. What exact permission do I have to request?

You have to declare Manifest.permission.REQUEST_INSTALL_PACKAGES in your AndroidManifest.xml, like so:

    <uses-permission android:name="android.permission.REQUEST_INSTALL_PACKAGES" />

3. How can I prompt the user to grant this permission?

You can send the user to the appropriate destination, with Intent ACTION_MANAGE_UNKNOWN_APP_SOURCES:

startActivity(new Intent(android.provider.Settings.ACTION_MANAGE_UNKNOWN_APP_SOURCES));

You may also send the user more directly to the specific setting for your application, with:

startActivity(new Intent(android.provider.Settings.ACTION_MANAGE_UNKNOWN_APP_SOURCES, Uri.parse("package:your.application.package")));

4. How do I prompt the user to install a specified .apk?

Once you made sure you are granted the appropriate permission, you can prompt the user to install your .apk file anywhere in your Activity's code (where this refers to your Activity's Context), using:

Intent intent = new Intent(Intent.ACTION_INSTALL_PACKAGE);
intent.setDataAndType(FileProvider.getUriForFile(this, "your.application.package.fileprovider", new File("/path/to/your/apk")), "application/vnd.android.package-archive");
intent.setFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);

startActivity(intent);

You may also add intent.putExtra(Intent.EXTRA_RETURN_RESULT, true) and start with startActivityForResult(Intent, int), if you want to know if the installation succeeded, was cancelled or failed.

For information on how to correctly get your .apk file's Uri, see FileProvider.

From adb shell you can mange app specific permissions using appops. Example:

# To get a list of packagenames that is allowed to install from other sources
appops query-op REQUEST_INSTALL_PACKAGES allow 

# To allow an app install packages from other sources
appops set <package name> REQUEST_INSTALL_PACKAGES deny 

Help text printed when running adb shell appops help:

AppOps service (appops) commands:
  help
    Print this help text.
  set [--user <USER_ID>] <PACKAGE | UID> <OP> <MODE>
    Set the mode for a particular application and operation.
  get [--user <USER_ID>] <PACKAGE | UID> [<OP>]
    Return the mode for a particular application and optional operation.
  query-op [--user <USER_ID>] <OP> [<MODE>]
    Print all packages that currently have the given op in the given mode.
  reset [--user <USER_ID>] [<PACKAGE>]
    Reset the given application or all applications to default modes.
  write-settings
    Immediately write pending changes to storage.
  read-settings
    Read the last written settings, replacing current state in RAM.
  options:
    <PACKAGE> an Android package name.
    <OP>      an AppOps operation.
    <MODE>    one of allow, ignore, deny, or default
    <USER_ID> the user id under which the package is installed. If --user is not
              specified, the current user is assumed.