How to trigger screen recording permission system modal dialog on macOS Catalina

macOS 11+

Apple now provides an API to determine if your app has Screen Recording access via CGPreflightScreenCaptureAccess() and also to request access via CGRequestScreenCaptureAccess(). Requesting access will present the system prompt and automatically add your app in the list so the user just needs to enable access. The system prompt will only appear once per app session. For an example, let's say they click Deny in the first prompt. If your app requests access again the prompt won't appear. But if they quit and relaunch your app, your app could request access again and the prompt would appear.

Original answer:

The screen recording prompt will appear only once - the first time you invoke an API that attempts to record the user's screen, such as:

CGDisplayStreamRef stream = CGDisplayStreamCreate(CGMainDisplayID(), 1, 1, kCVPixelFormatType_32BGRA, nil, ^(CGDisplayStreamFrameStatus status, uint64_t displayTime, IOSurfaceRef frameSurface, CGDisplayStreamUpdateRef updateRef) {
});
if (stream) {
    CFRelease(stream);
}

As you noted, your app won't appear in System Preferences under Screen Recording until you invoke a screen recording API thus triggering the system prompt.

If you trigger the prompt and the user denies it, you cannot bring up the prompt again - the user must manually enable it in System Preferences.

Helpful testing info:

While building and testing this, you can reset your app's permissions as if you never invoked a screen recording API, via tccutil reset ScreenCapture com.company.appname. Or use All instead of ScreenCapture to reset all things permissions for your app.

I answered this same question on Ask Different.SE. You need tccutil to reset these permissions.

Reset the privacy database for Screen Recording apps:

tccutil reset ScreenCapture

Or if you know the app bundle identifier, you can reset a single app.

tccutil reset ScreenCapture [com.WHATEVERBUNDLE.YOURAPPID]

Once you've reset the privacy permissions, you must quit your application before the change will take effect. Then you can restart your app and try screen recording again, and the prompt should reappear.