Apple - How can I safely install applications which aren't distributed via the Mac App Store?

There can be many reasons for app developers to stay away from the App Store. For example:

• Wanting to avoid paying fees/percentages to Apple
• App has functionality not possible in the App Store due to its sandboxing requirement or Apple's review guidelines
• Not wanting to spend time on getting in the App Store (usually for programs that have been on the market long before the App Store was a thing)

Checksums is definitely a very common way of validating downloaded files. Combined with a cryptographically secure certificate of authenticity for the checksum, and you can have a certain degree of trust in that the downloaded file originates from the app developer and is not tampered with by third parties.

If developers pay Apple $99 a year, they can cryptographically sign their applications, so that the OS will recognise their apps as being from trusted, identified developers.

The Security pane in System Preferences has a setting to allow apps from either the App Store only, or "App Store and identified developers". (There used to be a third option: "Any".)

On first launch, the app will be verified, and any 'modification' by malware should be picked up at this stage. Apple can revoke certificates from existing developers.

It should therefore be 'safe' to allow apps from identified developers to launch, even if not from the App Store.

However, some developers may baulk at having to pay Apple just to write some code. Products that are multi-platform, particularly those that originate on Linux or generic Unix products (e.g. open-source software) may not see any point or benefit from using the App Store, and may even be philosophically opposed to it.

Non-app software is also not eligible for the App Store, e.g. languages like Python, which is just an installed framework, accessed on the command line.

There are some types of apps that are not allowed in the App Store but are still useful or actually needed to get a job done. The accepted answer lists some of the reasons. Sometimes, we app developers need access to certain APIs to provide the intended functionality, but Apple does not allow apps in the App Store which access those. As a specific example, my company provides a VPN client and we need to access low-level APIs to be able to make VPN connections. But access to these APIs is forbidden for apps in the App Store.

Apple has recognized that there are apps which simply cannot be distributed via the App Store, for various reasons, and has provided an alternative to ensure at least some safety for users: Developers can (and now sometimes must) notarize apps. For example, we must notarize when distributing Kernel Extensions or System Extensions. As a user, you actually need to actively disable System Integrity Protection to allow un-notarized apps with Kernel or System Extensions on macOS 10.15 Catalina and 11.0 Big Sur.

What Notarization means is that we developers build an app, then pass it to Apple which scans it and if Apple thinks the app is free of malware, a "ticket" is saved with Apple that macOS can query. (Developers can then "staple" the ticket to the app so macOS doesn't need to query it via the Internet. It's cryptographically signed by Apple so you can't fake it.) The app is not reviewed like for the App Store, it's just an automated malware check and that's basically it.

When you start such a notarized app, you get a dialog that warns you that the app was provided by a third-party developer, but also informs you that Apple has scanned it and deemed it safe for execution:

When you see this dialog (wording may change, but will contain a phrase that Apple has checked it), it's reasonably safe for you as a user to start that third-party notarized app. The notarization ensures the app has not been tampered with (or else the cryptographic sign would break and macOS would refuse to start it). Apple vouches that it checked the app for "malicious content".

This does not guarantee that the app does not contain malicious content (developers can hide stuff from Apple if they really want to). Only that Apple wasn't able to find any. However, the notarization process allows Apple to retro-actively revoke a ticket if the app turned out to be malicious, thus preventing the app from running on your Mac. Apple was somewhat able to do so before with signed apps, but only by revoking the certificate of the developer, thus blocking all apps of said developer. Notarization allows Apple to block on a per-app-and-version level. So Apple is able to specifically block version 1.2.3 of Foo.app but allow version 1.2.4 to run.