Correct, Supported way to Add CLI Commands to Magento 2

The correct way is:

Create your module as you do for any kind of module

Just create your registration.php file

\Magento\Framework\Component\ComponentRegistrar::register(
    \Magento\Framework\Component\ComponentRegistrar::MODULE,
    'My_Module',
    __DIR__
);

And create you module.xml file:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Module/etc/module.xsd">
    <module name="My_Module" setup_version="0.1.0">
    </module>
</config>

Add an entry in di.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <type name="Magento\Framework\Console\CommandList">
        <arguments>
            <argument name="commands" xsi:type="array">
                <item name="my_command" xsi:type="object">My\Module\Command\Mycommand</item>
            </argument>
        </arguments>
    </type>
</config>

Create your command class:

<?php
namespace My\Module\Command;

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class Mycommand extends Command
{
    protected function configure()
    {
        $this->setName('my:command');
        $this->setDescription('Run some task');

        parent::configure();
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $output->writeln('Hello world!');
    }
}

To run your task just type:

php bin/magento my:command

About compatibility:

@api is not needed for commands, it is used for service contracts AFAIK.

If you need to let them compatible, just use an interface API inside your script instead of putting the logic inside it.

For example:

<?php
use My\Module\Api\TaskInterface;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class MyCommand extends Command
{
    protected $taskInterface;

    public function __construct(
        TaskInterface $taskInterface
    ) {
        $this->taskInterface= $taskInterface;
        parent::__construct();
    }

    protected function configure()
    {
        $this->setName('my:command');
        $this->setDescription('Run some task');

        parent::configure();
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $this->taskInterface->runTask();

        $output->writeln('Done.');
    }
}

cli_commands.php should be used in case the command is added in a non-modular package. So if the command is in the module and it's OK (expected) that it's available only when the module is enabled, di.xml should be used. If you don't want to add a module and want to have just an arbitrary Composer package, you can use cli_commands.php to register command there. Of course, it should be then really independent from Magento. Or, for now, this approach can be used to register command(s) necessary even if a module is disabled (ensure that it does not rely on any Module's logic that works only when it's enabled).

if I got it right, commands defined in the CommandList over DI are only available in an installed Magento Instance and also only for Magento Modules (since they have to be defined in the di.xml): https://github.com/magento/magento2/blob/6352f8fbca2cbf21de88db0cf7f4555bfc60451c/lib/internal/Magento/Framework/Console/Cli.php#L124

the Magento\Framework\App\DeploymentConfig::isAvailable() in the above method checks for an install Date in the Config to check for an installed Magento2: https://github.com/magento/magento2/blob/6352f8fbca2cbf21de88db0cf7f4555bfc60451c/lib/internal/Magento/Framework/App/DeploymentConfig.php#L83).

The commands defined in the Magento\Framework\Console\CommandLocator on the other hand are always available and even can be defined by non Magento Modules through the static CommandLocator::register method in a file autoloaded by composer (for example cli_commands.php)

https://github.com/magento/magento2/blob/6352f8fbca2cbf21de88db0cf7f4555bfc60451c/lib/internal/Magento/Framework/Console/Cli.php#L130

https://github.com/magento/magento2/blob/6352f8fbca2cbf21de88db0cf7f4555bfc60451c/lib/internal/Magento/Framework/Console/Cli.php#L146

So I think both methods are needed and have its right to exist