Drupal - What is the correct way to update existing module's rules programmatically?

You can export the rules configuration via the Rules UI and then implement hook_default_rules_configuration_alter() like this:

/**
 * Implements hook_default_rules_configuration_alter().
 */
function MODULENAME_default_rules_configuration_alter(&$data) {
  if (isset($data['commerce_cart_add_to_cart_message'])) {
    $data['commerce_cart_add_to_cart_message'] = entity_import('rules_config', '{ "commerce_cart_add_to_cart_message" : {
        "LABEL" : "Display an Add to Cart message",
        "PLUGIN" : "reaction rule",
        "OWNER" : "rules",
        "TAGS" : [ "Commerce Cart" ],
        "REQUIRES" : [ "rules", "commerce_cart" ],
        "ON" : { "commerce_cart_product_add" : [] },
        "IF" : [
          { "data_is" : { "data" : [ "commerce-order:commerce-nocart" ], "value" : "0" } }
        ],
        "DO" : [
          { "commerce_cart_add_to_cart_message" : { "commerce_product" : [ "commerce-product" ] } }
        ]
      }
    }');
  }
}

Just change the instances of $data['commerce_cart_add_to_cart_message'] to use your rule machine name and replace the export string with your own.

You can also use rules_import() instead of entity_import(). Either is acceptable.

Since the rules module loads default rules from code into the database and from then on loads them from the database, it won't immediately pick up your override. To make it do so navigate to the Rules UI and revert the rule. It will revert back to the default code, which will be your override.

To programmatically revert a rule you can do this (again replacing the rule machine name):

if ($rules_config = rules_config_load('commerce_cart_add_to_cart_message')) {
  $rules_config->delete();
}

An example usage of that may be to do it during an update function for a custom module.

It may seem wring to be deleting it but delete and revert are equivalent for rules that are provided by code because delete means delete from the database. If this was a rule created via the UI then it would be gone, however a rule from code will just delete from the DB and then read in again from the code.

Unfortunately there are currently only drush commands for enable/disable of rules. There is a feature request for a delete/revert command at https://www.drupal.org/node/2507187

To update module's rules programmatically you can load them and change them. Example of removing single action:

$rule = rules_config_load("some_rule");
foreach ($rule->actions() as $action) {
  if (strpos($action->getPluginName(), 'drupal_message') !== FALSE) {
    $action->delete();
  }
}
$rule->save();

If it's one time change, add it into hook_update.

See: RulesPlugin for API.