Drupal - Alter hook_field_formatter_view?

You may want to try to alter it with a preprocess function:

function example_preprocess_field(&$variables) {  
  $field_name = $variables['element']['#field_name'];  
  if ($field_name == 'field_example_field') {
    // Your code here.
  }
}

I ran a test to modify $variables['element']['#attached']['js'] but Drupal didn't recognize the changes to the element. Not sure why. So, you may have to use drupal_add_js.

The hook_field_formatter_view() specified by the display configuration is called in field_default_view(), which is itself called (a layer or two below it) by field_attach_view().

You can see from the second link that this calls hook_field_attach_view_alter() after it's finished with the field formatters:

drupal_alter('field_attach_view', $output, $context);

so you should be able to define that hook.

Note, though, that you'll have to re-find the field on the entity, as it runs the hook for the entire render array, not simply the render array for one field. But the example hook linked to above gives an example of how to do that.