Wordpress - Add custom options to the wplink dialog

The dialog HTML comes from WP_Editors::wp_link_dialog() but no hooks in there.

We could instead use jQuery to append the custom HTML to the link dialog and try to override e.g. the wpLink.getAttrs(), because it's very short ;-)

Demo example:

jQuery( document ).ready( function( $ ) {
    $('#link-options').append( 
        '<div> 
            <label><span>Link Class</span>
            <select name="wpse-link-class" id="wpse_link_class">
                <option value="normal">normal</option>
                <option value="lightbox">lightbox</option>
           </select>
           </label>
       </div>' );

    wpLink.getAttrs = function() {
        wpLink.correctURL();        
        return {
            class:      $( '#wpse_link_class' ).val(),
            href:       $.trim( $( '#wp-link-url' ).val() ),
            target:     $( '#wp-link-target' ).prop( 'checked' ) ? '_blank' : ''
        };
    }
});

I just did a quick test and it seems to work but needs further testing and adjustments when updating links. Here's an old hack that I did that might need a refresh.

Update

It looks like you want to add the rel="nofollow" option to the link dialog, so let's update the above approach for that case:

We add the rel link attribute with:

/**
 * Modify link attributes
 */
wpLink.getAttrs = function() {
    wpLink.correctURL();        
    return {
        rel:        $( '#wpse-rel-no-follow' ).prop( 'checked' ) ? 'nofollow' : '',
        href:       $.trim( $( '#wp-link-url' ).val() ),
        target:     $( '#wp-link-target' ).prop( 'checked' ) ? '_blank' : ''
    };
}

If the rel attribute is empty, then it will be automatically removed from the link in the editor.

Then we can hook into the wplink-open event that fires when the link dialog is opened. Here we can inject our custom HTML and update it according to the current link selection:

$(document).on( 'wplink-open', function( wrap ) {

    // Custom HTML added to the link dialog
    if( $('#wpse-rel-no-follow').length < 1 )
        $('#link-options').append( '<div> <label><span></span> <input type="checkbox" id="wpse-rel-no-follow"/> No Follow Link</label></div>');

    // Get the current link selection:
    var _node = wpse_getLink();

    if( _node ) {
        // Fetch the rel attribute
        var _rel = $( _node ).attr( 'rel' );

        // Update the checkbox
        $('#wpse-rel-no-follow').prop( 'checked', 'nofollow' === _rel );
    }

});

where we use the following helper function, based on the getLink() core function, to get the HTML of the selected link:

function wpse_getLink() {
    var _ed = window.tinymce.get( window.wpActiveEditor );
    if ( _ed && ! _ed.isHidden() ) {
        return _ed.dom.getParent( _ed.selection.getNode(), 'a[href]' );
    } 
    return null;
}

Here's the output:

with the following HTML:

ps: This could be tested further and might also be extended to support translations

Looking at the core, there's no trace of any filter or action in the wp_link_dialog function, which would have made your life easier...

Investigating how others has solved this problem, there's a plugin that does more or less the same as you want. Basically it deregisters the wplink.js, wp_deregister_script('wplink'); and registers again a modified version of the file, this time including the desired extra field.

Although I don't think this is the best method (taking into account possible subsequent conflicts while updating WordPress), I think that it is the easiest wat to get it.

Hope it helps!