Change icon on click (toggle)

Instead of overwriting the html every time, just toggle the class.

$('#click_advance').click(function() {
    $('#display_advance').toggle('1000');
    $("i", this).toggleClass("icon-circle-arrow-up icon-circle-arrow-down");
});

If your icon is based on the text in the block (ligatures) rather the class of the block then the following will work. This example uses the Google Material Icons '+' and '-' icons as part of MaterializeCSS.

<a class="btn-class"><i class="material-icons">add</i></a>

$('.btn-class').on('click',function(){
    if ($(this).find('i').text() == 'add'){
        $(this).find('i').text('remove');
    } else {
        $(this).find('i').text('add');
    }
});

Edit: Added missing ); needed for this to function properly.

It also works for JQuery post 1.9 where toggling of functions was deprecated.