Selecting all visible rows after a search using 'selectAll' button in Data-table

I believe your general problem is that you are not resetting deselected (not filtered) rows. You are just toggling .selected for filtered rows, ultimately ending up in all rows being selected. Also, I would place the code in the action method, since you are in fact overriding the selectAll default functionality.

{
   extend: 'selectAll',
   className: 'selectall',
   action : function(e) {
     e.preventDefault();
     table.rows({ page: 'all'}).nodes().each(function() {
       $(this).removeClass('selected')
     })
     table.rows({ search: 'applied'}).nodes().each(function() {
       $(this).addClass('selected');        
     })
   }
}

demo -> https://jsfiddle.net/sqmz7z76/

davidkonrad's code is mostly right, but it's not using the official methods. This is the correct code to use:

{
  extend: 'selectAll',
  className: 'selectall',
  action : function(e) {
    e.preventDefault();
    table.rows({ search: 'applied'}).deselect();
    table.rows({ search: 'applied'}).select();
  }
}

Instead of simply toggling the class 'selected', it calls the method 'select()', which in turn, enables the "Deselect all" button and displays the footer text telling you how many rows have been selected.