How to check if bootstrap modal is open, so I can use jquery validate?

You can use

$('#myModal').hasClass('in');

Bootstrap adds the in class when the modal is open and removes it when closed

To avoid the race condition @GregPettit mentions, one can use:

($("element").data('bs.modal') || {})._isShown    // Bootstrap 4
($("element").data('bs.modal') || {}).isShown     // Bootstrap <= 3

// or, with the optional chaining operator (?.)
$("element").data('bs.modal')?._isShown    // Bootstrap 4
$("element").data('bs.modal')?.isShown     // Bootstrap <= 3

as discussed in Twitter Bootstrap Modal - IsShown.

When the modal is not yet opened, .data('bs.modal') returns undefined, hence the || {} - which will make isShown the (falsy) value undefined. If you're into strictness one could do ($("element").data('bs.modal') || {isShown: false}).isShown