Disable gesture recognizer

Why don't you set the delegate for the swipe gesture recognizer too and handle them within the same delegate method.

- (BOOL)gestureRecognizer:(UIGestureRecognizer *)gestureRecognizer shouldReceiveTouch:(UITouch *)touch {
    if ( [gestureRecognizer isMemberOfClass:[UITapGestureRecognizer class]] ) {
        // Return NO for views that don't support Taps
    } else if ( [gestureRecognizer isMemberOfClass:[UISwipeGestureRecognizer class]] ) {
        // Return NO for views that don't support Swipes
    }

    return YES;
}

UIGestureRecognizer has a property named enabled. This should be good enough to disable your swipes:

swipeGestureRecognizer.enabled = NO;

Edit: For Swift 5

swipeGestureRecognizer.isEnabled = false