When is it appropriate to use the @TestVisible Annotation

I think the question in your title is too broad and opinion based to touch on here. I have worked with many developers who feel the @TestVisible annotation should never be used, but regardless, I have found some instances where it felt like the best option. For me, it boils down to using it only when I need to be able to control application behavior but want to avoid checking Test.isRunningTest().

For your use case, let's take a moment to think about the behavior you actually care about. You want the function to toggle the selection of all of your users (I would rename it from select to toggle to reflect that purpose). So what you really care about is not this Boolean flag at all, but rather your collection of wrapper instances. That is what you should be asserting against. Make sure its size hasn't changed, and that the individual selections are correct.

Then again, if you take a step back and look at what you are trying to accomplish, do you really need to involve the server at all for this mechanism? It may be worth looking at how you can move the functionality wholesale to the client side, since it is not exactly complex logic where unit tests offer significant advantages over acceptance tests run by end users.

---

(Just an aside to show the one time I do find @TestVisible perfectly acceptable)

When writing trigger handlers, it is useful to use a static flag to turn the functionality off when running certain tests to reduce the cost of DML Operations on that object. Granted, there are other strategies you can employ to reduce or remove the need to perform DML in the first place. But sometimes you can't, and making trigger execution cheaper to run can dramatically simplify/quicken your test suite.

public with sharing class MyTriggerHandler
{
    @TestVisible static Boolean bypassTrigger = false;

    // constructor logic

    public void beforeInsert()
    {
        if (bypassTrigger) return;

        // implementation logic
    }

    // other events
}

Then, in some test where you don't actually care about the trigger, but need your records to exist in the database:

static testMethod void testSomeFunctionality()
{
    MyTriggerHandler.bypassTrigger = true;
    insert myRecords;
    MyTriggerHandler.bypassTrigger = false;

    // other stuff
}

Generally, you'd use @TestVisible when you need to control the internal state of an object, or you need to observe the internal state of an object that can't be deduced from the outside. Generally speaking, this is only for particularly esoteric designs. I've personally never had a use for this annotation, but it can make writing some kinds of tests easier to perform.

In this case, I'd argue that since userList is observable from the outside, calling the method and observing the checked state of the users in the list would more desirable than making sure the internal state variable was correct; the state variable isn't directly tied to the users' selected state, and so testing it independently has no real value.

As an aside, your method is unnecessarily verbose. You could have written the code like this:

public void toggleAllUsers() {
  isAllUserSelected = !isAllUserSelected;
  for(uUser user: userList) {
    user.selected = isAllUserSelected;
  }
}

As an addendum to Adrian's answer:

I use a lot of custom messaging assigned to final strings and they have an access of private

@TestVisible private static final string REQERROR = 'There was an error processing your request';

Adding the @TestVisible annotation allows me to use this property in my test classes to assert the appropriate messages are displayed while allowing me to change the message and not break the test classes.

The alternative would be to make the access public but i like to only make things public when they need to be public.

You can find use cases for this in many places.

The one thing I would says is if you have a method that is private you should generally not need to use @TestVisible because the method itself is private and thus if a test does not cover it without the @TestVisible then either there is something wrong with your code, test, or the method is unused.

The exception to the above is a helper method that you make private. If you want to use the results of that method in your assertions then you would need to use the @TestVisible