How to use espresso matcher as a condition for If else statement?

Espresso was designed in a way to discourage devs from using conditionals, so there's no officially supported way to do this.

However, there are hacks you can try. I use try/catch statements. In your case, it would be something along the lines of:

private void SelectTransitBackendOnline(String env) {    
     onView(withText("Some Text")).perform(click());

     try {
           onView(withText(env)).check(matches(isChecked())))
           onView(withId(R.id.dialogCancel)).perform(click());
     } catch (AssertionFailedError e) {
           onView(withText(env)).perform(click());
     }
}

Depending on what you are doing in the try block, change the catch block's exception. I wanted to click on cancel_button if it exists, so I changed it to:

try {
    onView(withId(R.id.cancel_button)).perform(click());
} catch (NoMatchingViewException ignore) {
}