How to remove the delay when opening an Activity with a DrawerLayout?

I finally found a solution to this. I don't know why or how it worked out but I just know that it removed the delay in the animations. I added a handler in the OnCreate of the activity that would run the other statements for setting up, i.e. adding the initial fragment into view, after 300ms

Handler mHandler = new Handler();
mHandler.postDelayed(new Runnable() {
    @Override
    public void run() {
        switchFragment();
    }
}, 300);

Maybe its because lollipop has default layoutTransition on UI elements, have you tried?

drawerLayout.setLayoutTransition(null)

I would change your exit transition:

item name="android:windowExitTransition">@transition/fade_in_out_transition</item>
item 

To window return:

name="android:windowReturnTransition">@transition/fade_in_out_transition</item>

1. When you are using window exit, the visibility of the window is changed to invisible briefly before your next transition starts.

Sets the Transition that will be used to move Views out of the scene when the fragment is removed, hidden, or detached when not popping the back stack. The exiting Views will be those that are regular Views or ViewGroups that have isTransitionGroup() return true. Typical Transitions will extend Visibility as exiting is governed by changing visibility from VISIBLE to INVISIBLE. If transition is null, the views will remain unaffected.

setExitTransition

2. Returning the transition handles the window closing, as opposed to exiting and does not affect the window visibility.

Reference to a Transition XML resource defining the desired Transition used to move Views out of the scene when the Window is preparing to close. Corresponds to setReturnTransition(android.transition.Transition).

android:windowReturnTransition

3. I would also recommend using reenter to manage back presses.

Reference to a Transition XML resource defining the desired Transition used to move Views in to the scene when returning from a previously-started Activity. Corresponds to setReenterTransition(android.transition.Transition).

android:windowReenterTransition

Understanding exit/reenter shared element transitions

4. You can also set a bool value that will allow the transitions to overlap, however the overlap may be too long for what you want.

setAllowEnterTransitionOverlap(boolean)

5. Also I'd upgrade the lollipop to 5.0.1
   There are bugs in 5.0.0 that have been fixed in 5.0.1

This blog by Linton Ye covers in detail the issues surrounding Lollipop transitions and bugs.

My Journey to Lollipop Transitions: part 1