How to change animations used by animateLayoutChanges mechanism?

After much digging in the wrong places, found the answer myself. Might help somebody.

animateLayoutChanges property if enabled utilizes an instance of LayoutTransition to do its job in ViewGroup.

So in order to change animations, you can create an instance of LayoutTransition, set Animator for the transition that you need to change and then assign that instance to ViewGroup via setLayoutTransition().

In my case the result is:

Animator scaleDown = ObjectAnimator.ofPropertyValuesHolder((Object)null, PropertyValuesHolder.ofFloat("scaleX", 1, 0), PropertyValuesHolder.ofFloat("scaleY", 1, 0));
scaleDown.setDuration(300);
scaleDown.setInterpolator(new OvershootInterpolator());

Animator scaleUp = ObjectAnimator.ofPropertyValuesHolder((Object)null, PropertyValuesHolder.ofFloat("scaleX", 0, 1), PropertyValuesHolder.ofFloat("scaleY", 0, 1));
scaleUp.setDuration(300);
scaleUp.setStartDelay(300);
scaleUp.setInterpolator(new OvershootInterpolator());

LayoutTransition itemLayoutTransition = new LayoutTransition();
itemLayoutTransition.setAnimator(LayoutTransition.APPEARING, scaleUp);
itemLayoutTransition.setAnimator(LayoutTransition.DISAPPEARING, scaleDown);

ViewGroup av = (ViewGroup)v.findViewById(R.id.animated_layout);
av.setLayoutTransition(itemLayoutTransition);

More info in LayoutTransition reference documentation.