Are there any factors that affect how fast a pokemon gains experience on Isle Evelup?

The effect of the drinks increase with Isle Evelup's level. All the following values are per session.

Drink -- Level 1 -- Level 2 -- Level 3

Level Boost -- 100 exp -- 200 exp -- 300 exp
HP Boost -- 1 EV -- 2 EV -- 4EV
Attack Boost -- 1 EV -- 2 EV -- 4EV
Defence Boost -- 1 EV -- 2 EV -- 4EV
Sp. Attack Boost -- 1 EV -- 2 EV -- 4EV
Sp. Defence Boost -- 1 EV -- 2 EV -- 4EV
Speed Boost -- 1 EV -- 2 EV -- 4EV

However, I'm not sure how the other exp effects work in conjunction with Level Boost. So this only answers the base part of your question and the question in your comments.

So I've done my own test on this, and none of the boosts I was interested in - different trainer, Refresh, and Lucky Egg - had any effect. I had four level 1 Littens - one obtained through Wonder Trading, one who had more than 2 heart in Pokemon Refresh, one who was holding a Lucky Egg, and one I used as a control. After subjecting all four Littens to 40+ hours with the Level Up drink, none of the Littens had gained any more experience than the other. All of them were the same level, and even had the same "XP to next level" number. If you want more experience, it looks like you either have to let your pokemon play for more sessions, or just grind for experience the old fashion way.