In my experience it doesn't matter. I've played the entire time with the leadership xp going. That sucks 20% away from regular xp. Still, at level 61 I'm not far behind everyone else.
It matters a lot on how much you play, etc. Besides, when you reach level 70, it's the same level 70 as everyone else.