You need to watch them carefully.
It is a random amount of time, with several different factors that influence the duration.
I speak of Chanter and Druid charms Bards and Necros may be different, I know not.
The level of the mob relative to you is a factor, as you already know from your comments.
The inherant magic resistance of the mob is a factor.
Some Enchanters firmly believe that the more times the same mob is recharmed, the more magic resistant it becomes. I am inclined to support this theory based on my experience.
It is also firmly believed by many that your Charisma is a factor in determining the duration, I'm not too sure about this. As I recall, the only clues that SOE have given is that charisma has some effect on how much the mob hates you after charm expires.
In my experience, the mob has always hated me for the maximum possible amount every time charm has expired, so, not too sure about that one either.
The only proven fact (as far as I am aware) is that of MR, if you debuff MR successfully you will usually get the charm to stick for its maximum duration - but not always.
There is also an unwritten law (or is that lore?), charm will always fail just at the most dangerous moment. Usually just as the second add arrives in camp, so that you have the angry ex pet and at least two of his friends beating on you at the same time.