I'm in the very same boat and I've thought about doing the very same thing many times. What keeps me from dropping skinning is I don't want to have to buy mats for the times when I do craft something.
If I did a cost/benefit analysis, I'd probably be better off picking up herbing since, like you, I do buy mats (or pots) for raiding consumables.
I have considered stocking up on leathers, then dropping skinning. But, I just haven't been able to bring myself to do it. Part of it is I've been a skinner/LW ever since I first learned professions. I have a sentimental attachment, as well as a kind of role-playing attachment. I don't play on a RP server, but I always like the idea of a hunter being a skinner/LW...it just seems appropriate.
The other aspect for me is that I like to be self-sufficient. I don't like to bother people asking them to craft things for me. I'm sure some people are happy to do it for me, but I just don't like making a habit of it. So, if I were to pick up herbing without taking alchemy, it would put me in a position of being dependent on someone else to craft for me.
Anyway, I feel your pain. I don't have an answer one way or the other unfortunately, since I've been struggling with the same issue for quite some time.