If you drop your specialty (costs 50g IIRC), you'll lose all specialty-specific patterns, and can pick up the quest to become either specialty. Your Blacksmithing skill doesn't drop.
Note that most TBC specialty gear requires you to be an armorsmith or weaponsmith to equip it as well, so you can't make an item from one specialization and then switch to the other specialization without practically losing the item.