What you describe is how it happens to everyone: magic does slide through you, and disappear, and come back later looking like something else. And I'm sorry to tell you this, but where your magic lives will always be a great dark space with scraps you fumble for. You must learn to sniff them out in the dark.