I never had any anxiety issues about learning to drive. It was exciting, edifying, liberating, and just plain cool. It was months or even a year after I started driving when the anxiety phase hit me, as I started to appreciate the true risks that come with controlling thousands of pounds of metal at high speeds in confined spaces with countless obstacles. But that anxiety faded, with time and experience--and a few close calls. Everybody, I think, has a few close calls as they're learning.
My advice? Don't fuck around when you're driving. Don't try to impress your friends, don't try to file your taxes, and don't eat. There are better reasons, and ways, to die.