Audiodg.exe doesn't care how *many* audio sources you have, what matters is the amount of software processing it's doing. For games, music, teamspeak, movies, etc this should be next to none. I just opened Crysis while having music, a movie, netflix, and youtube all going at once and it used 2% CPU on my near 5 year old quadcore. If it's using more than a couple percent, something is set up wrong.
Make sure that your audio drivers are up-to-date and that you have all audio "enhancements" turned off. Open your playback devices, select the one you're current using, and under the Enhancements tab click "Disable all sound effects."