So Frameskipping made it a lot worse. What helped me the most is switching to PortAudio from directsound. I get no more jumpiness and audio skipping behavior even when speed shows dropping when hitting f11.
I guess somehow PortAudio is less taxing when running this driver? I dunno.