If you use Alt+Enter to toggle windowed mode, can you then use Alt+Tab to switch applications? I can't remember what Windows is supposed to do, but that's the behaviour on OSX (need to use Option-Return to get out of fullscreen before Cmd-Tab will work). MAME could be doing system shortcut capture in full screen.