Edit mame.ini in the /ini folder and make sure to have at least these two lines:
hlsl_enable 1 hlslpath hlsl
then open up a game and go to SLIDER OPTIONS and you should see it's a pretty huge window now. If that doesn't work, try the command line Mame: mame.exe -hlsl mk2
If you don't have a mame.ini, run mame64.exe -cc and it'll poop one out.