I had to switch to window mode for this to work. For some reason Windows 8 Pro won't take a snapshot from the MAME/MESS full screen.
This is an example.
You see what is doing wrong on the right side? It not scanlines right. You can see the mix of dark and light scanlines. Left side is perfect. This is why I'm using 1280x1024 instead of 1920x1080 to fix most of the problem. Anything past 1280x1024 can't be scale right. Older Arcade games and older systems games aren't made for higher aspects and resolutions. So I lower the settings.
shadow_mask_x_count 320 shadow_mask_y_count 256
is 320x240 on default.
4x 320x256 = 1280x1024 is a perfect fit. If I had a Arcade screen then it should look like the one from the left side.
Index 
Flat 
I will explain here what had happened on my end. (Edit)
Reply
I found away to fix the hlsl scanlines issue.
