Your shadow mask settings don't make sense anymore. The x/y count works different since MAME 0.162.
With the u/v size you define the area of the shadow mask texture that will be tiled over the screen. And with x/y count you define how many pixel should be used on screen for this area.