It's a well known fact that integer scaling is not supported by mainstream MAME's Direct3D interface. This is by design.
"Pixel perfect" rendering (or integer scaling as an special case of it) is fortunately still supported, but relegated to the DirectDraw interface. Read the "newvideo.txt" doc for details.
SailorSat created a patch for enabling integer scaling with Direct3D, named -cleanstretch, it's the only solution to the date (CabMAME and GroovyMAME implement this hack).