The HLSL shader of MAME does not have the goal to upscale or smooth pixel graphics. The shader tries to simulate the appearance of a CRT or vector monitor.
To get upscaled and smoothed pixel graphics use HqMAME which supports different xBR and HQx filters.