ilya-v, after reading your second post a second time I think maybe you have got confused with the 'shadow_mask_x_count' and 'shadow_mask_y_count'.
As far as I understand it you set those to what the original game resolution was. IE: 'The Simpsons' is 288[x] by 224[y]
Then set 'hlsl_snap_width' and 'hlsl_snap_height' to your output display resolution. If you were using bezel/border artwork you would instead put your 'aspect ratio display window' resolution here. Then in the bezel/border artwork settings file put your monitors native resolution for example.
If you get these things wrong then it could be what is causing your distortion??