The 'shadow mask' is applied over the original games resolution right? So if game is 320x240 then 'shadow mask' should be 320x240.
If you set your viewable output resolution to higher than that then HLSL will upscale the picture to your 'increased' chosen resolution afterwards (as a final step/stage).