So yeah, since MAMEPGUI is not supported by the MAME team and since I can avoid the error box with certain HLSL settings. That would indicate that perhaps it is something in the HLSL code, which would then be obvious to post a thread on the MESS board where the HLSL developer could respond.