You compile the shaders with the regular MAME build tools and the command line 'make shaders'. The shader binaries are built into /mame/bgfx/shaders/[backend]/[shader].
In addition you need to define some meta information like uniforms, sliders, targets and passes.