The latest MAME 0.171 GIT (2-8-2016) works just fine for me on Windows XP 32-bit.
D3D works fine.
It may be your .ini settings. The latest MAME GIT release now creates two .ini files in the root directory... mame.ini and ui.ini. As well as info_mame.ini in the UI directory.
EDIT: OpenGL isn't even an option that's listed to choose from in my MAME.
When I choose BGFX and try to Return to Previous Menu, MAME exits.