Can you post the exact steps you did trying to compile MAMEUI64?
Extract MINGW to a folder. As administrator, run setup-Python.bat and setup-Qt.bat once. Extract the MAME source to a folder. Extract the MAMEUI source overlaying the MAME source. If you did this correctly, you should have a structure something like this: D:\MAMECOMPILE\mame0157s\src\osd\winui\
Navigate to where makefile is. Open a command prompt and type: call D:\MINGW\mingw64-w64\setenv.bat Change to match where YOUR setenv.bat is. then... make all -j5 OSD=winui EMULATOR=MAMEUI64.exe (set the number after -j# to be one more than you have processor cores.
It should compile MAMEUI64.exe with tools. If you don't want the tools, omit all .