|
bit of a speedup experiment
#359784 - 10/21/16 08:47 AM
|
|
|
Trying a bit of an experiment.
While looking at something else I noticed in procmon that MAME has an interesting habit of reopening files that do not exist. Now on a local file system this is probably not too bad. But on a network filesystem it is sort of bad.
Current results are kind of promising.
I added in a std::map to cache the negative 'no file found' results returned from open_funcs[ i](m_fullpath, zip) in the fileio.cpp file in the attempt_zipped method. Just bashed it in as a global for a quick test. Would be nicer to make it part of the class.
I fed in '-verifyroms pacman' for the parameters
It went from 2574 file events most of which were negative to 1251 file events. A good portion of the remaining were dll loads and mame.ini reads (which oddly it seems to do twice).
From my testing with '-verifyroms pacman*' (with wildcard) I am getting about 9 to 15 seconds faster reads fairly consistently. That is with wall clock time tests. Would need to add a bit of logging to get it more exact. At least for my network.
Now obviously the best case would just to be to run the whole thing from a local machine. Do not have the room at the moment. I suspect it would still help but not as dramatically.
Just for an interesting test I did a drag race between the two methods for verifyroms. The old way got to 12 roms before I killed it. The new one got through 76 roms before I killed. Killed them within 1-2 seconds of each other. Not bad and noticeably faster. Individual game load does not seem to have much difference at all.
Downside to this method is it probably just looks straight up like a memory leak with an ever growing map. Though I have not noticed much in memory manager. Another downside would be if someone starts the prog up and then afterwords drops files in. It would think the files do not exist until restart. Also it probably would be better to run down why it is trying to open paths that do not exist in the first place. But I was a bit bored and have been meaning to try this for a few weeks.
Edited by lharms (10/21/16 08:49 AM)
|
|
|
|
Re: bit of a speedup experiment
[Re: lharms]
#359785 - 10/21/16 02:03 PM
|
|
|
Great.
...does it mean I'll get an extra free 10fps in Naomi based games?
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9716
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: bit of a speedup experiment
[Re: lharms]
#359787 - 10/21/16 05:08 PM
|
|
|
mame.ini is read twice to allow the first found mame.ini to set the inipath to something different and have it take effect.
|
|
|
|
Re: bit of a speedup experiment
[Re: Ziggy100]
#359815 - 10/22/16 08:12 AM
|
|
|
> Great.
>
> ...does it mean I'll get an extra free 10fps in Naomi based games?
If naomi games open and close their file handles a lot (which I doubt they do), it would be slightly slower. Otherwise about the same
But it gives a nice boost on verify.
|
|
|
|
Re: bit of a speedup experiment
[Re: R. Belmont]
#359816 - 10/22/16 08:19 AM
|
|
|
> mame.ini is read twice to allow the first found mame.ini to set the inipath to
> something different and have it take effect.
Interesting choice. Did not changing it on the fly work out as good or other reasons? Was it too far apart from where the inipath was needed? Or just a leftover from an older part of the design?
You peaked my curiosity. I am interested in choices people make on design, sometimes  I am not saying it is bad or good. I personally would have went with a buffered read and a two pass across the buffer. But that has drawbacks too. Sometimes seeing why others come up with their choices is an interesting way to learn something new.
|
|
|
|
Re: bit of a speedup experiment
[Re: lharms]
#359820 - 10/22/16 04:12 PM
|
|
|
> > mame.ini is read twice to allow the first found mame.ini to set the inipath to
> > something different and have it take effect.
>
> Interesting choice. Did not changing it on the fly work out as good or other reasons?
> Was it too far apart from where the inipath was needed? Or just a leftover from an
> older part of the design?
>
> You peaked my curiosity. I am interested in choices people make on design, sometimes
> I am not saying it is bad or good. I personally would have went with a buffered read
> and a two pass across the buffer. But that has drawbacks too. Sometimes seeing why
> others come up with their choices is an interesting way to learn something new.
First a minor gripe, your curiosity gets piqued, it doesn't get "peaked". For your curiosity to be peaked, you must literally be experiencing the most amount of curiosity you've ever had in your life.
With that out of the way, I think you're not understanding Belmont's answer properly. If you set the inipath option to point to, say, c:\foo\bar\, and you put a copy of mame.ini there, and you ran your test, you'll find (I assume) that the mame.ini in the same directory as the executable is read only once, at which point it reads the one in the location specified by the base ini's inipath. You're seeing the base ini get read twice because you haven't changed your inipath setting and don't have any other ini files to be read.
|
|
|
|
Re: bit of a speedup experiment
[Re: MooglyGuy]
#359831 - 10/23/16 02:16 AM
|
|
|
|
|
| R. Belmont |
|
Cuckoo for IGAvania
|
|
|
|
|
|
Reged: 09/21/03
|
|
Posts: 9716
|
|
Loc: ECV-197 The Orville
|
|
|
|
Send PM
|
|
|
Re: bit of a speedup experiment
[Re: MooglyGuy]
#360165 - 11/02/16 05:01 PM
|
|
|
> With that out of the way, I think you're not understanding Belmont's answer properly.
Correct. On a default install, it'll read ./mame.ini, set that inipath (which will be .) and then read mame.ini from the new path (which is the same as the old path). Nothing special happens then
Whereas if you've got a system-installed version of MAME on Linux in /usr/bin and a mame.ini in /etc/mame that sets inipath to $(HOME), it'll first read /etc/mame/mame.ini, then set the inipath to the user's home directory, and read mame.ini from there. That way each user has their own personalized settings, roms, artwork, etc. (You could do something similar on Windows 7 or later but it'd require a little bit of support from MAME, and nobody installs Windows multi-user).
|
|
|
|
|
|
|
|
Re: bit of a speedup experiment (with verify)
[Re: lharms]
#360795 - 12/01/16 08:49 AM
|
|
|
> From my testing with '-verifyroms pacman*' (with wildcard) I am getting about 9 to 15
> seconds faster reads fairly consistently. That is with wall clock time tests.
>
Fiddled a bit more with this. Not sure of the side effects so I have not created a patch. I suspect it will break something
So I changed from std::map to std::unordered_map. Not much difference. I told the unordered map to start with a 64k buffer size instead of 0. That beats the std::map by 2 seconds (~11-17 seconds faster). So kinda meh and expected...
I know my files are 7z. So I flipped 7zip and zip order on trying to find files. No real noticeable change. So 'meh'.
Then I went back to procmon and found some more stuff to see if I could filter out.
My 'roms' setting in my ini file has 4 folders. That does seem to be much slower on 'file miss'. As it basically searches through all 4 folders for individual files and both 7z and zip. As expected and designed.
Now I dug a bit deeper. I changed osd_file::error osd_file::open in winfile.cpp to cache ERROR_PATH_NOT_FOUND errors. Now this has got me another decent speed up. This is because of the way it searches for individual files. It looks into a folder of the set name. But on a miss that folder will not exist and the next set of calls will also not be there because the path is non existent. So now I am seeing consistently 33-36 seconds faster on the verify test from above. Have not isolated it from the above change to see what it would do. Would need to dig into the unix ver to see if it could do the same thing. Had to do it at this level as this call eats the path not found error and turns it into a util::archive_file::error::FILE_ERROR. So from a level above I can not tell the difference and I need to for this to work right.
Was a bit hacky to get right as that method can also create directories with the right flags passed in. Have to see if I can simplify it.
Overall not too bad from 59 seconds to verify to 26 seconds on my last run. I also did a race between orig and my 2 new ways with just verifyroms. The orig got through the 'ones' the new one was pretty almost through the 'Cs' before I just killed it.
tl;dr? Opening things repeatedly that do not exist on a network share slows things down.
|
|
|
|
Re: bit of a speedup experiment
[Re: lharms]
#362198 - 01/04/17 01:04 AM
|
|
|
With a bit of nudging from Vas I dug a bit deeper. I think it is a bug in part_of_mediapath.
The if condition is this
if (path.compare(mediapath.substr(0, mediapath.length())))
Probably should be this
if (mediapath.compare(path.substr(0, mediapath.length())) == 0)
It is comparing the wrong parts of the string and not checking for the 0 result. I think the intent is to keep it to the rompath dirs. However it is pretty much always letting it through.
This is as fast as my cache way. There are a couple of other minor tweaks that could be made. But those do not do anything significant.
The second cache I added breaks the ability to add files/dirs randomly. It is about 8x faster but I am going to leave it as experimental. Samba shares are apparently really bad if you have a path_not_found miss.
|
|
|
Index 
Threaded 
bit of a speedup experiment
Reply
[Re: 
I am not saying it is bad or good. I personally would have went with a buffered read and a two pass across the buffer. But that has drawbacks too. Sometimes seeing why others come up with their choices is an interesting way to learn something new.
