MAMEWorld >> EmuChat
View all threads Index   Threaded Mode Threaded  

Pages: 1

ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Making new hash files for MAME (ZX Spectrum)
#381263 - 02/27/19 11:22 AM


So, in anticipation of the future improvements in Z80 emulation, I've finally converted (after a really long night + morning ) the latest TOSEC DAT for Spectrum +3 games to MAME's hash file format.

Right now, the entries look like this:

(yeah, maybe I should remove all those redundant "(+3)")

Now all that's left is to add a unique "name" field for the repeated entries, selectively replace some instances of "&" with "and" (except for those in the "description" fields), and I guess trim some of those really long names.

So I was wondering, what's the recommended maximum character length in this field? Among the 1000+ names, there's things like:

ace2theultimateheadtoheadconflict
actionforceiiinternationalheroes
adidaschampionshiptiebreak
daleythompsonsolympicchallenge


Also, there's around 300 entries with names like "Title of the Game, The". Those should have their description converted to "The Title of the Game", right?


EDIT: If somebody can make a script for doing those things automatically, it will come in real handy when adding the 20000+ Spectrum cassettes.

Edited by ICEknight (02/28/19 12:30 AM)



Haze
Reged: 09/23/03
Posts: 5245
Send PM


Re: Submitting MAME hash files new [Re: ICEknight]
#381264 - 02/27/19 11:35 AM


> So, in anticipation of the future improvements in Z80 emulation, I've finally
> converted (after a really long night + morning ) the latest TOSEC DAT for Spectrum +3
> games to MAME's hash file format.
>
> Right now, the entries look like this:
>
> (yeah, maybe I should remove all those redundant "(+3)")
>
> Now all that's left is to add a unique "name" field for the repeated entries,
> selectively replace some instances of "&" with "and" (except for those in the
> "description" fields), and I guess trim some of those really long names.
>
> So I was wondering, what's the recommended maximum character length in this field?
> Among the 1000+ names, there's things like:
>
> ace2theultimateheadtoheadconflict
> actionforceiiinternationalheroes
> adidaschampionshiptiebreak
> daleythompsonsolympicchallenge
>
>
> Also, there's around 300 entries with names like "Title of the Game, The". Those
> should have their description converted to "The Title of the Game", right?
>
>
> EDIT: If somebody can make a script for doing those things automatically, it will
> come in real handy when adding the 20000+ Spectrum cassettes.

it is recommended to keep setnames to something 'short to type' as they're designed for commandline use.. so something like "daleythompsonsolympicchallenge" would be considered ridiculous. It should be "daleyoc" or similar

also, you don't need to duplicate the company names etc. in the description fields unless there is potential for them to be ambiguous.



F1ReB4LL
Reged: 03/09/05
Posts: 332
Send PM


Re: Submitting MAME hash files new [Re: ICEknight]
#381269 - 02/27/19 02:02 PM


> So, in anticipation of the future improvements in Z80 emulation, I've finally
> converted (after a really long night + morning ) the latest TOSEC DAT for Spectrum +3
> games to MAME's hash file format.

Shouldn't MAME primarily use the more precise SPS/No-Intro dumps and only take the missing ones from TOSEC?



Pernod
Lurker
Reged: 01/12/04
Posts: 82
Loc: UK
Send PM


Re: Submitting MAME hash files new [Re: ICEknight]
#381272 - 02/27/19 02:56 PM


>
> So I was wondering, what's the recommended maximum character length in this field?
> Among the 1000+ names, there's things like:
>
> ace2theultimateheadtoheadconflict
> actionforceiiinternationalheroes
> adidaschampionshiptiebreak
> daleythompsonsolympicchallenge
>
>
> Also, there's around 300 entries with names like "Title of the Game, The". Those
> should have their description converted to "The Title of the Game", right?
>
>
> EDIT: If somebody can make a script for doing those things automatically, it will
> come in real handy when adding the 20000+ Spectrum cassettes.

The maximum short name length is usually 8, but not enforced, only exceed if you really need to, ie. for clones. Alot of Spectrum games were released on other machines so try to match names used in other software lists for the same game.
Your examples would reduce to:
ace2
aforce (already in specpls3)
tiebreak (already in specpls3)
daleyoc

There is no magic script to trim the short names, each requires individual attention.

Your descriptions should be the title only, ie. ACE 2 - The Ultimate Head to Head Conflict. Only add to this if needed to identify regional clones, re-releases, etc.

If you intend to tackle the spectrum_cass list then you should be familiar with what you are adding, don't blindly add everything. Do we really need over 40 versions of Jet Set Willy? There are lots of hacks in there so you need to determine whether it's worthy of inclusion, and marked appropriately.



Sthiryu
MAME Fan
Reged: 03/09/16
Posts: 117
Send PM


Re: Submitting MAME hash files new [Re: F1ReB4LL]
#381279 - 02/27/19 06:04 PM


I think the most precise files are those from World Of Spectrum, better than TOSEC and No-Intro. They cover cassete files in tzx and tap format (Last set from No-Intro only covered disk, if I remember well).



RETRODANUART.COM



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: F1ReB4LL]
#381283 - 02/27/19 11:39 PM


> Shouldn't MAME primarily use the more precise SPS/No-Intro dumps and only take the
> missing ones from TOSEC?

The IPFs from SPS are already in the softlist except for LED Storm, Rock Star Ate My Hamster and Tiger Road (which nobody seems to have ever seen). Thing is, they're just 130 so I wanted to add the existing (almost 2000) DSK files.

About the duplicates, I was also wondering how to handle them since I don't think that anybody has verified each IPF against the DSK that they supposedly correspond to, so it may not be safe to just remove the DSKs that match the title as they could have slightly different contents... So maybe the DSKs with "matching" IPFs should just be labeled as "alt"?




> I think the most precise files are those from World Of Spectrum, better than TOSEC
> and No-Intro. They cover cassete files in tzx and tap format (Last set from No-Intro
> only covered disk, if I remember well).

The TOSEC sets now include all kinds of files (No-Intro's just has the SPS IPFs which MAME's softlist already covers). They seem to have updated them with most (all?) of the World of Spectrum contents, last year.

Regarding cassette files, TZX are basically the same as TAPs only more accurate, and they've been the way to go for preservation since forever... So I think that, save for the few exceptions that somebody may find, only TZX should be considered for any future Spectrum cassette softlists.




> If you intend to tackle the spectrum_cass list then you should be familiar with what
> you are adding, don't blindly add everything. Do we really need over 40 versions of
> Jet Set Willy? There are lots of hacks in there so you need to determine whether it's
> worthy of inclusion, and marked appropriately.
> There is no magic script to trim the short names, each requires individual attention.

Yeah, but scripts could be made for other stuff that could simplify work greatly, such as:
-Removing any entries with "[t]", "[h", etc in the filename (which gets rid of the thousands of Jet Set Willy hacks altogether)
-Converting games with many sides/disks to one proper entry with all of them.

(Note: I didn't say I'm doing the cassettes, I may need a couple more lifetimes to get those done without help!)



I've attached some yet-to-be-polished XMLs with every single DSK present in the most recent TOSEC DATs, including all categories, games, magazines, demos, etc (1726 entries total). EDIT: Removed (outdated)

This is how they look right now:



Stuff that's yet to be done:
-"name" field: Make some of them more descriptive
-"name" field: Add trailing characters for duplicate names (when they're alt sets)
-"description" field: Add "(set 2)", "(set 3)", etc, to entries with the same name but which don't have more disks/sides as the following entries.
-"description" field: Move any trailing instances of ", The", ", El", ", La", etc, to the start of each
-Remove any entries with "[t]", "[h*]", "[b*]", "[cr ", etc, in the "filename" field
-Merge entries with many sides/disks into the same one

The first one has to be done case by case, but if the others could be achieved via script, things would be sped up significantly (removing trainers and mods would be of much more help with the cassettes softlist).



EDIT: Updated, now all "name" fields have 8 characters, which obviously brings many duplicates to the XML.

I'm currently using Notepad++, if you know of any tool which could make looking for duplicates or just editing these entries easier, please let me know.

Edited by ICEknight (03/01/19 07:07 PM)



Vas Crabb
BOFH
Reged: 12/13/05
Posts: 4462
Loc: Melbourne, Australia
Send PM


Re: Submitting MAME hash files new [Re: Pernod]
#381284 - 02/28/19 02:09 AM


> The maximum short name length is usually 8, but not enforced, only exceed if you
> really need to, ie. for clones. Alot of Spectrum games were released on other
> machines so try to match names used in other software lists for the same game.
> Your examples would reduce to:
> ace2
> aforce (already in specpls3)
> tiebreak (already in specpls3)
> daleyoc

No, the maximum is 16 for systems and software items (and 32 for devices), and it is enforced. As a guideline though you should try to fit the parent set's shortname in 8 characters if you can without making it incomprehensible. Clone set shortnames should be made by appending suffixes to the parent set's shortname.



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: ICEknight]
#381286 - 02/28/19 09:05 AM


Ok, so I've done all +3 categories except for "Games" and "Compilations - Games". Please let me know if you notice anything incorrect in the attached files.

EDIT: removed (outdated)

Edited by ICEknight (03/02/19 07:06 AM)



Vas Crabb
BOFH
Reged: 12/13/05
Posts: 4462
Loc: Melbourne, Australia
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: ICEknight]
#381288 - 02/28/19 10:07 AM


> Ok, so I've done all +3 categories except for "Games" and "Compilations - Games".
> Please let me know if you notice anything incorrect in the attached files.

Look, I'm sorry but I tend to agree with Pernod - trying to do this automatically is going to get us a pile of unsorted stuff of indeterminate quality that's going to be more time-consuming to clean up properly later.



Haze
Reged: 09/23/03
Posts: 5245
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: Vas Crabb]
#381292 - 02/28/19 11:54 AM


> > Ok, so I've done all +3 categories except for "Games" and "Compilations - Games".
> > Please let me know if you notice anything incorrect in the attached files.
>
> Look, I'm sorry but I tend to agree with Pernod - trying to do this automatically is
> going to get us a pile of unsorted stuff of indeterminate quality that's going to be
> more time-consuming to clean up properly later.

I disagree, just based on the amount of data.

It's literally impossible for one person to do all this work, at least if you're looking at the cassette side of things; there are more tapes than somebody could actually load up and properly test in a lifetime.

The only way it's going to get done is if somebody does bulk convert a lot of it, then a bunch of people refine it over time, replace dumps with better ones, document all the different loader schemes used etc. It's a lifetime project for a whole group of interested people.

It *is* an issue if nobody continues the work by doing that refinement after the initial check-in, freeing the original devs to do other things (as no main dev really wants to commit their entire life to doing one thing that's easily done by non-programmers) but that's kinda where the supporting community around MAME do end up letting us down sometimes, as those are tasks that almost anybody could do.

The disk situation is a bit easier and could be handled by hand as disks weren't the primary platform here, plus 3 wasn't too common, and hardly anybody had a drive for the older model, but I'm looking at the bigger picture here.

Something does need doing in terms of lists for these systems tho, there is Software being developed today (tested on real systems) and lost just as quickly because nobody is documenting it, revisions of it, or anything else. It needs to be made clear that the lists here are open for contribution, and that attempts to contribute to them aren't going to be met with hostility at least.

To get people interested in refinements the Spectrum emulation probably needs to be a lot more solid however, that patch posted on bannister.org did do wonders for that, but the way it was implemented was definitely questionable. MAME also faces competition from things that simply HLE all the loading, which is a problem I don't think much can be done about, but does make it less favourable to people which might mean less people want to help out.



Pernod
Lurker
Reged: 01/12/04
Posts: 82
Loc: UK
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: ICEknight]
#381294 - 02/28/19 03:05 PM


> Ok, so I've done all +3 categories except for "Games" and "Compilations - Games".
> Please let me know if you notice anything incorrect in the attached files.

Applications:
- Re-releases should be tagged as clones, ie. software name="3dgamemkdro" cloneof="3dgamemk"
- Where multiple disks/sides are included then specify a part_id for each, ie. feature name="part_id" value="Side A"/. When selecting items with long filenames in the internal UI the useful info like (side a).dsk is usually truncated. Specifying a part_id will let the UI offer you either 'Side A' or 'Side B', these should match disk labels if known.
- alkdevlo: the side b image is only 256 bytes, really?
- edduced: filename suggests it's a master disk, maybe worth adding to the description.
- lifeguar: filename suggests it needs a multiface, so tell the user this with info name="usage" value="Requires Multiface"
- revelado: if it's Spanish language then add (Spa) to description.
- Do we really need all these [aka ...] comments in the filenames?

Compilations-Applications:
- unknown years should be 19??, I believe 19xx will fail validation.
- Re-order names, ie. Lancaster, Gary -> Gary Lancaster.

Demos:
If a publisher is unknown then enter as unknown, or in this case homebrew may be more appropriate.

Educational:
- funsc268: why all the alts? We want to document best available. I'd just keep the first, which matches what WoS lists.

All:
- No need to include comment for each item indicating source, ie. remove . If you want to credit a source then add it to the file header.

These are just quick comments, and are not meant to put you off, but I see little to no quality control being applied to these additions. Anyone can blindly add TOSEC collections but MAME software lists are supposed to be definitive collections.



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: Vas Crabb]
#381301 - 02/28/19 09:43 PM


> trying to do this automatically is going to

All the automatic stuff has already been done for converting all the needed info from the documented floppy images to MAME's hash file format, I'm going through all the entries one by one now...



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: Haze]
#381305 - 02/28/19 10:07 PM


> It's literally impossible for one person to do all this work, at least if you're
> looking at the cassette side of things; there are more tapes than somebody could
> actually load up and properly test in a lifetime.
>
> The only way it's going to get done is if somebody does bulk convert a lot of it,
> then a bunch of people refine it over time, replace dumps with better ones, document
> all the different loader schemes used etc. It's a lifetime project for a whole group
> of interested people.

Thank you. Even though I don't think much refinement will be needed for the +3 softlist, the cassette list is very likely going to be something else, indeed.
Right now there's just a few cassettes added and many of them seem to be the same release in a different file formats (TZX and TAP), which isn't optimal either even if it's been done manually without a batch conversion. I think the way to go would be to add all known TZX files (current TOSEC) and work from there, adding new releases and any TAP exclusives that may be found.


> To get people interested in refinements the Spectrum emulation probably needs to be a
> lot more solid however, that patch posted on bannister.org did do wonders for that,
> but the way it was implemented was definitely questionable. MAME also faces
> competition from things that simply HLE all the loading, which is a problem I don't
> think much can be done about, but does make it less favourable to people which might
> mean less people want to help out.

Well, I was told the other day that somebody was working on the cycle-by-cycle Z80 emulation so there's hope for that. I just want to have a good fraction of the system's library available right away when it happens.

I'm currently streaming Spectrum games using MAME on weekends and still get the occasional comments of people who didn't have a clue that MAME could emulate computers, but I can see some interest very slowly building up. Of course I can only show them old games that don't do any fancy tricks yet (I wish I could show them games like Cobra, but that will have to wait).



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: Pernod]
#381312 - 02/28/19 10:58 PM


> - Re-releases should be tagged as clones, ie. software name="3dgamemkdro"
> cloneof="3dgamemk"
> - Do we really need all these [aka ...] comments in the filenames?

Thing is, I had initially removed all those comments from the filenames... but then I realized that not only other softlists keep them but, when going through all these entries, they prove incredibly useful for adding clone info. The AKAs seem to be there whenever a clone has a different title than the original, which would be just crazy to figure out otherwise!
Once they're not needed I guess they could be renamed to "a.dsk", "b.dsk", 2a.dsk", etc, but they're good for reference in the meanwhile.


> - Where multiple disks/sides are included then specify a part_id for each, ie.
> feature name="part_id" value="Side A"/. When selecting items with long filenames in
> the internal UI the useful info like (side a).dsk is usually truncated. Specifying a
> part_id will let the UI offer you either 'Side A' or 'Side B', these should match
> disk labels if known.

Thanks, I was wondering about this but couldn't find it in the Amstrad CPC softlist which I was using for reference.
Would it be fine if that line was added by default to all the images that only have one side preserved (those that don't specify they're including both sides within the same file)? I mean, it would be technically correct and it would also save a lot of hours.


> - alkdevlo: the side b image is only 256 bytes, really?

Downloaded it from WoS to check and:

Looks like it.


> If a publisher is unknown then enter as unknown, or in this case homebrew may be more
> appropriate.

I have a feeling that more or less half of the known Spectrum games to this day may be non-commercial which would label them as "homebrew", so I think that "unknown" would be a better term to use in this case.


> Educational:
> - funsc268: why all the alts? We want to document best available. I'd just keep the
> first, which matches what WoS lists.

Those three alts could be actually redumps that fix some outdated images in World of Spectrum (which has been in a crumbling state for some years now). These are the kind of softlist entries that I (or anybody else) could investigate one day out of boredom and find out which of them is the correct one and which are just bad dump attempts... But until then, it may be a good idea to keep them since there may be a reason for their existance.


> All:
> - No need to include comment for each item indicating source, ie. remove . If you
> want to credit a source then add it to the file header.

I just added those because I saw that all IPFs had that line identifying their SPS release, but I also think that it could be useful for anybody who wants to contribute, so that they know there's no need to go through that DAT anymore and can work on adding the games in later TOSEC updates.

Edited by ICEknight (02/28/19 11:59 PM)



Pernod
Lurker
Reged: 01/12/04
Posts: 82
Loc: UK
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: ICEknight]
#381322 - 03/01/19 02:29 PM


>
> Those three alts could be actually redumps that fix some outdated images in World of
> Spectrum (which has been in a crumbling state for some years now). These are the kind
> of softlist entries that I (or anybody else) could investigate one day out of boredom
> and find out which of them is the correct one and which are just bad dump attempts...
>

We appreciate your enthusiasm but we're now looking at another actively maintained collection that could be preferred over TOSEC. So for now I'd like to put you on hold. You've at least prompted some movement on the subject.



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: Pernod]
#381323 - 03/01/19 06:17 PM


> We appreciate your enthusiasm but we're now looking at another actively maintained
> collection that could be preferred over TOSEC.

The TOSEC set has been updated last year with all (or most) of the software in World of Spectrum, but with all the needed info for the MAME entries in their file name rather than the random, non-standarized names in WoS. What alternative are you looking for?


>So for now I'd like to put you on
> hold. You've at least prompted some movement on the subject.

I've been working on fixing the lists since last post and, except for the parent/clone relationships, they're almost finished already (correct titles and people names and all) and I even have started working on the cassettes by now, getting those ready for a manual case by case revision.

You really want me to stop in favor of... nothing?



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: Making new hash files for MAME (ZX Spectrum) new [Re: ICEknight]
#381327 - 03/02/19 06:45 AM


Just leaving this here so that it won't get lost... (I just had to continue after seeing the alternative)
EDIT: Removed (outdated)

Lots of work done, testing disk files, downloading equivalents with different names, checking websites and even looking at eBay listings (I even found something inside a compilation which wasn't listed at WoS). So right now the "Games" list is still WIP, but others (including "Compilations - Games", which probably includes most of the system's catalogue) may be fine already.

Edited by ICEknight (03/03/19 10:07 AM)



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


New specpls3_flop softlist. new [Re: ICEknight]
#381343 - 03/03/19 10:16 AM Attachment: specpls3_flop TOSEC.zip 247 KB (5 downloads)


Ok guys, I think it's done.

Had to check and download lots of stuff and found a few weird things in the process (such as how many of those "master disks" from a certain company share the same file for Side B in the World of Spectrum archives).

The DSK files which already have an IPF file are now (alt), in case that they may not come from the same version (and have a comment explaining this).

Gun Stick-ready games and password protected ones have a "usage" label.

Many of the games even share names with their arcade versions (except those whose IPF already had a different name).

Oh yeah, and found some 80-track floppy images in the TOSEC set which don't belong in this softlist, so I put them in an "extra" file.


May need some polishing, but I'm a bit tired right now.



Haze
Reged: 09/23/03
Posts: 5245
Send PM


Re: New specpls3_flop softlist. new [Re: ICEknight]
#381344 - 03/03/19 01:40 PM


> Ok guys, I think it's done.
>
> Had to check and download lots of stuff and found a few weird things in the process
> (such as how many of those "master disks" from a certain company share the same file
> for Side B in the World of Spectrum archives).
>
> The DSK files which already have an IPF file are now (alt), in case that they may not
> come from the same version (and have a comment explaining this).
>
> Gun Stick-ready games and password protected ones have a "usage" label.
>
> Many of the games even share names with their arcade versions (except those whose IPF
> already had a different name).
>
> Oh yeah, and found some 80-track floppy images in the TOSEC set which don't belong in
> this softlist, so I put them in an "extra" file.
>
>
> May need some polishing, but I'm a bit tired right now.

This does not constitute a submission as it stands, you need to make a Pull Request over at Github for it to be considered
https://github.com/mamedev/mame/pulls



Pernod
Lurker
Reged: 01/12/04
Posts: 82
Loc: UK
Send PM


Re: New specpls3_flop softlist. new [Re: ICEknight]
#381345 - 03/03/19 02:50 PM


As Haze has already mentioned, you need to create a PR in GitHub for it to be officially considered. In it's current state I will reject it though.

You've ignored my advice to hold off for now whilst another preferred source is considered, one that has been verified, unlike TOSEC.
You also ignored my suggestion to remove various alts that have been checked by Wos/TZX Vault and removed from their collections.
You've included some personal collections in there from Jesus Tejero, Javier Herrera. These are not appropriate and should be removed.

I can see you've spent considerable time on this but your insistence on using TOSEC as the source is not the best use of anyone's time.
I'd rather have a softlist that can be added to rather than one that someone else (me) will have to clean up. The likes of Wos/TZX Vault/SPA2 have already done the filtering for us, use their experience.



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: New specpls3_flop softlist. new [Re: Pernod]
#381363 - 03/04/19 03:49 PM


> I'd rather have a softlist that can be added to rather than one that someone else
> (me) will have to clean up. The likes of Wos/TZX Vault/SPA2 have already done the
> filtering for us, use their experience.

Allow me to explain why, due to my experience with Spectrum dumps, I think it's important not to remove all the alternate versions which are not currently in World of Spectrum.

I used to own a website in the late 90s where I steadily released dumps from my Spectrum collection (many by-then-MIAs included) and I came across an odd version of Manic Miner. By then, somebody kept submitting my releases to WoS, but I noticed that this specific dump wasn't there even though many of the submitted releases from my site were... so I just assumed that it was a bad dump or that my cassette had been tampered with. Many years later, I got another copy of the same revision and it matched that dump (it had the same peculiarities, down to a certain master error).

So right now, there's an alternate version of Manic Miner (the most iconic game in the ZX Spectrum library) missing in World of Spectrum, because somebody's common sense just decided that it was a "bad dump". Stuff like this is why I'm insisting on keeping the alts until it's somehow proved that they're just worse versions of the preferred ones. Nobody's going to start adding alternate revisions when those not in WoS are presumed "bad" (even when they've never been labeled as such).

Another reason (from my experience) for me to be so cautious is that, a few years ago, I released another dump of a game which had no TZX available, which, as of today, still shows up as "Missing In Action" at WoS... and also, no TZX of it can be found at SPA2 (they have a really big backlog of submitted games to upload since 2010). Those sites seem to have been in a zombie state for some years now and it may not be a good idea to consider them up to date, specially when discarding existing dumps only because they can't be found in there.

I'll gladly go through the DSKs from the TZX vault and make sure that they're not missing, if I can get ahold of the 38 "Distribution Denied" files and the 19 which gave a 404 error (or their checksums)... But also, I'd really like to know what kind of methods are being used to compare DSK files and decide which one's best. If there's an infallible tool for that, it could prove really useful here. Else, we can only cross our fingers that we're not already losing some rare releases and prototypes already.


Until we're really able to sort out the alt dumps, I think that we should focus on the improvements that can be made without actually risking losing any truly valuable images.


> You've included some personal collections in there from Jesus Tejero, Javier Herrera.
> These are not appropriate and should be removed.

Yeah, I was thinking of labeling those as "<homebrew>", but they can be removed if you guys don't think they're worth it. There's another collection of 48K games ("MicroByte - Serie Clasicos Spectrum") ported to disk by fans which may also have to be removed.

Just like the trained games, alt files specifically labeled as "bad dumps" have already been removed, except for one 4-in-1 pack which didn't have any other dump available (all these things are explained in the comments inside the XMLs). Also removed are the DSK files generated with the ZXZVM interpreter and the games from the "Crap Games Competition", for obvious reasons.

Anything else you feel that should be tweaked before submitting, please let me know.


...And speaking of tweaks, I've noticed a lack of Spanish accents and symbols in the titles, does MAME support them right away or do they have to be added as "&aacute;", etc?



Haze
Reged: 09/23/03
Posts: 5245
Send PM


Re: New specpls3_flop softlist. new [Re: ICEknight]
#381364 - 03/04/19 04:17 PM


> > I'd rather have a softlist that can be added to rather than one that someone else
> > (me) will have to clean up. The likes of Wos/TZX Vault/SPA2 have already done the
> > filtering for us, use their experience.
>
> Allow me to explain why, due to my experience with Spectrum dumps, I think it's
> important not to remove all the alternate versions which are not currently in World
> of Spectrum.
>
> I used to own a website in the late 90s where I steadily released dumps from my
> Spectrum collection (many by-then-MIAs included) and I came across an odd version of
> Manic Miner. By then, somebody kept submitting my releases to WoS, but I noticed that
> this specific dump wasn't there even though many of the submitted releases from my
> site were... so I just assumed that it was a bad dump or that my cassette had been
> tampered with. Many years later, I got another copy of the same revision and it
> matched that dump (it had the same peculiarities, down to a certain master error).
>

I can believe this, I know there are Speccy games I had as a kid that had certain bugs, I even took them back and got replacements, but the replacements had the same bug on my machine, same bug on my friend's machine (and yes, they were meant to be compatible with our models) I've not seen those bugs in emulated versions, regardless of machine selected, so I'm assuming that it was just decided that they were 'bad' and scrapped / never documented, but I'm also inclined to believe they were probably from a badly mastered batch instead.

> Another reason (from my experience) for me to be so cautious is that, a few years
> ago, I released another dump of a game which had no TZX available, which, as of
> today, still shows up as "Missing In Action" at WoS... and also, no TZX of it can be
> found at SPA2 (they have a really big backlog of submitted games to upload since
> 2010). Those sites seem to have been in a zombie state for some years now and it may
> not be a good idea to consider them up to date, specially when discarding existing
> dumps only because they can't be found in there.
>

Right, everywhere seems to have a backlog, that's one reason I think a system like the Software Lists is good, it's open to anybody submitting a Pull Request and adding to it at any point. As I said, there seems to be a severe problem with documenting more recent releases, they're simply not being covered as they're dropping off the internet before being added anywhere else, or only being noticed after a handful of versions, with the original releases being lost.

> I'll gladly go through the DSKs from the TZX vault and make sure that they're not
> missing, if I can get ahold of the 38 "Distribution Denied" files and the 19 which
> gave a 404 error (or their checksums)... But also, I'd really like to know what kind
> of methods are being used to compare DSK files and decide which one's best. If
> there's an infallible tool for that, it could prove really useful here. Else, we can
> only cross our fingers that we're not already losing some rare releases and
> prototypes already.
>

I've noticed people often ended up using 'worse' format images for some of this stuff due to that, which is kinda silly when the end result is the same. Hopefully there won't be any politics with the Softlist like that.

> Yeah, I was thinking of labeling those as "", but they can be removed if you guys
> don't think they're worth it. There's another collection of 48K games ("MicroByte -
> Serie Clasicos Spectrum") ported to disk by fans which may also have to be removed.
>
> Just like the trained games, alt files specifically labeled as "bad dumps" have
> already been removed, except for one 4-in-1 pack which didn't have any other dump
> available (all these things are explained in the comments inside the XMLs). Also
> removed are the DSK files generated with the ZXZVM interpreter and the games from the
> "Crap Games Competition", for obvious reasons.
>

Some of the tape->disk conversions can be interesting, they're filling a use-case hole for real systems and are a valid part of the current scene (much like cracked images, even recently done ones etc.) Likewise some game that have been patched with information to fix serious ingame bugs are handy to have (I believe some versions of Werewolves of London need a loader like that, with the original fix being published in a magazine back in the day) Again those would need to be marked an an appropriate way tho, but are very much part of the scene.

> Anything else you guys feel that should be tweaked before submitting, please let me
> know.
>
>
> ...And speaking of tweaks, I've noticed a lack of Spanish accents and symbols in the
> titles, does MAME support them right away or do they have to be added as "á",
> etc?

The Software List support unicode (if the file is saved with the correct encoding) although they won't render properly on the commandline.

Honestly it would be nice to see people work together with this, not against each other. You *really* do need to make a Pull Request tho, or it is all wasted effort that will be lost, not refined.

Moving off-topic, one more annoying thing with TZX cassette stuff is that the format has maybe 'too many' features, containing things like 'optional blocks' that are sometimes used for trainers in otherwise clean images (and other features like 'loop points' IIRC) This is a bit of a problem for MAME because MAME converts everything to an audiostream on startup so everything is treated as unconditional. As you're not doing cassettes yourself it's not so much of a worry, but it was on my mind. There are also multiple ways of representing the same thing, which is why there are some redundant entries in the cassette list right now; they're for improving the TZX decoder, which isn't perfect.



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: New specpls3_flop softlist. new [Re: Haze]
#381372 - 03/05/19 12:56 AM


> You *really* do need to make a Pull Request tho, or it is all wasted effort
> that will be lost, not refined.

Ok, thank you. I'll keep those entries for now, check their contents and present each case in the pull request so that their inclusion can be debated (I think the MicroByte collections have many commercial games otherwise unavailable in disk form, for example, so those may be acceptable).



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Sorting redundant entries new [Re: Haze]
#381373 - 03/05/19 01:15 AM


> There are also multiple ways of representing the same thing, which is why there are some
> redundant entries in the cassette list right now; they're for improving the TZX
> decoder, which isn't perfect.

It would be really useful it there was a way to mass-compare checksums from the data blocks inside the files... Not just for TZX, but for any other similar formats where each "full dump" is inevitably going to have its own checksum due to small differences detected in pauses between blocks, etc.

Then again, there may also exist certain releases where only the pause lengths are different (we have to think of every possible situation), so there would have to be a threshold for considering them separate things.



Haze
Reged: 09/23/03
Posts: 5245
Send PM


Re: Sorting redundant entries new [Re: ICEknight]
#381375 - 03/05/19 01:34 AM


> > There are also multiple ways of representing the same thing, which is why there are
> some
> > redundant entries in the cassette list right now; they're for improving the TZX
> > decoder, which isn't perfect.
>
> It would be really useful it there was a way to mass-compare checksums from the data
> blocks inside the files... Not just for TZX, but for any other similar formats where
> each "full dump" is inevitably going to have its own checksum due to small
> differences detected in pauses between blocks, etc.
>
> Then again, there may also exist certain releases where only the pause lengths are
> different (we have to think of every possible situation), so there would have to be a
> threshold for considering them separate things.

pretty sure there are copy protections / loaders where the pause lengths matter too, so even data that isn't data potentially makes a difference ;-)



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


Re: New specpls3_flop softlist. new [Re: ICEknight]
#381398 - 03/06/19 08:15 PM


I made a pull request listing those collections which might have to be left out:
https://github.com/mamedev/mame/pull/4724



ICEknight
MAME Fan
Reged: 07/06/15
Posts: 166
Send PM


A headstart for spectrum_cass.xml new [Re: ICEknight]
#381436 - 03/08/19 11:52 PM Attachment: spectrum_cass TOSEC WIP 20190319.zip 1694 KB (9 downloads)


Here's the 25000+ Spectrum TZXs from the latest TOSEC converted to MAME's XML format.

As pointed out before, it would be too crazy for me to do all these alone (it was already a lot of work to get to this starting point) but, in case that somebody wants to get started, here's the details:

Already done:
-All TOSEC entries are in the proper MAME softlist format.
-All articles preface the game titles (no more commas).
-Every name is correctly ordered (now commas separate the different people/entities involved).
-Removed some dumps which have been made obsolete recently.
-Identified some releases from my old site and labeled them
-Removed files generated with the ZXZVM interpreter.
-Removed games from the Crap Games Competition.


To do:
-Purging bad dumps, trained games, Jet Set Willy hacks, and other unwanted entries.
-Merge sides/tapes from the same release into one entry.
-Label each side when there's more than one.
-Add a unique name for each entry.
-Parent-clone relationships.
-Label the publisher in certain entries as "homebrew" or "unknown".
-Specify required hardware which may be needed for certain games.


The "Games" file alone is 17612 entries while the rest combined is 9108, so... having at least some automation would definitely help make this goal more feasable.



EDIT: The owner of SPA2 is sending me the DSKs from the backlog, so that I can "provisionally" add them to the +3 softlist while they're still being verified for their (undetermined) official release.

EDIT 2: Files have been updated.

Edited by ICEknight (03/19/19 02:24 AM)


Pages: 1

MAMEWorld >> EmuChat
View all threads Index   Threaded Mode Threaded  

Extra information Permissions
Moderator:  Robbbert, Tafoid 
0 registered and 240 anonymous users are browsing this forum.
You cannot start new topics
You cannot reply to topics
HTML is enabled
UBBCode is enabled
Thread views: 1348