You are not logged in.

Read the FAQ and Knowledge Base before posting.
We won't make a 3DS/2DS emulator.



#1 2009-05-19 20:57:38

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Request for comment: desmume save file format

In the desmume SVN, we are working on a new save file system which may possibly accomplish 100% autodetection. I won't claim that it succeeds yet, but it very well could.

In order to do this, we have needed to add additional information to the .sav file format. This changes it away from its raw file format which desmume users are accustomed to.

You can see documentation on the proposed new file format here: http://desmume.svn.sourceforge.net/view … me/sav.txt

You can see the code implementing this at: http://desmume.svn.sourceforge.net/view … ision=2222

Do not be dismayed by the hardcoded list of autodetections. Currently it is confirming to the rule addr_size = autodetect_size&3; but we are gathering data to prove this.

There are two questions: do people like this file format? and, is the incompatibility/differences between raw .sav a problem?

The strange footer file format was chosen to maximize the probability of compatibility, although it does not always succeed.

You can see a some discussion of this here, including some justification for the metadata and an alternative proposal: https://sourceforge.net/tracker/index.p … tid=832291

Here is a 4kbit EEPROM sav:

00000000 42 4F 4D 42 31 02 03 04 20 10 10 10 10 10 10 10 10 10 10 10 00 00 00 00 00 00 00 00 00 00 CF AB BOMB1... .......................
00000020 BB 05 00 00 10 00 00 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 79 00 6F 00 70 00 79 00 ........................y.o.p.y.
00000040 6F 00 70 00 20 00 20 00 20 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 o.p. . . . .....................
00000060 00 00 00 00 00 00 00 00 00 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000080 D5 0F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000000E0 D5 0F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000140 D5 0F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001A0 D5 0F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................................
00000200 7C 3C 2D 2D 53 6E 69 70 20 61 62 6F 76 65 20 68 65 72 65 20 74 6F 20 63 72 65 61 74 65 20 61 20 |<--Snip above here to create a 
00000220 72 61 77 20 73 61 76 20 62 79 20 65 78 63 6C 75 64 69 6E 67 20 74 68 69 73 20 44 65 53 6D 75 4D raw sav by excluding this DeSmuM
00000240 45 20 73 61 76 65 64 61 74 61 20 66 6F 6F 74 65 72 3A 00 02 00 00 00 02 00 00 00 00 00 00 01 00 E savedata footer:..............
00000260 00 00 00 00 00 00 00 00 00 00 7C 2D 44 45 53 4D 55 4D 45 20 53 41 56 45 2D 7C                   ..........|-DESMUME SAVE-|

Offline

#2 2009-05-19 21:16:17

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

zefiris has suggested using raw .sav + .dsc desmume-specific data, encapsulated in a zip file. That way it would always be trivial to get what you needed. However, moving between systems and emulators would not be totally seamless and would require some additional steps.

Offline

#3 2009-05-19 21:43:54

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

As mentioned elsewhere, I think its better than desmume and ideas corrupting each other's save data.

I'd much rather do a little more work converting saves if i need to transfer to another emulator.. than risk having corruption.

Offline

#4 2009-05-19 21:53:43

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

I'm not specifically against the new format, i just don't want it to have the same filename, because ideas would corrupt the save data.

That said, i think zefiris idea is a good one, but not necessarily required.  Just give it a different extension, like .dsv or something.  If desmume can't find the file, try to load a sav file instead.

It would be trivial to have an export to .sav file option for people that need it later.

Offline

#5 2009-05-20 13:54:27

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

Well, I know how to fix my copy.  I think if you still want to do it, if anyone complains later, you can tell them the plans were listed here, and if they can't take an interest in local affairs, it is not your problem. smile

Offline

#6 2009-05-20 17:56:22

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

Another idea: We could dump a .sav (100% pure raw) and a .dsv (new format). We would always try to load the dsv first. This would still break in case you went back and forth to ideas, because when you ewnt back from ideas to desmume, desmume would be loading the dsv instead of the sav which you had played more recently in ideas. So that point would be confusing. Also, subsequently, the game might load the dsv and then write back to the savefile, clobbering the sav you changed in ideas.

Offline

#7 2009-05-20 18:41:04

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

Yeah that would be confusing.  And, not really better than just using the raw SAV file and some additional desmume config data in a separate file.

Offline

#8 2009-05-29 21:42:07

Luigi__
Member
From: Peach__'s castle
Registered: 2009-05-29
Posts: 18
Website

Re: Request for comment: desmume save file format

So we suddently changed the save file format to proprietary?
I'm okay with this change ONLY if the conditions below are met:

1. If DeSmuME can load raw save files and convert them to its own format, just like NO$GBA
2. If the folder for save files is changeable, because otherwise it'll be a pain to manage the save files
because, when you run a game on another emu with a savefile created by desmume, the game will
most likely erase your save file.
3. If users can convert desmume saves to raw saves.


Kuribo64 - If you're wondering where Mario__ is, he's currently saving Peach__ once again.

Offline

#9 2009-05-31 04:21:21

Pokefan999
Member
Registered: 2009-05-24
Posts: 114

Re: Request for comment: desmume save file format

The current new save format seems to be working well between desmume and ideas because the addtional info is appended at the bottom  and ideas can load desmume save directly without issue (even with the additional info).  When ideas save the game again, it will wipe out the additional info.  Even save file created from ideas savestate can be loaded directly by desmume without issue.  On games that created 65KB instead of its 256KB on initial run, it is also compatible with ideas because it is reading sufficient data to load the game.  Instead of creating 2 separate files, why not keep its current form.

If this post doesn't make sense, please delete it.  Sorry for the intrusion.

Offline

#10 2009-05-31 08:39:52

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

Apparently not everybody gets the same perfect results that you do.

The system is working as intended for the games you tested, but there are some games that do not work out so well. And when it doesn't work, it is catastrophic, since it wipes your save.

But thanks for sharing your results.

I have a question: a lot of people seem concerned about quick ideas<->desmume save file transporting. Why? Why do people use ideas at all? What are you trying to accomplish whenever you are switching back and forth?

Offline

#11 2009-05-31 09:00:37

Pokefan999
Member
Registered: 2009-05-24
Posts: 114

Re: Request for comment: desmume save file format

No offence to both nds emu developers, they are great programmers.  For me, I use ideas to quickly verify the bug I found in desmume before reporting (just to be sure). ideas is strong in its graphical accuracy and game compatibility while desmume is very good in sound and speed stability etc.

I would say both emu complement each other in the areas that they lack or weak.  I use desmume as my primary emu because I see better interaction with the developers in getting bug fixed.  For ideas, I won't know if I am reporting something that the developer is interested in fixing until next release.

Of course desmume has make up alot of ground in the areas it is weak for the upcoming 0.9.3.

The only save file created by desmume that ideas reject are those created wrongly by desmume auto detection.  I am really curious which game created a good save file in desmume (i.e. can continue from save point) that is rejected by ideas.

Just my thought.

Last edited by Pokefan999 (2009-05-31 09:40:40)

Offline

#12 2009-05-31 12:22:58

nash67
Member
Registered: 2009-02-28
Posts: 23

Re: Request for comment: desmume save file format

Pokefan999 wrote:

No offence to both nds emu developers, they are great programmers.  For me, I use ideas to quickly verify the bug I found in desmume before reporting (just to be sure). ideas is strong in its graphical accuracy and game compatibility while desmume is very good in sound and speed stability etc.

I would say both emu complement each other in the areas that they lack or weak.  I use desmume as my primary emu because I see better interaction with the developers in getting bug fixed.  For ideas, I won't know if I am reporting something that the developer is interested in fixing until next release.

Of course desmume has make up alot of ground in the areas it is weak for the upcoming 0.9.3.

The only save file created by desmume that ideas reject are those created wrongly by desmume auto detection.  I am really curious which game created a good save file in desmume (i.e. can continue from save point) that is rejected by ideas.

Just my thought.

but how do you know if it is a bug in both ideas and desmume
the only real way is to know if it is a real bug is to test it on real hardware
IMHO Desmume is way ahead of ideas in compatibility, sound and speed
Im not interested if you can change saves between both emulators
I think the team is doing and excellent job without having to worry about it their save is going to work on another emulator
Desmume is coming along in leaps and bounds and will overtake No$gba in compatibility very soon
just my 2 cents smile

Offline

#13 2009-05-31 14:01:10

Pokefan999
Member
Registered: 2009-05-24
Posts: 114

Re: Request for comment: desmume save file format

nash67 wrote:

but how do you know if it is a bug in both ideas and desmume
the only real way is to know if it is a real bug is to test it on real hardware
IMHO Desmume is way ahead of ideas in compatibility, sound and speed
Im not interested if you can change saves between both emulators
I think the team is doing and excellent job without having to worry about it their save is going to work on another emulator
Desmume is coming along in leaps and bounds and will overtake No$gba in compatibility very soon
just my 2 cents smile

You may run DS game in desmume and then move save file to real hardware to verify (that is your style) and nobody will say it is wrong.  I prefer to do my verification in seconds so long as it does the job (that is my style).  If desmume developers felt that the bug that I reported lack accuracy or depth, I am sure they will put their frank comment in my bug post.  If my bug report is not welcome by desmume developers, I can stop posting (so far I have not received such signal from the developers).

I do not understand why you get so upset with my post as I am responding to zeromus question.  If my reply breaks a rule or antagonize the administrator or developers, please freel free to ban me.

Offline

#14 2009-05-31 14:14:24

nash67
Member
Registered: 2009-02-28
Posts: 23

Re: Request for comment: desmume save file format

Pokefan999 wrote:
nash67 wrote:

but how do you know if it is a bug in both ideas and desmume
the only real way is to know if it is a real bug is to test it on real hardware
IMHO Desmume is way ahead of ideas in compatibility, sound and speed
Im not interested if you can change saves between both emulators
I think the team is doing and excellent job without having to worry about it their save is going to work on another emulator
Desmume is coming along in leaps and bounds and will overtake No$gba in compatibility very soon
just my 2 cents smile

You may run DS game in desmume and then move save file to real hardware to verify (that is your style) and nobody will say it is wrong.  I prefer to do my verification in seconds so long as it does the job (that is my style).  If desmume developers felt that the bug that I reported lack accuracy or depth, I am sure they will put their frank comment in my bug post.  If my bug report is not welcome by desmume developers, I can stop posting (so far I have not received such signal from the developers).

I do not understand why you get so upset with my post as I am responding to zeromus question.  If my reply breaks a rule or antagonize the administrator or developers, please freel free to ban me.

that was just my opinion
I was not upset with your post at all

Offline

#15 2009-05-31 18:24:43

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

Pokefan, uhh I will let you know if I find any more of those games.

The time is rapidly approaching where we need to finalize this and ship 0.9.3. Here is my current favorite idea:

desmume will use its own .dsv format. There will be options to import and export raw .sav. This is a very simple and non-confusing system, since it matches a workflow that everyone has experience with: sometimes you have to save things out in different file formats to get them into other applications.

raw .sav files can have their size specified at the point of import or export. Therefore if you have a tricky game, you can always pick the choice manually which works.

Desmume will still automatically load a .sav file but only if there is no .dsv file. If it imports it incorrectly automatically, you can always use the file menu to import it manually.

pokefan and nash: I do not think that having to export a .sav will appreciably slow down your testing.

Offline

#16 2009-06-01 01:22:42

Pokefan999
Member
Registered: 2009-05-24
Posts: 114

Re: Request for comment: desmume save file format

I suppose it is the best way because I found that if the save file has the last byte written with valid data instead of x'00', ideas will create a 64KB save file for the game if there is additional data after the last byte even if the savetype is 1KB e.g. Autobots or Decepticon or Harry Potter or 8KB e.g. Pokemon Link.  It will either allow a new game to start or simply reject the save.  Those save file that does not have the last byte written with valid data will not have portability problem.

Offline

#17 2009-06-01 08:40:41

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

Alright here is the current state of the save system. I didnt do exactly what I said in my post earlier.

The file extension has been changed to dsv.
If a dsv is missing, then the emu will try and load a raw sav.

The emulator will be unable to load new-format .sav files (because now, the only kind of sav is a raw sav) which have been created over the past few weeks. You'll need to rename those to .dsv.

There is now a file > export backup memory command which will write a raw sav. The size of this file is identical to the size of the dsv which is current right now, except of course the footer is gone. This size is not determined by the user's current setting.

Every time the emu loads a duc or a sav, it will use the current save type manual/autodetect configuration. If autodetect is chosen, it will autodetect it FROM THE FILE based on the size. This is different from before*. So, if you are going to load a sav from no$gba, you MUST manually specify the size since theyre all 512KB. If you are going to load a sav from ideas, you will probably be able to leave it on autodetect. When a duc or a sav is loaded, it overrides the previous state of the emulator and resets it with this new size.

The emulator still uses the savetype manual/autodetect options when a rom is first booting. You should, of course, need to use these far less frequently. You may still have to for some games, though: it would seem that 100% autodetection is impossible. Currently the only game I know for a fact that must be manually detected is spiderman 3. We will keep a list of these as we verify them (there are a couple of cases on the bug tracker that I can't test very well due to the game sucking and me not being able to save)

*before, it would use the current state of the emulator, which half the time was mis-autodetected.

Offline

#18 2009-06-01 14:34:24

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

Sounds pretty reasonable.

I would have left the ability to load the experimental .sav files a few builds.... just for anyone that didn't get the memo.  But I'm just nice like that.  smile  Anyway, that's minor, hopefully any testers are keeping up on this change.  Will desmume crash if it tries to load the .sav file with the footer? 


If there are still a few instances that saves aren't autodetected correctly, I'll keep an eye out to see if there is another solution.  I believe there's always a way, its just a question of if its happening enough to be worth a fix.

Offline

#19 2009-06-03 07:55:14

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

Generally not, the game's save system will just freak out if it tries to load a new format file with sav extension.

it wasnt a matter of leaving in the ability to load the experimental sav with the footer. that wouldve needed adding and the code was already getting dangerously complex so i wanted to leave it out

Offline

#20 2009-06-03 13:40:18

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

Yeah, no biggie.  I just meant for a few builds.

Really, I don't think the code is complex at all.... just not particularly documented.  yikes
I pretty much need to step through the whole program in my head to decypher what is going on.

Offline

#21 2009-06-03 18:38:09

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,214

Re: Request for comment: desmume save file format

i dunno i guess we all have different thresholds of complexity. but .. i guess ill comment it better before 0.9.3

Offline

#22 2009-06-03 21:11:02

loki1405
Member
Registered: 2009-04-08
Posts: 17

Re: Request for comment: desmume save file format

Eh well, a lot of projects don't get documented much, but it makes things easier for people who didn't write the code. 

I'm probably an over-documenter myself. smile

Offline

Board footer

Powered by FluxBB