You are not logged in.

Read the FAQ and Knowledge Base before posting.
We won't make a 3DS/2DS emulator.



#1 2015-07-20 17:29:06

hcf
Member
Registered: 2014-02-18
Posts: 23

Trimmed games regression in 0.9.11?

Hi, I was not sure if I should open this thread in the Technical subforums, or in the Compatibility ones, or in the bug tracker, as I'm not sure if Desmume developers were awared of this thing, and if it has been made purposedly or it is a regression (or simply, if nobody cares about this matter).

The fact is that we could play trimmed games until version 0.9.10 without troubles (i.e., making roms smaller with programs like "NDSTokyoTrim25Beta2.exe". But in Desmume 0.9.11 I have had several problems when working with trimmed games. One clear example to reproduce this issue, is with the game "Chessmaster - The Art of Learning": the original version plays well with both 0.9.10 and 0.9.11 versions. After we trim it with the program that I said above, it works very well in Desmume 0.9.10, but it crashes on start with Desmume 0.9.11.

As I said, I don't know if anybody cares about this. I was (finally!) updating the core of the Xbox port to the last 0.9.11 version, to be finally synchronized with your project, and I found that several games crashed... until I realized that it was due to trimming! So, as I didn't find informartion about this, I decided to tell it. Maybe other people are suffering the same issue and the don't know the reason.

Thank you very much for your amazing work!

Offline

#2 2015-07-20 22:11:35

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

This was caused by the rom being trimmed to 0xB alignment. The code didnt expect anything other than 0x0 0x4 0x8 or 0xC alignment. The rom reading internals need to be modified to make them safe for over-reading.
I've already fixed the code, I'll commit it when SF gets its shit together

Offline

#3 2015-07-21 06:44:16

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

zeromus wrote:

I've already fixed the code, I'll commit it

Wow, thank you very much! I would love to see the final solution to this problem! I tried to fix it before posting this message, by changing the way to calculate the romsize/cartsize, without success.

zeromus wrote:

when SF gets its shit together

Is there any problem with sourceforge? It seems to be working well for me today...

Offline

#4 2015-07-21 14:27:37

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

It isn't working well.

Offline

#5 2015-07-21 14:54:12

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

zeromus wrote:

It isn't working well.

It seems that you are right, sorry. I entered in an old project that I had, and apparently it seemed to work well, I was able to download, browse and change options. Sorry if I lead you to confusion.

I hope this patch can be applied to 0.9.11 source. I am really excited because I have finally updated DesmumeX to the 0.9.11 core and I hope that now our Xbox project will be always synchronized with yours. I have everything ready for a release, but I am going to wait until this patch is commited. I know that a lot of Xbox users have trimmed roms (due to the small size of the stock Xbox hard disk) so the ability of playing with trimmed roms is really important for them. And on the other hand, I can't wait to learn by looking at the source code of this patch, after having being unable to fix it myself  smile

Last edited by hcf (2015-07-21 14:54:47)

Offline

#6 2015-07-21 15:38:32

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

It should patch cleanly.
I should add some provision for decompressing roms on the fly. sounds kind of fun actually. it'd have to be a custom format though.
It would complement some other work i need to do of making the rom reading more modular anyway.
I think the way it would work is each 64KB or so will be compressed individually. Got to be careful to keep it small so random reads dont crater performance by having to decompress large buffers. But if it's too small the compression will suffer. I fear whatever number I pick will be done without adequate research and people will chew me out later.

Offline

#7 2015-07-21 19:15:20

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

That sounds awesome, can't wait to see it! Please, tell us whenever it is commited smile

Thanks for the amazing work!

Offline

#8 2015-07-25 23:58:19

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

ok SF finally got svn un-broken and the trimmed rom fix is committed.
drastic dearchives roms to a cache when theyre first opened and then skips the successive dearchivings when they exist in the cache. therefore supposedly compressed roms are convenient to use. why couldnt you do something like that?

Offline

#9 2015-07-26 19:36:54

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

zeromus wrote:

ok SF finally got svn un-broken and the trimmed rom fix is committed.

Thank you very much! I am going to look at it right now! big_smile

zeromus wrote:

drastic dearchives roms to a cache when theyre first opened and then skips the successive dearchivings when they exist in the cache. therefore supposedly compressed roms are convenient to use. why couldnt you do something like that?

That point is very interesting. The problem of the Xbox is the lack of RAM (we have only 64 MB) so we can't have a very huge cache, but I still think that something can be done in that way. One of the first things that we tried in our first versions of the port, was using a little array in RAM and read blocks of the cartridge in the readROM method (instead of reading single data) but surprisingly, this didn't improve performance. We guessed that the "fread" method maybe does the same thing internally (reading blocks of data in the cache of the processor, even if we only ask for one byte) and that was the reason why we didn't get any boost.

But yes, this is one point where we must come back in the future. And if we manage to do that with compressed roms too, that would be awesome!

Last edited by hcf (2015-07-26 19:39:50)

Offline

#10 2015-07-26 20:20:08

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

theres no guessing needed, fread certainly does that, and filesystem caching will be implemented at various other levels of the software and operating system. Anyway I didn't know this was about performance, you said it was about hard disk space. Compressing roms could be useful for performance if the hard disk is very slow relative to whats needed (store more compressed rom data in memory) but that isn't what you said at first.  That would also be a good answer to my question of why couldnt you do something like "that"--because it wouldnt even be relevant.

I'm not sure I'm inspired to do this work if it's so you can hack around with disk io performance. You can do it yourself. It will be easier if you wait until I repair some of the problems in the readROM methods and such to make it more modular, you'll see. I suggest watching the commits and waiting til I do that.

Offline

#11 2015-07-26 21:58:25

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

zeromus wrote:

theres no guessing needed, fread certainly does that, and filesystem caching will be implemented at various other levels of the software and operating system. Anyway I didn't know this was about performance, you said it was about hard disk space. Compressing roms could be useful for performance if the hard disk is very slow relative to whats needed (store more compressed rom data in memory) but that isn't what you said at first.  That would also be a good answer to my question of why couldnt you do something like "that"--because it wouldnt even be relevant.

I'm not sure I'm inspired to do this work if it's so you can hack around with disk io performance. You can do it yourself. It will be easier if you wait until I repair some of the problems in the readROM methods and such to make it more modular, you'll see. I suggest watching the commits and waiting til I do that.

Sorry for the misunderstanding, you are right. Well, in Xbox we have several struggles: performance is one of them, lack of hard disk space is another one, and lack of RAM is another one tongue  So, doing something that may help in any of these points, would be very usefull! When I tried to create the cache system, I was thinking about performance (and I had no success) but I didn't imagine the possibility of using that system to use compressed files. That is what I said that was a very interesting idea!

Of course I will keep an eye in your commits. The main goal of updating the emulator to the last core, is trying to keep it updated if possible. It is much easier to update the emulator from 0.9.11 to 0.9.12... than doing it from 0.9.0 to 0.9.12! The big surprise has been that, after upgrading to the 0.9.11 core, I have noticed an important performance improvement too (besides the compatibility improvement that I expected). That was a big surprise, because in other emulators that we have ported, we have seen that modern versions give more compatibility, but they are also slower. In Desmume you have made an impressive work recently, because modern cores seem to be more compatible, but also much faster! So I must say congratulations for the impressive achievement! big_smile

EDIT: I have tried your patch to support trimmed files, and it works like a charm! Thank you very much! smile

Last edited by hcf (2015-07-26 22:02:07)

Offline

#12 2015-07-26 22:05:10

zeromus
Radical Ninja
Registered: 2009-01-05
Posts: 6,169

Re: Trimmed games regression in 0.9.11?

Well, between 0.9.0 and 0.9.12 you wouldve got the recompiler. Is that working on xbox? In recent years I doubt we have increased performance much but back in the 0.9.0 days we were still improving things.

Offline

#13 2015-07-27 06:29:00

hcf
Member
Registered: 2014-02-18
Posts: 23

Re: Trimmed games regression in 0.9.11?

Yes, the dynamic recompiler works very well on Xbox (the processor of the Xbox is an Intel x86, similar to a PC one). Also, I have noticed that the frameskip is much more polished between these versions (we skip lots of things now when frameskip is enabled), the nested loops of NDSSystem.cpp where removed, and besides there are options that can make things faster, like the ones related to the sound. Overall, there are several games where the speed improvement is very noticeable smile

Offline

Board footer

Powered by FluxBB