Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post here about all aspects of D2 mod making whether it's information, problems or whatever. Please specify whether your post is relating to Classic D2 or the Expansion.

Moderator: Nizari

Post Reply
User avatar
ZOMB
Posts: 51
Joined: Fri Feb 04, 2011 8:45 am

Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by ZOMB » Tue Oct 01, 2019 7:47 pm

Hi,

so I have this idea.

Since Diablo 2 remaster wasn't even announced (by Blizzard) yet, what stops us from creating our own remaster mod for Diablo 2? Just to be clear - when I say remaster, I mean only higher resolution with upscaled images, nothing else.

Even though we have some HD mods, all they do is just zooming out, leaving the original textures in low resolution.
But with the AI / machine learning boom, we have some new possibilities - as I've demonstrated today in this reddit post.
The results are rather good (monsters looking good - for example when the Fatality model is being used - the one in the middle in the image grid).
I've even seen some quite impressive examples of some locations in D2 upscaled using the same technique.

So I have this idea - a FullHD mod (but with 4:3 ratio - 1440x1080), but no zooming out - everything stays in place, just like in the 800x600 resolution. This means upscaling every single texture in the game by factor 1.8.

I've done some thinking about what is possible and what is not.

What is possible:
  • extracting all textures
  • upscaling all textures by factor 1.8
  • putting the upscaled textures back in the game

Possible problems:
  • Don't DC6, DCC or other texture formats have some size limit? In that case putting the upscaled textures back in the game would be problematic.
  • The underlaying (surface) grid (not the tile images, the model layer underneath) should be upscaled too. Don't know if the factor 1.8 is ok in this case - maybe the float would cause problems, but maybe a factor of 2 would be okay. In that case all textures regarding surface (monsters, characters, map tiles, missiles etc.) would be rescaled by a factor of 2.

What do you think about it? Will the image size be a problem for the game image formats? Could the underlaying grid be also extended?
Or are there any other problems you see that I am currently unaware of? Or any other suggestiong?

Thank you.
Last edited by ZOMB on Tue Oct 01, 2019 9:50 pm, edited 2 times in total.
Fan, player and modmaker of game Diablo 2
Diablo 2: Is Alive
Diablo 2: Skiller

tk90
Posts: 4
Joined: Wed Sep 18, 2019 4:13 pm
Austria

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by tk90 » Tue Oct 01, 2019 9:44 pm

The idea? love it!

It would be a lot of work though i feel like... im not sure how the end result would look for different textures... Like i feel that for Andariel Fatality would be the best, for Diablo on the other hand i would prefer Deviantpixelhd. do you have a "whole" screen upscaled already or a link to one? like a town or whatever, id love to see what it looks like!

Regarding your other questions im sorry but unfortunately im way too noob to help out with any of that :(

Edit: from some game screenshots i saw on reddit the only problem seem to be the faces, apart from that they look really good.

User avatar
Necrolis
Senior Admin
Throne
Posts: 9125
Joined: Sat Mar 25, 2006 1:22 pm
Location: The Land of the Dead
Contact:
South Africa

Hand-picked

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by Necrolis » Tue Oct 01, 2019 10:56 pm

ZOMB wrote:
Tue Oct 01, 2019 7:47 pm
Possible problems:
  • Don't DC6, DCC or other texture formats have some size limit? In that case putting the upscaled textures back in the game would be problematic.
  • The underlaying (surface) grid (not the tile images, the model layer underneath) should be upscaled too. Don't know if the factor 1.8 is ok in this case - maybe the float would cause problems, but maybe a factor of 2 would be okay. In that case all textures regarding surface (monsters, characters, map tiles, missiles etc.) would be rescaled by a factor of 2.
DC6 Files are limited to 256x256 (due to 3dfx cardsa at the time having a max texture size of 256x256). Upping it requires a few minor code edits. DCC files are limited to approx 200x200, due to the limited size of the decode buffer. This can also be upped with some edits. DT1's also have their own size limitations, due to the way floors/roofs are encoded (walls though use an arbitrary amount of pixel blocks, so that is fine), these require writing a whole new decoder to get around. The bigger issue here is the limited palette, you might upscale it nicely, but if you can't represent the new intermixed colors, its meaningless.

As for upscaling the coordinate system: this would require replacing a ton of code unless you do some weirdness purely on the client side (by abusing the gfx coord list, but even then...). It would be better to use some compositing trickery to render the same scene at a higher resolution (the same way a 3D title would), You'd just need to split the world and UI layers (very easy, cause they are thanking neatly split at the top level).
ZOMB wrote:
Tue Oct 01, 2019 7:47 pm
What do you think about it? Will the image size be a problem for the game image formats? Could the underlaying grid be also extended?
Or are there any other problems you see that I am currently unaware of? Or any other suggestiong?
Apart from what I mentioned above, there is another issue I see often with the AI upscaling, it tends to make small details fuzzy, thing means certain image section lose detail. The other thing is that the models work by recognising similar patterns to watch they have been trained with, which means if the input is small, there is very little to work with, meaning upscaling character/monster components is going to be tricky (not you have an 8x4px image to work with, a lot less info to go on).

Some things to consider:
1) These are pretrained models, so you can increase accuracy if you train a new model specifically for D2, this means you can account for palettes.
2) Current upscaling assumes that there is only one image to upscale, but if you can share info from all frames in a units animation (or in all related tiles) you should be able to increase the quality.
3) D2 might be a D2 game, but why keep it there? if you can upscale enough to preserve edges, it might be possible with photogrametry to create a 3D model (and again, you have the whole thing to work from, 8 directions with 2 or more frames), once you have a 3D model, repainting the texture is done once, then you can rerender them at a higher quality (or render them in realtime if you replace the GFX driver like I have done for D2GFEx). This is quite an amount of work (monstats has 208 unique tokens alone), but if you can automate the right parts, might be just as fast as doing every single token (thing like palshift masks can also easily be made in an automated fashion).
Image
Netiquette, Do you USE it?!?! | Nefarius' Fixed TXT Files | Terms Of Service
Blackened | Day of Death | D2GFEx
"What was yours is mine. Your land, your people, and now your life." - Lim-Dul, the Necromancer
Judgement is Final, Death is Eternal

User avatar
ZOMB
Posts: 51
Joined: Fri Feb 04, 2011 8:45 am

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by ZOMB » Sun Oct 13, 2019 10:48 pm

Thank you for the comments, especially to you, Necrolis, for your helpful analysis.

Yes, the palettes occured to me too. Turns out the required code-editing part would be much more complicated than I originally thought and the overall results are not very certain all things considered.

So I came to the conclusion to do some demo first - a web app, a demo version of D2, where all the textures are rescaled (that would be the ideal case - without the palettes restriction). That way me and everyone else can see how the rescaled game could (ideally) look like. Then I'll decide on what I'll do next.
Fan, player and modmaker of game Diablo 2
Diablo 2: Is Alive
Diablo 2: Skiller

User avatar
ZOMB
Posts: 51
Joined: Fri Feb 04, 2011 8:45 am

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by ZOMB » Tue Jan 21, 2020 4:19 pm

In case you're interested, here's the promised web game: https://quiet-springs-45295.herokuapp.com
Warning: it needs first download about 55 MB of data, you'll see a loading screen. Then you can walk around the Rogue Encampment, visit merchants etc. I know it hasn't much, it's only a proof of concept.
What do you think?
Fan, player and modmaker of game Diablo 2
Diablo 2: Is Alive
Diablo 2: Skiller

User avatar
tmuhlhausen
Junior Member
Paladin
Posts: 104
Joined: Sat Aug 24, 2019 6:43 pm
Canada

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by tmuhlhausen » Tue Jan 21, 2020 6:36 pm

That is actually pretty cool Zomb, Good luck

User avatar
Necrolis
Senior Admin
Throne
Posts: 9125
Joined: Sat Mar 25, 2006 1:22 pm
Location: The Land of the Dead
Contact:
South Africa

Hand-picked

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by Necrolis » Tue Jan 21, 2020 7:41 pm

ZOMB wrote:
Tue Jan 21, 2020 4:19 pm
What do you think?
Unfortunately the unit animations come out look very clay like (and I suppose using multiple frames to do inter frame sharpening might help), but the UI looks VERY good.
However there are two concerns:
1) it seems the tiles are upscaled as one large image, rather than the actual tiles, what happens if you scal them as they are in the resource files?
2) palatalization doesn't work at this kind of level, cause once you got to 24 or 32bit, you can't palshift, not without a masking texture (or a hardcoded alpha value for shader shenanigans) and that has its own issues, and mainly only works for single color replacement.
Image
Netiquette, Do you USE it?!?! | Nefarius' Fixed TXT Files | Terms Of Service
Blackened | Day of Death | D2GFEx
"What was yours is mine. Your land, your people, and now your life." - Lim-Dul, the Necromancer
Judgement is Final, Death is Eternal

User avatar
ZOMB
Posts: 51
Joined: Fri Feb 04, 2011 8:45 am

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by ZOMB » Tue Jan 21, 2020 11:01 pm

Thank you for the comments.

Necrolis:
Agree, the animations don't look as good as other upscaled textures, maybe I should use different model for the characters. I guess the problem is with faces - we as a human race are really good at reading each other's faces, so the faces must be of especially high quality when upscaled. The tool I've used doesn't cover that I'm afraid.
To your points:
1) Indeed, I just loaded the Rogue Encampment map into Paul Siramy's map editor and saved the map as a large image and the upscaled it (for convenience). To be honest, I don't remember how the raw resource files look like - is it something like this? https://d2mods.info/resources/wt/Act1/T ... or.dt1.htm
Do you mean what would happen if I've upscaled not the whole town, but this image with separate tiles instead and then put it together manually?
2) Yes, that is a problem. I've struggled with the alpha layer itself for some time to make ESRGAN handle it well. I also had difficulties extracting the animations I needed - again I did not extract the raw files, I used the MergeDCC tool - but the version I've used doesn't support export to image formats with alpha channel. I'm sure there is a way to export the alpha channel, but I just used this tool for convenience. And this was just the extraction and upscaling part. Inserting these upscaled textures back into game would be much bigger problem (or at least I think). I have never edited D2's core code, but I'm sure much of its graphics part would have to be rewritten to support this kind of textures.
Fan, player and modmaker of game Diablo 2
Diablo 2: Is Alive
Diablo 2: Skiller

User avatar
Necrolis
Senior Admin
Throne
Posts: 9125
Joined: Sat Mar 25, 2006 1:22 pm
Location: The Land of the Dead
Contact:
South Africa

Hand-picked

Re: Diablo II Upscaled (an idea on mod with upscaled textures based on machine learning)

Post by Necrolis » Wed Jan 22, 2020 4:10 pm

ZOMB wrote:
Tue Jan 21, 2020 11:01 pm
Agree, the animations don't look as good as other upscaled textures, maybe I should use different model for the characters. I guess the problem is with faces - we as a human race are really good at reading each other's faces, so the faces must be of especially high quality when upscaled. The tool I've used doesn't cover that I'm afraid.
When I get time, I am seriously considering making something to build 3D models + a texture, would solve almost all the issues WRT to resolution scaling, upscaling, animation interpolation etc.
ZOMB wrote:
Tue Jan 21, 2020 11:01 pm
1) Indeed, I just loaded the Rogue Encampment map into Paul Siramy's map editor and saved the map as a large image and the upscaled it (for convenience). To be honest, I don't remember how the raw resource files look like - is it something like this? https://d2mods.info/resources/wt/Act1/T ... or.dt1.htm
Do you mean what would happen if I've upscaled not the whole town, but this image with separate tiles instead and then put it together manually?
Yes, because you would need to upscale each individual tile, meaning there is less info for the RNN to work with. The alternatives are putting everything inot one giant input and then reslicing it, so somehow giving the RNN access to all related tiles at once.
ZOMB wrote:
Tue Jan 21, 2020 11:01 pm
2) Yes, that is a problem. I've struggled with the alpha layer itself for some time to make ESRGAN handle it well. I also had difficulties extracting the animations I needed - again I did not extract the raw files, I used the MergeDCC tool - but the version I've used doesn't support export to image formats with alpha channel. I'm sure there is a way to export the alpha channel, but I just used this tool for convenience. And this was just the extraction and upscaling part. Inserting these upscaled textures back into game would be much bigger problem (or at least I think). I have never edited D2's core code, but I'm sure much of its graphics part would have to be rewritten to support this kind of textures.
D2GFEx has the tricky parts with 32bit textures handled, the issue is one can no longer palshift (because there is no feasible way to do this). Once I have the last 2 bugs worked out (currently have an issue with the lighting engine and tile fades), I will start looking into a proper upscale, and various other cool features (like SDF or FT based fonts).
Image
Netiquette, Do you USE it?!?! | Nefarius' Fixed TXT Files | Terms Of Service
Blackened | Day of Death | D2GFEx
"What was yours is mine. Your land, your people, and now your life." - Lim-Dul, the Necromancer
Judgement is Final, Death is Eternal

Post Reply

Return to “General Mod Making”