Getting around the block ID limit.

[Thamus_Knoward]

Problem A: Minecraft (or I believe Forge rather) has a hard limit of 4096 block IDs that can exist at a given time. Since there are 16 metadata slots we can have a total of 65535 combinations. And according to Emote's post here (https://forum.westeroscraft.com/threads/westeroscraft-texture-pack-megathread.34/page-23#post-9966), we're close to having used them up (re: his comment about a 4___ ID glitch).

Problem B: Adding a ton of new blocks is expensive and adds to the loading time and memory requirement.

Disclaimer: I don't have any experience with how this is implemented in MC or how to effectively manage memory in modern programming languages, but I'm convinced there are some solutions to this that we should consider if we wanna keep growing.

So what are our options?

1. Push the update to 1.13. (according to 'rumors' the block ID limit has been removed:

https://www.reddit.com/r/minecraftsuggestions/comments/6lkvh8

) - Could solve A but likely not B and is a moderate amount of work for Mike having to translate all mods to the most recent Forge API.

2. Move to an open-source Minecraft clone server/ client combination that is more efficient or at the very least customizable in terms of memory management? A few examples are mentioned here: https://forum.westeroscraft.com/threads/westeroscraft-texture-pack-megathread.34/page-23#post-9961 - Could solve A and B but its a lot of work for our understaffed coding department.

3. Chop up Westeroscraft blocks into mini-mods and our server into separate minis each hosting a part of the world (yes apparently that's a thing https://www.spigotmc.org/threads/splitting-up-the-servers-or-one-big-server.2736/). We'd have to split up WesterosBlocks into a base-set and regional set, which could possibly give us more options in each region. - Could solve A not B though. And man, I can't even fathom the amount of work to split everything.

4. Tile entity ALL THE THINGS. Apparently, 'tile entity magic' is a workaround (https://github.com/MinecraftForge/MinecraftForge/issues/1769). No idea what that means /shrug. - A? B? Effort!

5. Same as 3. but instead of splitting the server we create several regional resource packs for WesterosBlocks and only keep a minimum necessary set of common styles. Think MCME but with a custom mod too in addition to the base blocks. Fixes A (for a while), slight memory/startup time increase due to multiple resource packs.

Do any of you see any other options? Is @mikeprimm still the only guy in the 'IT department' that can do the heavy lifting in Java? @Dav4 and @Emotione11 how are your degrees coming along I think if we ever wanted to fix this we'd need some solid computer science-inclined people

Edit: Here's another thing I'm curious about. Would it lower our memory strain if we lazy-loaded, rendered block models and textures per visible chunk instead of globally at the start of the game? Technically that should be possible with a custom client right? What are technical limitations to going to a completely custom client?

 

[Iwan]

4. Tile entity ALL THE THINGS. Apparently, 'tile entity magic' is a workaround (https://github.com/MinecraftForge/MinecraftForge/issues/1769). No idea what that means /shrug. - A? B? Effort!

Just going to cherrypick this point. Tile entities are the same as block entities in mojang speak. If I remember correctly, they're generally used when you want a block to store more than 4bits of data. Think chests. A whole map of chests? Laggy as hell.

 

[CashBanks]

1.13 might be the most viable long term plan, but it would bring some big changes. Here's how the WorldEdit team explains it: https://matthewmiller.dev/blog/updating-worldedit-to-minecraft-113/

User-Facing Changes
Firstly, I'll go over the user-facing changes. These are minimised where possible, except in some places they had to happen. The areas these mostly cover are block parsing and schematics.

Block Parsing
Due to the significant changes to blocks and IDs in 1.13, block parsing required an entire rewrite. The officially supported format for referencing blocks is to use the official Minecraft format. For example, grass blocks are used by typing minecraft:grass_block, and a snowy grass block would be accessed by typing minecraft:grass_block[snowy=true]. If the block is from the vanilla game, minecraft: can be omitted. For example, grass_block[snowy=true] will still place a snowy grass block.

To retain some aspect of legacy support, most legacy numeric IDs can still be referenced. Doing this is, however, NOT officially supported. Placing wool using their colour is also supported (Eg, red places red wool).

Schematics
The other major change required, schematics, also contains some legacy support code. As the MCEdit Schematic format is built to leverage numerical IDs, WorldEdit has moved to the Sponge Schematic Format. The Sponge Schematic Format is very future proof and will be the supported way of saving and loading schematics going forwards.

For the end user, the transition period should be relatively seamless. Previously created schematics will still load, however with some oddities such as corner stairs losing their corner pieces due to format restrictions. Any newly created schematic will be created using the Sponge format. Any schematic generated on 1.13 will not be usable on prior Minecraft versions.

The bit about legacy numeric ID's still being able to be referenced is promising, but I wonder if writing out block names would be more convenient anyway, e.g. minecraft:whiteharbor_cobble rather than having to use it's ID.

Would it be worth trying to head hunt some new blood on the coding team to help Mike, Will and Dav? We could try and back channel some ideas with the conquest/arda teams to see how they handle it too. Or hell, we could just pay someone to do it

 

[lemonbear]

How does the Conquest Reforged team get around these issues?

 

[Emoticone11]

Hard no to any solution regarding splitting up our world into different servers/mods/resource packs. Some servers I’ve seen do that, and it’s always annoying as hell.

Now, my understanding of the whole 1.13 & tile entities thing is that MC is already (in 1.11 and 1.12) supporting a system where every block is essentially a tile entity (or state I guess?), and are defined in model files of their own. I don’t believe that metadata is formally “a thing” anymore, so theoretically a block could have any number of states. Upon upgrading to 1.11, Mike basically hardcoded an interface into WesterosBlocks which parses the big JSON into all the separate model files in a way resembling ID:metadata divisions, which allows us to access blocks by ID/meta still. This is how it is currently. 1.13 I believe takes steps to eliminate ID and metadata identifiers altogether. I’m sure Mike could comment more on this though.

Cash, it sounds like WorldEdit’s bit about legacy numeric ID's still being able to be referenced isn’t anything to do with minecraft 1.13, but rather some simple aliases that WE hardcoded into the mod, so that it maps “1” to “minecraft:stone” for instance. I’m sure we could define our own aliases for WE, but I don’t think it has ramifications for the problem here.

Also Tham, my degree is done! I’m a PhD slave student now ^^ Anyways, the coding isn’t really the issue for me, but rather the time it would take to get familiar with minecraft’s codebase and modding libraries.

 

[AttackOnMars]

As I always propose (related to option 2) a port is going to be the best way to ensure the survival of the server. With Oracle moving Java to a more profitable subscription-based model it's only a matter of time before Java Edition gets deprecated IMO. A daunting though and a lot of work, BUT

1. There are a bunch of FOSS clones of varying quality, or we could make our own. Scary, but pitching such a project to something like Google Summer of Code or other well-publicised community coding efforts could yield a great starting point and even get the word out to people who may join the coding team longer-term. And if it doesn't go well, you just don't port; the only loss is some time and contributor enthusiasm.

2. Porting to Bedrock Edition. With their store being pack-focused they are bundling custom maps with custom sounds/textures/resources already, and the word on the street is their tools for managing these are really polished and the engine is _so performant_. We could either attempt a way in through their Minecraft Partner Program to release single cities and drum up some interest on the tail the end of the show's airing, or just keep talking to Microsoft peeps until we get the right person internally that feels like helping port the whole thing (which I have been trying at conferences for a bit).

(Or,. I keep waiting until I finish however many computing degrees and hope I magically have time to do this all myself. One can dream.)

 

[CashBanks]

If I could get someone who’s much smarter than me to synthesize exactly what our server’s technical specs are and all the relevant details of what we want to do, I could try reaching out to some of those servers who are in the Minecraft Partner Program to get their impressions of it, and just help with some general coder recruitment.

 

[Thamus_Knoward]

1. There are a bunch of FOSS clones of varying quality, or we could make our own. Scary, but pitching such a project to something like Google Summer of Code or other well-publicised community coding efforts could yield a great starting point and even get the word out to people who may join the coding team longer-term. And if it doesn't go well, you just don't port; the only loss is some time and contributor enthusiasm.

I gotta say, that switching to a FOSS clone would be my favorite option as this would grant us maximum flexibility and independence. From my limited perspective, it seems we would probably have more freedom to manipulate rendering and memory management than through modding via Forge. Imagine: We could have our own world storage format optimized specifically for our current and future needs (something that is both memory efficient and easily convertible to UE4 meshes?), we could improve the way rendering works (perhaps tweak it like in https://kotaku.com/horizon-zero-dawn-uses-all-sorts-of-clever-tricks-to-lo-1794385026 i.e. simplifying complex models or loading low-res texture for far-away chunks), and I suppose on the client side we could lazy-load into memory what you see instead of loading it all before even joining the server.

Going to the Bedrock Edition sounds enticing for the performance gains, but it also seems to me that it would complicate a few things and make us slightly less free/ more corporate (?) by having to involve Microsoft. It would tie us to their infrastructure, and frankly I'm not sure if thats what we wanted. It also seems that development of the bedrock server isn't far beyond and early-alpha?

 

[Iwan]

We briefly discussed a full-fletched port of both the 'production' aspects as well as the 'mmo' ones to UE4. Mike and Hal shared their respective insight in this thread. If the UE route is still an option, and if we were willing and able to commit the workforce to a port, it'd make sense to switch to a UE interpretation of our world data rather than taking the extra step of going through a foss clone or via ms, wouldn't it?

 

[Thamus_Knoward]

:I I've really neglected poor Hal and the whole MMO stuff in the last 4 years...

Thanks for rubbing that in But yeah, absolutely agree Iwan, it seems that @Hal9007 has totally figured out the basics in UE although the whole idea of collaboratively BUILDING the world still seems somewhat infeasible or difficult if I understand this correctly?:

Perhaps, but only the latter of the two. I've already mentioned the problems associated with UE4 level design for a large open world like this - as a default, Unreal Engine isn't readily equipped to handle massive worlds without some form of asset streaming based on preset load volumes. Traditional MMOs use this, to my knowledge, but were also created with streaming volumes to begin with, which is a tough act to follow. The latter point of UE4-based level design is what I see as the most reasonable pipeline for RPG development - work within the scope of the Unreal editor to develop an RPG-editing client through which a dedicated RPG team can populate the server with UActors.

If we work with UE I'm keen to get involved more again. I really wanna get the hang of that.

 

[CashBanks]

Seems like we’re spinning away from making a decision or coming up with a plan, what’s a step we can take right now to make one of these solutions happen?

 

[Hal9007]

In the interest of not derailing this thread, my up-front recommendation (or my hope, rather) for this particular issue would be to try and push the production server to at least 1.13.

I know you've brought up UE in both this thread and the other - I apologize for not really putting my work on it out in the open after the team fragmented; in essence, I have been working on it in relative silence for the last few years or so, but mostly because I was afraid that people might take its discussion as necroing an otherwise dead topic. In any case, during the course of development, I've made a number of useful realizations about the way Minecraft presents its "configurable" data. As Emote pointed out, Minecraft made some changes back in 1.8 to the way block shapes and metadata were handled, condensing them into a more or less uniform blockstate system. This was a great change overall - it allowed me to incorporate small changes at a time into my codebase which would eventually allow me to parse nearly the entire vanilla resource pack (with the exception of some special-case blocks, such as water) neatly into Unreal Engine without any manual intervention, as below from almost an year ago:



In a nutshell, 1.8+ is workable - after taking a look at the WesterosBlocks mod, I realized that uniformity would get a bit rough there, but the way blocks are presented is mostly the same, barring some CTM logic. Version 1.13 takes all of this a bit further, eliminating some of the redundancy between blocks of similar type and offloading a good amount of the hard-coded logic into new blockstate properties. But the real kicker is that it assigns a unique ID per block variation, which is VERY useful in terms of not reintroducing a lot of arbitrary metadata grouping. One of the main reasons they were able to do this was because of changes at the chunk data level - in storage, Minecraft used to write an ID and meta to each XYZ position in the chunk. This was intuitive, but it creates a low ceiling for the number of bits that should be used for storing those values, hence the block limit. In 1.13 and forward, chunks use a "block palette" at the section level, which tells Minecraft what types of blocks are used in ONLY that section and creates a mapping of unique IDs to palette IDs, which allows them to reduce the number of represented bits for most areas of the world (which only use a handful of unique blocks), with the added benefit of functionally removing the upper block limit.

All of these are great changes and carry us far away from the nonsensical Minecraft storage schema hodge-podge thing we had before, but are essentially what broke major plugins such as world editing tools in a big way. After seeing the added uniformity that 1.13 would bring, I immediately stopped writing my UE parser for 1.8+ and moved toward that. One thing I can tell you for sure is that the UE implementation, regardless of where it goes, is not geared toward being a world editing tool and almost certainly never will be; adding that whole range of features adds a ton of overhead, and is a big part of the reason why Minecraft uses so much memory (a lot of savings to be made when you can just assume chunks to be static meshes after they're generated!).

I know I'm pretty unfamiliar with plugins on the Minecraft side, so I can't speak to the difficulty of this and previous porting attempts. But because I am personally making use of 1.13-flavored data, I wouldn't be opposed to taking a look at where things are and where they'd need to go to help make it happen. Because block limit is more of a storage issue than a memory issue, I'm inclined to believe most custom client options are not going to help a tremendous amount, so as Cash said, focusing on a port to 1.13 is likely the most viable long-term option. At least, I shifted focus to 1.13 primarily with the intention of breaking even whenever - if ever - the server got to that point.

 

[Margaery_Tyrell]

Ah yes, I too know things about coding and the blocks and such

 

[Thamus_Knoward]

Have you mod folks been able to set the port to 1.13 in motion? @mikeprimm, I suppose you have begun porting Dynmap a while ago already? What are the impediments in moving WB and the server ahead?

 

[CashBanks]

Waiting for @mikeprimm be like

 

[Hal9007]

Been looking into this issue a bit more in the last few days with the hopes of getting a port on the rails - the largest roadblock on this front (and a rather large one) is that Forge and OptiFine are currently incompatible on all versions of Minecraft 1.13+. Again, this has to do with 1.13 being such a massive rewrite of Minecraft internals; looks like the Forge folks made a decision to rewrite much of their internal code around this time to provide a cleaner framework moving forward, while OptiFine devs began pushing releases without waiting for concurrent support. Forge has since provided APIs for both 1.13 and 1.14, but the interim delay has caused an apparent lack of compatibility between the two.

Thankfully, it seems like they are at least internally compatible; at this time, it seems to be an issue of OptiFine hooking into Forge as a core mod vs. as a library. The two teams seem to be doing a fair amount of finger-pointing toward each other as to where exactly the issue lies, so it's hard to say exactly when we might see full compatibility between them (could also point toward a direct hop to 1.14 over 1.13). However, the recent outlook seems good. We'll have to be patient on this front - Forge 1.13+ is a big step, with literally only one of the eight integral client mods we're currently using having stable releases for it.

We'll also have to monitor Fabric, which is a modding API for Minecraft 1.14+ which seems to have garnered quite a bit of support within the modding community - a number of the core mods we're using that haven't yet forward-ported their Forge implementations (such as WorldEdit) have 1.14 compatible versions with Fabric. It uses a rather ambitious code "mixin" system which could become problematic for scaling the mod environment, although it also seems to have its own merits in being somewhat more performant than Forge and relatively painless to use overall. There's a fair amount of discussion within the community as to the pros and cons of each API, and I can really only hope that their mutual success doesn't cause important mods we're using to go exclusively with either one of them. Unless Forge runs itself aground somehow, I'd be very hesitant to approach Fabric - it's outside the ecosystem that we've made extensive use of, and I expect in the long run, it'll be easier for heavyset, veteran mods / modpacks to stick to their roots.

TL;DR: Modding community is still in a state of adaptation in the wake of 1.13 - we'll have to wait a bit longer and see where things align before proceeding.

 

[CreativeRealms]

Now that Forge and Optifine work together in 1.14 and the block ID has been removed have there been any updates on this?

We could try and back channel some ideas with the conquest/arda teams to see how they handle it too.

How does the Conquest Reforged team get around these issues?

While this is irrelevant if an update to 1.14 happens, Conquest condenses blocks into blockstates depending on block properties. To use an example, a dirt block with the ID of 1408 has 14 blocks in it with two unused blockstates and a dirt snow layer with the ID of 1424 has 2 block types with 8 blockstate variants each. This decreases the amount of wasted blockstates and number of IDs that need to be used in general to push off reaching the ID limit for as long as possible.



Regardless of which route ends up being taken it sounds like there will need to be a full rewrite of the mod if you are close to or hitting the ID limit. Conquest is updating to 1.14 to solve this issue in the future.

 

[CashBanks]

Thanks for that insight CreativeRealms!
Correct me if i’m wrong but this is roughly what we do with our blocks Emote? As in using each block state :1 - :14

I wonder how many unique block ID’s Westeros uses compared to Conquest.

 

[Emoticone11]

Yeah, that’s the same thing that we do.

 

[Thamus_Knoward]

There is one more option that extends my points in the original post:
6. Clean up the pack. Remove redundancy wherever possible and consider removing obscure and underused blocks altogether in favor of high-value content. Optionally and additionally, we could start using all non-functional vanilla blocks again. Specifically, the newly added concrete, wood, and flower blocks. Might give us some breathing room towards A. Doesn't really change B.