UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed

[SplendidAlakey]

I'm not sure if this is a problem with the script or my region file (the latter is more likely as it's the only one that is having this issue). Whenever I run the script, it will stop at 1 specific mca file with the following error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 9: invalid continuation byte.

Full log: https://gist.github.com/Footage2-Amply-Pounce/756933e4a2a83b1215b0fcd804da5f30

Is there anything that can be done to fix it? I don't really need to clean this file, but I actually noticed that I get a significant FPS lag in 1 specific place of this region and completely wiping it helps, but I won't be doing that for obvious reasons. That makes me think that the problem is actually with the chunk file, rather than the script, but I'm hoping you could at least help me identify the issue as I have no clue what this error means.

I could send you the file if needed, however, you won't be able to load it in game as it's heavily modded and reproducing my mod setup is going to be too much to ask for.

[Nyveon]

Definitely interesting, not something I've ever seen before.

It seems something is up with the encoding of that file in particular.

I'd love to check out the file if possible! This might be fixable

[SplendidAlakey]

There you go: [deleted]

[Nyveon]

Ok, so after a bit of digging it seems this is a reported issue here, with one of the packages this project uses:
twoolie/NBT#144

Turns out, NBT is stored as mutf-8 and not utf-8, in particular it seems this can cause issues with in-game books?

Regardless, it can be fixed, all you have to do is apply the patch mentioned in the thread.

A more direct (but I guess, not proper) way of doing it would be:
pip install mutf8
Going to where your python packages are, specifically where the nbt library is (e.g: C:\Users\username\AppData\Local\Programs\Python\Python39\Lib\site-packages\nbt)

In nbt.py, adding import mutf8 after line 8
In the TAG_String class, _parse_buffer method: switching

• self.value = read.decode("utf-8") for
• self.value = mutf8.decode_modified_utf8(read)
  In the TAG_String class, _render_buffer method: switching
• save_val = self.value.encode("utf-8") for
• save_val = mutf8.encode_modified_utf8(self.value)

And that should fix it, at least until you update the package. Good enough for just handling this single file I'd think.

I tested it out with your file, and it seem to work now :). Judging by the error, I'm not sure if my tool will be what can fix your issue (it might not be a problem with structures), but I hope it does! Best of luck

[SplendidAlakey]

Alright, thanks a lot!

seems this can cause issues with in-game books?

Might be why writing "================================" in a book causes player data corruption and kicks the player. Found that out the hard way a while ago. Also could be why this is the only region out of more than 2000 in my world that throws an exception during the cleaning process, as it's the only region that contains books.

I'm not sure if my tool will be what can fix your issue

Yeah, that's alright, I wasn't really hoping to, but I found it interesting that I've been having FPS issues in the same region that failed to get cleaned. I will try the fix and see if that somehow helps.

Other than that your script was a life saver to me. I had to remove leftover references to structures after deleting a mod, because it was creating insane spam in the logs. There were millions of references, the only other option would've been resetting the world, so I'm glad your tool exists 👍

[Nyveon]

Glad I could help!