MapTool 1.3 Bugfix Build 72 / 73

[Natha]

Tested too. Everything's ok as for saving/reloading things with accents.

But I'm slighlty disappointed as I misunderstood Azhrei and thought that he would "debug" the macro engine too

Things like these that don't work and are sometimes show stoppers for us, accentuated alien fools :

Code: Select all

[var1="éàù"]

Well ... kinda have to wait for 1.4 I guess

[Azhrei]

Tested Tables, campaign properties, tokens and campaign files. All working as expected.

Heh, and hopefully "expected" means "correctly"!!

Thanks!

Things like these that don't work and are sometimes show stoppers for us, accentuated alien fools :

LOL -- I didn't even know that was a problem!

And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.

I'm going to finish the version number thing and post a build, then I'll look at the parser...

[Natha]

And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.

Ok, thanks for looking at it.

On another subject, there is still a bug in b73 : a PC token saved is always reloaded as NPC (when dropped from the ressource library.

[mfrizzell]

And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.

Ok, thanks for looking at it.

On another subject, there is still a bug in b73 : a PC token saved is always reloaded as NPC (when dropped from the ressource library.

That happens in B70 and B63.

[LeopoldVonRanke]

Things like these that don't work and are sometimes show stoppers for us, accentuated alien fools :

LOL -- I didn't even know that was a problem!
And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.
I'm going to finish the version number thing and post a build, then I'll look at the parser...

Azhrei, thanks again for all your work! This is truly fantastic, and your work seems to be progressing at a very nice and smooth pace.
Two notes on this, one unrelated:

• - I hope we are talking about the same thing when talking about non-ASCII/UTF-8 problems. In b68 (which I'm still using) the problem was that a token ID with an Umlaut in it was miscarried into nonsense elsewhere, and after saving/reloading on the original machine as well; e.g. a token called "Brütal" (which is non-sensical btw) would turn into "Br?tal" over time. This might not be the same as a UTF-8 parsing error for Umlauts or accents in Macro names; since I haven't checked out b72+ yet, I dare not say which one is fixed or not. Seems our Spanish friends can chime in on this?
  - I noticed some translation errors in the German version; I mentioned this before, and have tried to get in contact with aPawn who did the German version, but he is not exactly fast in responding. If I can help correcting errors in some string list table, where and how do I do that? Is aPown the only one who is allowed to do this?

[Azhrei]

I am not familiar with the input encoding inside the GUI components. I believe that Swing simply uses whatever input encoding your platform is set to, so it should work when typed in and when displayed.

However, saving that information in external format and bringing it back in again is what I have been working on. So a token with umlauts entered in the Name field should AFAIK always work in MT as long as the campaign/token stays in memory. When written out and read back in, it may get screwed up.

Natha's comment about accented characters inside macro strings is a different problem. The GUI component which is the large text box should accept accented characters just fine, but the parser subroutine which is responsible for reading that text is likely not encoding-agnostic.

Second, the German translation is embedded inside the MapTool JAR file. Anyone is welcome to extract the current file, make updates to it, and then send it to me for inclusion in the next build. (I have on my list to externalize the translations so they can be put into a directory instead, but that's more work and I'm into bug fixing right now.)

To extract the file you can right-click on the JAR file in most desktops and choose to "Extract..." as the JAR file is simply a ZIP file with a particular format. Note that making changes and storing those changes back into the JAR file may or may not work -- there is an internal structure that could get messed up when using a generic JAR utility.

Another possibility would be to visit the source code repository and download the translation that you want to update. Note that the repository is likely to be even more up to date that the file embedded inside the JAR file.

If you do send me an updated translation, specify how you want your credit listed in the "credits" on the Help > About menu option. Although I typically will add people there after they've made 3-4 contributions, as just a single contribution would make the list pretty huge!

[aku]

yea, even my donkey would be on the list

[Azhrei]

And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.

Ok, thanks for looking at it.

It appears that the current ANTLR library we're using (2.7.6) does not 100% support Unicode and therefore the lexer (the component that reads text) doesn't either.

ANTLR 3.x (currently 3.2) does support Unicode but it has a slightly different Java interface. I will look at this to see how much work it would be to drop in v3.2 and make it work. Don't hold your breath.

On another subject, there is still a bug in b73 : a PC token saved is always reloaded as NPC (when dropped from the ressource library.

This is fixed in b74 (it's already in SVN).

[Craig]

And that's part of the parser -- really not my bailiwick. I'm guessing that the parser has some kind of single-byte limitation. I'll take a look, but all I was looking at was the file I/O portion of it.

Ok, thanks for looking at it.

It appears that the current ANTLR library we're using (2.7.6) does not 100% support Unicode and therefore the lexer (the component that reads text) doesn't either.

ANTLR 3.x (currently 3.2) does support Unicode but it has a slightly different Java interface. I will look at this to see how much work it would be to drop in v3.2 and make it work. Don't hold your breath.

There has also been a significant change in the syntax that is used to define grammars between ANTLR 2 and 3. You would have to rewrite the grammar if you wanted to update the version.

[patoace]

ANTLR 3.x (currently 3.2) does support Unicode but it has a slightly different Java interface. I will look at this to see how much work it would be to drop in v3.2 and make it work. Don't hold your breath.

That would be extra super cool, I'm officialy holding my breath now.

But meanwhile, do you have some specific place where you want accented characters, Natha? I have found workarounds to make this issues almost invisible to my players and I do like my ñs and ¿s

[Azhrei]

There has also been a significant change in the syntax that is used to define grammars between ANTLR 2 and 3. You would have to rewrite the grammar if you wanted to update the version.

Well, given that MTscript is on its way out, this probably isn't worth the effort.

If a volunteer wants to take a look at it I'd be happy to entertain the idea of applying patches. But replacing the grammar for MTscript is more time than I want to spend on this right now. You should see the list of bugs I have on my fix-it list.

[Natha]

Well, given that MTscript is on its way out, this probably isn't worth the effort.

Yep ... that sad but logical

But meanwhile, do you have some specific place where you want accented characters, Natha? I have found workarounds to make this issues almost invisible to my players and I do like my ñs and ¿s

The problem is when using a framework like DevientNull's, all the flavour texts and other non numeric data from powers are stored in JSON/variables/whatever ... At best, the accent is replaces by "square" character, at worst the macro doesn't work
We have the option to replace all accents by the equivalent HTML code but it's a PITA to write (for once) and even sometimes it's not correctly replaced in the output (HTML frame or chat) : a ";" is sometimes still printed (but I don't really get the how and when of it).

[patoace]

But meanwhile, do you have some specific place where you want accented characters, Natha? I have found workarounds to make this issues almost invisible to my players and I do like my ñs and ¿s

The problem is when using a framework like DevientNull's, all the flavour texts and other non numeric data from powers are stored in JSON/variables/whatever ... At best, the accent is replaces by "square" character, at worst the macro doesn't work
We have the option to replace all accents by the equivalent HTML code but it's a PITA to write (for once) and even sometimes it's not correctly replaced in the output (HTML frame or chat) : a ";" is sometimes still printed (but I don't really get the how and when of it).

Yeah, that's a problem because if you are using a framework, you'd have to write all the framework with the international characters in mind.

[LeopoldVonRanke]

Yeah, that's a problem because if you are using a framework, you'd have to write all the framework with the international characters in mind.

I honestly recommend not going that route. It is much better to have an all-Unicode accepting code base internally than to cover this up via patching things on the macro side in your framework with re-translating/patching all variable content through subroutines.

Speaking of which:
I understand that Azhrei has a rather long to-do list, and this is clearly not mission-critical, though if you are a user of non-ASCII characters it can be more than slightly annoying if your carefully typed accents and umlauts werewolf off into screen uglies.

It seems that, if the Lords of coding find it too intensive to transfer the code to a framework that handles Unicode universally, it must wait -- but that feature should definitely be on the list for 1.4!
Having Unicode handled correctly in all strings inside is a modern coding must.
In the meantime, if you really need using lots of international characters, maybe we can code an opensource macro lib that translates strings with international characters into correct html display symbols?

Just an idea...

[Azhrei]

It seems that, if the Lords of coding find it too intensive to transfer the code to a framework that handles Unicode universally, it must wait -- but that feature should definitely be on the list for 1.4!

As I mentioned above -- and as has been discussed here many times -- 1.4 will at some point deprecate MTscript in favor of JavaScript, probably with both of them co-existing for some amount of time. JS doesn't have any problems with Unicode.

[...] maybe we can code an opensource macro lib that translates strings with international characters into correct html display symbols?

Won't work. There's no way to get the accented characters into the string in the first place. Hence the problem.

As I said before, if someone wants to visit antlr.org (I think that's the site) and grab ANTLR v3 to try this out, I'd happily accept patches and do some testing. As someone who doesn't know or use accented characters I'm really not qualified to work on the "here's the solution" part of the problem.

Anyone here can install Eclipse and Subversion (should take about 30 minutes), then checkout the parser project (should take about 20 minutes; there are threads in the Developer Notes forum on how to do this for MapTool, just replace "maptool" with "parser"). There's even a test module that will exercise the parser so you don't need the full MapTool code base to try it out.

Or you can do it the "pen and paper" way and visit the SourceForge.net project and browse the SVN code. The grammar file is under parser/src/.../expression.g, IIRC.