MapTool 1.3 Bugfix Build 65

New build announcements plus site news and changes.

Moderators: dorpond, trevor, Azhrei, Craig

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool 1.3 Bugfix Build 65

Post by Azhrei »

Excellent! I'll take a look at your macro tomorrow (the rest of the today is likely to be pretty busy).

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool 1.3 Bugfix Build 65

Post by Azhrei »

Natha wrote:Another one : importing macroset (that has been previously exported with b63) with accents in macros' name or coding messes the strings :
Image
Okay, so I'm looking into this one and I have a question, but first some background.

It appears that the text fields where the user types in the macro name and all the other stuff uses whatever character encoding is the default for the platform, and thus, for MapTool. When the macro is exported, it therefore has the same encoding when converted to XML. The XML data is then read back in and the same characters are created. Those characters are stored into the various fields and then show up incorrectly.

Here's what the data in the test macro's XML looks like:
XML for macro.png
XML for macro.png (44.99 KiB) Viewed 1207 times
As you can guess, the characters shown in reverse video are just a single byte of data. It appears that you're using the latin-1 character set as UTF-8 would have encoded the same symbol as a two-byte sequence.

In any case, the character set being used is not written into the XML and is not restored when reading. If the XML is written and read by the same machine, I would expect it to work anyway but there must be some kind of auto-conversion going on.

Please identify which character set you're using, which encoding you're using, and any other locale-related information. I may have to play around with this one a bit.

If anyone out there is a Java programmer that deals with internationalization on a regular basis, I could use some help with this one as this is unfamiliar territory for me! :?

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: MapTool 1.3 Bugfix Build 65

Post by jfrazierjr »

Azhrei wrote:
Natha wrote:Another one : importing macroset (that has been previously exported with b63) with accents in macros' name or coding messes the strings :
Image
Okay, so I'm looking into this one and I have a question, but first some background.

It appears that the text fields where the user types in the macro name and all the other stuff uses whatever character encoding is the default for the platform, and thus, for MapTool. When the macro is exported, it therefore has the same encoding when converted to XML. The XML data is then read back in and the same characters are created. Those characters are stored into the various fields and then show up incorrectly.

Here's what the data in the test macro's XML looks like:
XML for macro.png
As you can guess, the characters shown in reverse video are just a single byte of data. It appears that you're using the latin-1 character set as UTF-8 would have encoded the same symbol as a two-byte sequence.

In any case, the character set being used is not written into the XML and is not restored when reading. If the XML is written and read by the same machine, I would expect it to work anyway but there must be some kind of auto-conversion going on.

Please identify which character set you're using, which encoding you're using, and any other locale-related information. I may have to play around with this one a bit.

If anyone out there is a Java programmer that deals with internationalization on a regular basis, I could use some help with this one as this is unfamiliar territory for me! :?
Well, if Maptool directly calls a decent XML parser, you should be able to set the encoding header of the XML as well as the filestream encoding(UTF-8) and I would expect that would solve the problem. But I expect it is being serialized totally through the abstraction layer (I can't think of it for the life of me right now), so we are limited to what it exposes.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool 1.3 Bugfix Build 65

Post by Azhrei »

jfrazierjr wrote:Well, if Maptool directly calls a decent XML parser, you should be able to set the encoding header of the XML as well as the filestream encoding(UTF-8) and I would expect that would solve the problem.
Well, it doesn't work like that. :? Or at least, not that easily.

http://xstream.codehaus.org/faq.html
But I expect it is being serialized totally through the abstraction layer (I can't think of it for the life of me right now), so we are limited to what it exposes.
The problem seems to be that some portions of the MapTool code give XStream a String and some give it an InputStream. When given an InputStream it can extract the encoding information and write that encoding information to the output file. With a String it seems you're out of luck. :(

I'm going to look through the code and see if the spots that provide a String can be easily converted to using an InputStream. But I'm shooting in the dark here since I'm not intimately familiar with the rules for when character encoding should be performed and when it shouldn't. (I know the basics about the difference between a Reader and an InputStream, and a Writer and a OutputStream, but that's only a small part of the picture apparently.)

But as I said, I'm going to see what I can do about it. Worst case I should be able to create an ArrayInputStream from the bytes stored in the String and use that...

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: MapTool 1.3 Bugfix Build 65

Post by Azhrei »

It required some code restructuring but I think I've got it working now. Whenever an XML object is saved, it will be written using the current locale. The same thing applies when reading the object back in. This should preserve the language encoding. I tested it with Natha's macro and it looks fine. (Although I can't seem to enter accented characters on my Mac. I normally use Cmd-E followed by the letter, such as "A", to enter an accented A, yet it doesn't seem to work anywhere in MapTool. It may have something to do with the BasicUIMac library grabbing the Cmd keystrokes. Sigh.)

However, the XStream library does not recognize the character encoding as they claim that's the job of the parser, not the serializer. MapTool uses the Xpp3 parser so now I'm off to figure out if there's an option for making them play nicely together. :)

Natha, this means that macros you save should be read back in properly. But macros you save may not be read back in properly when the user has a different locale than yours. If you stick with latin-1 a.k.a. ISO8859-1 and the other user is doing the same, then there should be a lot of portability between MapTool instances. I'll have more information when I get some time to research Xpp3 in detail.

Post Reply

Return to “Announcements”