MapTool 1.3 Bugfix Build 65
Moderators: dorpond, trevor, Azhrei, Craig
Re: MapTool 1.3 Bugfix Build 65
Excellent! I'll take a look at your macro tomorrow (the rest of the today is likely to be pretty busy).
Re: MapTool 1.3 Bugfix Build 65
Okay, so I'm looking into this one and I have a question, but first some background.Natha wrote:Another one : importing macroset (that has been previously exported with b63) with accents in macros' name or coding messes the strings :
It appears that the text fields where the user types in the macro name and all the other stuff uses whatever character encoding is the default for the platform, and thus, for MapTool. When the macro is exported, it therefore has the same encoding when converted to XML. The XML data is then read back in and the same characters are created. Those characters are stored into the various fields and then show up incorrectly.
Here's what the data in the test macro's XML looks like:
As you can guess, the characters shown in reverse video are just a single byte of data. It appears that you're using the latin-1 character set as UTF-8 would have encoded the same symbol as a two-byte sequence.
In any case, the character set being used is not written into the XML and is not restored when reading. If the XML is written and read by the same machine, I would expect it to work anyway but there must be some kind of auto-conversion going on.
Please identify which character set you're using, which encoding you're using, and any other locale-related information. I may have to play around with this one a bit.
If anyone out there is a Java programmer that deals with internationalization on a regular basis, I could use some help with this one as this is unfamiliar territory for me!
- jfrazierjr
- Deity
- Posts: 5176
- Joined: Tue Sep 11, 2007 7:31 pm
Re: MapTool 1.3 Bugfix Build 65
Well, if Maptool directly calls a decent XML parser, you should be able to set the encoding header of the XML as well as the filestream encoding(UTF-8) and I would expect that would solve the problem. But I expect it is being serialized totally through the abstraction layer (I can't think of it for the life of me right now), so we are limited to what it exposes.Azhrei wrote:Okay, so I'm looking into this one and I have a question, but first some background.Natha wrote:Another one : importing macroset (that has been previously exported with b63) with accents in macros' name or coding messes the strings :
It appears that the text fields where the user types in the macro name and all the other stuff uses whatever character encoding is the default for the platform, and thus, for MapTool. When the macro is exported, it therefore has the same encoding when converted to XML. The XML data is then read back in and the same characters are created. Those characters are stored into the various fields and then show up incorrectly.
Here's what the data in the test macro's XML looks like:
As you can guess, the characters shown in reverse video are just a single byte of data. It appears that you're using the latin-1 character set as UTF-8 would have encoded the same symbol as a two-byte sequence.
In any case, the character set being used is not written into the XML and is not restored when reading. If the XML is written and read by the same machine, I would expect it to work anyway but there must be some kind of auto-conversion going on.
Please identify which character set you're using, which encoding you're using, and any other locale-related information. I may have to play around with this one a bit.
If anyone out there is a Java programmer that deals with internationalization on a regular basis, I could use some help with this one as this is unfamiliar territory for me!
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Re: MapTool 1.3 Bugfix Build 65
Well, it doesn't work like that. Or at least, not that easily.jfrazierjr wrote:Well, if Maptool directly calls a decent XML parser, you should be able to set the encoding header of the XML as well as the filestream encoding(UTF-8) and I would expect that would solve the problem.
http://xstream.codehaus.org/faq.html
The problem seems to be that some portions of the MapTool code give XStream a String and some give it an InputStream. When given an InputStream it can extract the encoding information and write that encoding information to the output file. With a String it seems you're out of luck.But I expect it is being serialized totally through the abstraction layer (I can't think of it for the life of me right now), so we are limited to what it exposes.
I'm going to look through the code and see if the spots that provide a String can be easily converted to using an InputStream. But I'm shooting in the dark here since I'm not intimately familiar with the rules for when character encoding should be performed and when it shouldn't. (I know the basics about the difference between a Reader and an InputStream, and a Writer and a OutputStream, but that's only a small part of the picture apparently.)
But as I said, I'm going to see what I can do about it. Worst case I should be able to create an ArrayInputStream from the bytes stored in the String and use that...
Re: MapTool 1.3 Bugfix Build 65
It required some code restructuring but I think I've got it working now. Whenever an XML object is saved, it will be written using the current locale. The same thing applies when reading the object back in. This should preserve the language encoding. I tested it with Natha's macro and it looks fine. (Although I can't seem to enter accented characters on my Mac. I normally use Cmd-E followed by the letter, such as "A", to enter an accented A, yet it doesn't seem to work anywhere in MapTool. It may have something to do with the BasicUIMac library grabbing the Cmd keystrokes. Sigh.)
However, the XStream library does not recognize the character encoding as they claim that's the job of the parser, not the serializer. MapTool uses the Xpp3 parser so now I'm off to figure out if there's an option for making them play nicely together.
Natha, this means that macros you save should be read back in properly. But macros you save may not be read back in properly when the user has a different locale than yours. If you stick with latin-1 a.k.a. ISO8859-1 and the other user is doing the same, then there should be a lot of portability between MapTool instances. I'll have more information when I get some time to research Xpp3 in detail.
However, the XStream library does not recognize the character encoding as they claim that's the job of the parser, not the serializer. MapTool uses the Xpp3 parser so now I'm off to figure out if there's an option for making them play nicely together.
Natha, this means that macros you save should be read back in properly. But macros you save may not be read back in properly when the user has a different locale than yours. If you stick with latin-1 a.k.a. ISO8859-1 and the other user is doing the same, then there should be a lot of portability between MapTool instances. I'll have more information when I get some time to research Xpp3 in detail.