Questions about the Java code...

Progress reports and musings from the developers on the current gaming tools.

Moderators: dorpond, trevor, Azhrei

Post Reply
The_Grand_User
Kobold
Posts: 14
Joined: Tue Nov 18, 2008 6:22 pm

Questions about the Java code...

Post by The_Grand_User »

Originally posted to this thread. -- Azhrei

Hmm, there is the occasional place where it's a bit tricky to figure out what the intent of the code was, but I can generally figure it out. However, I found one spot that may be a bug in the original code, AssetManager.rememberLocalImageReference. It looks to be chcking the lnkfile whether it has the MD5Key of the image file as a single line somewhere in there, but if it doesn't find it then it sticks in the image file's full path in. I am presuming that it should be checking whether the full path is in there and not the id.

Here is the full code for reference

Code: Select all

public static void rememberLocalImageReference(File image) throws IOException {

	MD5Key id = new MD5Key(new BufferedInputStream(new FileInputStream(image)));
	File lnkFile = getAssetLinkFile(id);

	// See if we know about this one already
	if (lnkFile.exists()) {

		List<String> referenceList = FileUtil.getLines(lnkFile);
		for (String ref : referenceList) {
			if (ref.equals(id.toString())) {

				// We already know about this one
				return;
			}
		}
	}

	// Keep track of this reference
	FileOutputStream out = new FileOutputStream(lnkFile, true); // For appending

	out.write((image.getAbsolutePath() + "\n").getBytes());

	out.close();
}
For future reference, should I continue asking for clarifications in this thread, or would another thread be better?

Answer: Post them in the Developer Notes forum. I've moved this question there. -- Azhrei

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Questions about the Java code...

Post by Azhrei »

I can't answer that off-hand. I don't understand the use of the .lnk file. I think it was supposed to contain the pathname to any image that was used but wasn't available in any library location. So for instance, an image dragged from a web browser and dropped into MapTool would generate one of these (it's not in any Asset Library, nor any repository). In that case, it would contain whatever id the browser generated during the drag operation. When dragging from the filesystem it would have the pathname.

I don't know the above to be fact, though. The image management stuff was written by Trevor a long time ago and I've never understood certain parts of it (like this part) or even why it was done (once in MapTool, everything is identified by using an asset ID, so why the .lnk file at all?).

username
Dragon
Posts: 277
Joined: Sun Sep 04, 2011 7:01 am

Re: Questions about the Java code...

Post by username »

I'm with The_Grand_User on this one. The net effect will be that (duplicate) lines will be added to the lnk-file. I also don't know what they are used for. If you happen to know how this piece of code is triggered, I would wager you can put some print in the innermost if-clause and never see output from it, no matter what you do. A comment to this observation should be made there, even though you probably won't touch the code itself, Azhrei. When code is refactored, this should not be lost.

The_Grand_User
Kobold
Posts: 14
Joined: Tue Nov 18, 2008 6:22 pm

Re: Questions about the Java code...

Post by The_Grand_User »

Oh, heh, this is where it went.

Ok, I'll keep this in mind, and leave my change in the C# code.
once in MapTool, everything is identified by using an asset ID, so why the .lnk file at all?
Probably because that's where the only reference to the file location is stored, if as you say it's for drag & drop, rather than from a library.

I'll see what's using it once I get the thing converted and running. I suppose I might be the first one to have gone through the code with a fine toothed comb for a long time, and there'll be plenty of other spots that look odd ;)

And yes, I'm not going to touch the official Java code at all.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Questions about the Java code...

Post by Azhrei »

The_Grand_User wrote:Probably because that's where the only reference to the file location is stored, if as you say it's for drag & drop, rather than from a library.
Yeah, but I see no use for that. So what if the asset came from a d/d operation? All that matters is that MT has it now...

The_Grand_User
Kobold
Posts: 14
Joined: Tue Nov 18, 2008 6:22 pm

Re: Questions about the Java code...

Post by The_Grand_User »

I've yet to get to the inner workings of saving campaigns, but how else is that handled, except by having some well known place where you can look up the images location after you have closed maptool and opened it again?

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Questions about the Java code...

Post by Azhrei »

When an image is dropped onto an MT map, the contents of the image are copied to a file under the assetcache/ directory. The name of that file is based on the MD5 checksum of the content, resulting in a 32-character filename that consists of hexadecimal characters (0-9 and a-f).

Inside MapTool all references to images are made via that MD5 value (called the MD5Key in the code). When an image is associated with a token, the MD5Key is what is really associated with it, and when the token needs to be drawn the MD5Key is used as an index into a table of all images currently in use (meaning on the current map). Note that the image itself (ie. the content of the JPEG file) is never actually stored on the Token. This is actually quite important because it allows the same image to be referenced from multiple places without complicating the code by having to point "into" the Token object.

Once the file is created in the assetcache directory there's no further need to reference the name of the image. Note that dropping an image onto the map causes a Preference to be checked as to whether the Token is named "Creature" or is named based on the dropped image, but that's the Token, not the image itself. Dropping the image causes a new image to be written to the assetcache using its MD5 checksum and a corresponding Token object is simultaneously created that references that same MD5Key.

Anytime the map is panned or zoomed or otherwise needs to be refreshed, the position and size of each token is determined and the corresponding MD5Key is looked up and the image retrieved so that it can be blitted to the map buffer before being sent to the screen.

(Most of the above is pretty typical for a graphics-oriented application. The MT-specific parts are the storage of the graphical elements in the assetcache directory and the use of the MD5 checksum as the key.)

When a campaign is saved, each map is retrieved one at a time. From within the map, all tokens are retrieved and written in XML. Those tokens will contain MD5Key XML elements inside them. When all tokens have been written, then all images referenced by the map are also retrieved and they are written to the campaign file (using no compression) into the assets/ subdirectory of the ZIP file.

When a campaign is loaded, ALL assets from the assets directory are copied to the assetcache even if the campaign's XML doesn't reference them (so editing the XML isn't enough to delete graphical assets; the campaign must be saved and then reloaded again, with the save being the only time that the campaign's assets are determined). The reason all assets are copied to the assetcache has to do with both consistency in regards to how the campaign is saved, but also because it isn't worthwhile to search the ZIP directory for each MD5 key over and over when the 99% case is going to be that all assets will be needed anyway.

Whew. Hope that helps. ;)

The_Grand_User
Kobold
Posts: 14
Joined: Tue Nov 18, 2008 6:22 pm

Re: Questions about the Java code...

Post by The_Grand_User »

That's quite a bit of info, thanks :)

username
Dragon
Posts: 277
Joined: Sun Sep 04, 2011 7:01 am

Re: Questions about the Java code...

Post by username »

While you are in the mood for explaining, I have seen the following "bug": I created a lot of tokens by hand. In order to save time, I created a script that turned the relevant attributes and images from another format into an rptok file, which I later dropped on my map. But instead of using MD5 (I didn't know then how those fancy numbers came about) I used UUIDs. That all worked fine for a while. The second (or third?) time I opened the campaign, all images were represented as questionmarks, although everything was available. I cleaned out the cache to no avail. But maybe the cleaning of the cache - which I guess was done by MT itself the first time around - was the root cause. Is there any reason why another hash than MD5 could fail (in this manner)?

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: Questions about the Java code...

Post by Azhrei »

username wrote:I cleaned out the cache to no avail. But maybe the cleaning of the cache - which I guess was done by MT itself the first time around - was the root cause.
No, MT never cleans the assetcache directory (I assume that's what you're referring to?). Which is itself a bit of a bug. At some point MapTool should either clear that cache or at least notify the user when it hits some configurable size limit. Otherwise most users will never even know it exists until it's multiple gigabytes in size and they run out of disk space!
Is there any reason why another hash than MD5 could fail (in this manner)?
I can't think of anything off-hand. But if the name of the file were ever lost, MapTool would assume that it could simply recalculate the MD5 checksum and that's what the name would be. It sounds like you ran into such a situation.

Maybe when campaigns are loaded the MD5Key stored in the Token is compared to the asset name, which is then compared to the calculated checksum? I do remember seeing a comment in the code that if the MD5Key is ever wrong that the whole house of cards could come tumbling down. Apparently Trevor was pretty careful about that. ;)

The complete operation of the AssetManager, AssetLoader, ImageManager, and ImageLoader classes is a little opaque to me -- I'm not sure exactly what all the relationships are. I was able to expound on some of this in my last message because I rewrote a good portion of the campaign load/save stuff awhile back in order to support localization, but AFAIAC much of the asset management stuff is all "black box".

I also looked for some way to add a search feature to the Asset Panel GUI, but all images are stored only by MD5Key and there was no practical way to add a search. It was at that point that I decided those classes needed to be cleaned up and heavily documented in 1.4 or a lot of the knowledge was going to be lost. :|

Post Reply

Return to “Developer Notes”