HOWTO: Put your images on a webserver (Repository)

Articles on how to do things in MapTool (answers only here -- questions should stay in the main MT forum)

Moderators: dorpond, trevor, Azhrei, giliath, jay, Mr.Ice

Forum rules
Please discuss all HOWTO topics in the main forum. They will be moved here by a moderator when appropriate.
Craig
Great Wyrm
Posts: 2107
Joined: Sun Jun 22, 2008 7:53 pm
Location: Melbourne, Australia

Re: HOWTO: Put your images on a webserver

Post by Craig »

Phergus wrote:
The users could just dump the contents of the assets directory from the exported repository file but asking the typical user to do this is asking for trouble.

Being able to import a exported repository file or being able to specify additional local local repositories would probably be good.
I don't disagree with you, was kinda just musing out loud (so to speak), thinking about how it could work right now. :) Something I will have to put on my to look at list but thats starting to get too long.

Phergus
Deity
Posts: 7132
Joined: Fri May 12, 2006 8:56 pm
Location: Middle of Nowhere, NM
Contact:

Re: HOWTO: Put your images on a webserver

Post by Phergus »

That will teach you to demonstrate competence. :wink:

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

Is there an online repository for the 2+Gig mapping_objects.zip for maptool?

If not, how do you calculate the MD5 values, just hash the files in-place?

(I'm contemplating uploading it to my site.)

Thanks,
IMarv

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: HOWTO: Put your images on a webserver

Post by Azhrei »

IMarvinTPA wrote:Is there an online repository for the 2+Gig mapping_objects.zip for maptool?
The Downloads link on the main site has a link to the torrent tracker and a link to the thread here on the forum that talks about it.
If not, how do you calculate the MD5 values, just hash the files in-place?

(I'm contemplating uploading it to my site.)
Windows does not come with an MD5 checksum utility that is practical for renaming the files. Perl has an MD5 module that can calculate checksums and Perl runs on Windows. Other languages probably have similar tools.

On Unix/Linux it's one command line:

Code: Select all

find . -type f | xargs -L 1 -I {} bash -c 'mv "{}" $(md5 -q "{}")'
Of course, that assumes that your xargs command uses the same options as OSX does and that your MD5 checksum program is called md5 and has a "quiet" option.

After you enter that command, go get a BIG cup of coffee and watch a football game or something. I count 21,766 files in the mapping_objects.zip archive and almost 3GB when unpacked from the ZIP. That's a lot of data to be calculating a 128-bit checksum on...

(Edit: Just out of curiosity I think I'll do it. I won't be renaming the files, just figuring out the checksum. I'll let you know how long it takes. :))

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

I'm trying to upload the whole of it. The plan is to put it at http://www.imarvintpa.com/Mapping

I just can't seem to manage to FTP a 3GB file, so I have to chunk it down to smaller zips.

Thank you for the commands, I'll see if they work once the files are up, this weekend...

IMarv

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: HOWTO: Put your images on a webserver

Post by Azhrei »

Okay, minus the actual rename operation, calculating checksums came to just 7m05.562s. Not bad considering how many files there are...

Put the attached script on your Unix/Linux machine in the top-level directory of the extracted ZIP file. On my system, that's a directory called "Mapping Objects" (yeah, I like to keep things simple!). The forum won't let me attach a filename with a .sh extension, so I zipped it up. You'll need to unpack it either on your machine or on the Unix side. Post again if you need more info on this.

I generated a list of mv commands that accomplish the deed. So all you need to do is execute the script. There are four steps and a message is displayed as each one begins. Expect step two (the rename step) to take a while to finish. Step 1 is to create a subdirectory named assets. Step 2 is the rename operation. Step 3 is to create the index.gz file needed by MapTool. Step 4 is to find all directories that are now empty and remove them.

If you get any output other than the status messages, an error has occurred and the script has terminated. Cut/paste the text from the last "Step" message and anything following it to another post here and we can figure out what to do next.

You do not need to change the permissions on the script at all in order to run it:

Code: Select all

bash ./rename.sh
Attachments
rename.zip
Script to rename images for repository use
(568.97 KiB) Downloaded 215 times

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

The script is pretty cool, but I don't think you need to rename the files for the assets to work since the second column in the index file is simply the relative path to the file. I'll see if I can rework this into what I'm hoping for.

IMarv

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

Question for Trevor,
How does the index file work when the file name contains spaces?

IE:

Code: Select all

6333b2e265f9843ecc4ff6c731281557 Compass Roses/compass_rose_silver_beve_bw.png
7a83efafe9fff8105b58c35c827aa0ee Enemies/Ant Spiders/antspiderumber_hybrid2_bw.png
8b1c663a7c0270cab3aba810def8cfc4 Enemies/Ant Spiders/antspiderumber_hybrid_bw.p
Thanks,
IMarv

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java:

Code: Select all

	protected Map<String, String> parseIndex(List<String> index) {
 
 		Map<String, String> idxMap = new HashMap<String, String>();
 
 		for (String line : index) {
 
 			String id = line.substring(0, 32);
 			String ref = line.substring(33).trim();
 
 			idxMap.put(id, ref);
 		}
 
 		return idxMap;
 	}
The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.

All that is left is to get all the files posted.

Thanks All,
IMarv

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: HOWTO: Put your images on a webserver

Post by Azhrei »

IMarvinTPA wrote:Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java
Sorry, I've been AFK and didn't see your posts. I could've told you that. :(
The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.
What needed to be rearranged? The mv's were all of files, so they were not order-dependent.

Just curious in case I need to do something like this again some time. :)

And kudos, Marvin. This will be very useful!

Edit: Oh, I think I see. You swapped the position of the pathname with the checksum so that the index.gz had the original names. Hmm, that should probably work. I know there are situations where MT finds the entry in the AssetMap by calculating the checksum of the image. If it simply replaces the entry in the Map with a new value under some condition, it may point to a non-existent pathname as the value. But what the heck -- try it and see. But be sure to try it first and see if it works, then shutdown MT and empty your assetcache and try again.

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: HOWTO: Put your images on a webserver

Post by Full Bleed »

Any chance in the future that all of this could be done with a single click solution?

1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."

This would be a much more useful feature if it required less technical competency.


It would also be very nice to be able to see at a glance what assets in your library were on a repository.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: HOWTO: Put your images on a webserver

Post by Azhrei »

Full Bleed wrote:Any chance in the future that all of this could be done with a single click solution?

1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."

This would be a much more useful feature if it required less technical competency.
True. Short answer: yes, it can be done.
It would also be very nice to be able to see at a glance what assets in your library were on a repository.
Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?

User avatar
Full Bleed
Demigod
Posts: 4736
Joined: Sun Feb 25, 2007 11:53 am
Location: FL

Re: HOWTO: Put your images on a webserver

Post by Full Bleed »

Azhrei wrote:
It would also be very nice to be able to see at a glance what assets in your library were on a repository.
Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?
How does it know when to look at a repository now for a particular asset?
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: HOWTO: Put your images on a webserver

Post by Azhrei »

I think I misunderstood your question. :( Using the checksum, it is possible to determine which image came from which repository. It's not possible to know if it was actually downloaded from there or already existed on the local machine in the assetcache directory.

Background:

When you drag an image onto the map, the checksum is calculated and stored in a list. In addition, the image is stored in the assetcache directory under the same name as the checksum. Whenever the token needs to be displayed, the checksum is retrieved from the token. This is then used as the asset name and the file in the assetcache directory is opened and read into memory, stored in a BufferedImage object. (I'm not 100% positive on some of this because it's Trevor tricky code!)

Since the repository is remote, I'm thinking that when the campaign is loaded, so are the index.gz files from each repository URL. Those files have a list of checksums and relative pathnames on their server. Now when MapTool needs a particular image (i.e. checksum), it first searches in memory to see if the BufferedImage object is already loaded. If not, it looks in the local assetcache. Failing that, it looks through each of the index.gz files one at a time. The first one that has the checksum in it identifies the URL to use and the filename relative to the index.gz file on the server. That file is grabbed off the server and BufferedImage is created to represent it. At the same time, it's stored in the local assetcache so that the download can be avoided the next time the image is needed and there isn't a BufferedImage for it. This might happen when you change maps -- all image data for the current map is thrown away and the assets from the new map are loaded.

As I said, this is a fairly tricky section of code because the images are also sent asynchronously over the network to other hosts on an as-needed basis. As Trevor posted recently, the long-standing "?" bug was a result of this section of code.

User avatar
IMarvinTPA
Dragon
Posts: 467
Joined: Mon Sep 11, 2006 10:12 am
Contact:

Re: HOWTO: Put your images on a webserver

Post by IMarvinTPA »

I haven't had time to verify that all the files have been uploaded, nor if it works as intended yet, but the upload is reportedly complete. I'll verify when I have some time.

http://www.imarvintpa.com/Mapping/index.gz

IMarv

Post Reply

Return to “How To”