RPTools.net

Discussion and Support

Skip to content

It is currently Wed Oct 18, 2017 2:13 am 




Forum rules


Please discuss all HOWTO topics in the main forum. They will be moved here by a moderator when appropriate.



Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

Previous topic | Next topic 

  Print view

Author Message
 Offline
Great Wyrm
 
Joined: Sun Jun 22, 2008 6:53 pm
Posts: 2102
Location: Melbourne, Australia
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Mon Jan 12, 2009 7:38 pm 
Phergus wrote:

The users could just dump the contents of the assets directory from the exported repository file but asking the typical user to do this is asking for trouble.

Being able to import a exported repository file or being able to specify additional local local repositories would probably be good.


I don't disagree with you, was kinda just musing out loud (so to speak), thinking about how it could work right now. :) Something I will have to put on my to look at list but thats starting to get too long.


Top
 Profile  
 
User avatar  Offline
Deity
 
Joined: Fri May 12, 2006 7:56 pm
Posts: 6300
Location: Las Cruces, NM
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Mon Jan 12, 2009 8:11 pm 
That will teach you to demonstrate competence. :wink:

_________________
Call me, Hex Boy.

Visit http://www.darkisle.com for photos of castles, cairns, standing stones, and other ancient sites in the British Isles.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 1:08 pm 
Is there an online repository for the 2+Gig mapping_objects.zip for maptool?

If not, how do you calculate the MD5 values, just hash the files in-place?

(I'm contemplating uploading it to my site.)

Thanks,
IMarv


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 3:23 pm 
IMarvinTPA wrote:
Is there an online repository for the 2+Gig mapping_objects.zip for maptool?

The Downloads link on the main site has a link to the torrent tracker and a link to the thread here on the forum that talks about it.

Quote:
If not, how do you calculate the MD5 values, just hash the files in-place?

(I'm contemplating uploading it to my site.)

Windows does not come with an MD5 checksum utility that is practical for renaming the files. Perl has an MD5 module that can calculate checksums and Perl runs on Windows. Other languages probably have similar tools.

On Unix/Linux it's one command line:
Code:
find . -type f | xargs -L 1 -I {} bash -c 'mv "{}" $(md5 -q "{}")'

Of course, that assumes that your xargs command uses the same options as OSX does and that your MD5 checksum program is called md5 and has a "quiet" option.

After you enter that command, go get a BIG cup of coffee and watch a football game or something. I count 21,766 files in the mapping_objects.zip archive and almost 3GB when unpacked from the ZIP. That's a lot of data to be calculating a 128-bit checksum on...

(Edit: Just out of curiosity I think I'll do it. I won't be renaming the files, just figuring out the checksum. I'll let you know how long it takes. :))


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 3:32 pm 
I'm trying to upload the whole of it. The plan is to put it at http://www.imarvintpa.com/Mapping

I just can't seem to manage to FTP a 3GB file, so I have to chunk it down to smaller zips.

Thank you for the commands, I'll see if they work once the files are up, this weekend...

IMarv


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 4:45 pm 
Okay, minus the actual rename operation, calculating checksums came to just 7m05.562s. Not bad considering how many files there are...

Put the attached script on your Unix/Linux machine in the top-level directory of the extracted ZIP file. On my system, that's a directory called "Mapping Objects" (yeah, I like to keep things simple!). The forum won't let me attach a filename with a .sh extension, so I zipped it up. You'll need to unpack it either on your machine or on the Unix side. Post again if you need more info on this.

I generated a list of mv commands that accomplish the deed. So all you need to do is execute the script. There are four steps and a message is displayed as each one begins. Expect step two (the rename step) to take a while to finish. Step 1 is to create a subdirectory named assets. Step 2 is the rename operation. Step 3 is to create the index.gz file needed by MapTool. Step 4 is to find all directories that are now empty and remove them.

If you get any output other than the status messages, an error has occurred and the script has terminated. Cut/paste the text from the last "Step" message and anything following it to another post here and we can figure out what to do next.

You do not need to change the permissions on the script at all in order to run it:

Code:
bash ./rename.sh


File comment: Script to rename images for repository use
rename.zip [568.97 KiB]
Downloaded 118 times
Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 11:11 pm 
The script is pretty cool, but I don't think you need to rename the files for the assets to work since the second column in the index file is simply the relative path to the file. I'll see if I can rework this into what I'm hoping for.

IMarv


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Thu Feb 05, 2009 11:27 pm 
Question for Trevor,
How does the index file work when the file name contains spaces?

IE:
Code:
6333b2e265f9843ecc4ff6c731281557 Compass Roses/compass_rose_silver_beve_bw.png
7a83efafe9fff8105b58c35c827aa0ee Enemies/Ant Spiders/antspiderumber_hybrid2_bw.png
8b1c663a7c0270cab3aba810def8cfc4 Enemies/Ant Spiders/antspiderumber_hybrid_bw.p


Thanks,
IMarv


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Fri Feb 06, 2009 12:25 am 
Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java:

Code:
   protected Map<String, String> parseIndex(List<String> index) {
 
       Map<String, String> idxMap = new HashMap<String, String>();
 
       for (String line : index) {
 
          String id = line.substring(0, 32);
          String ref = line.substring(33).trim();
 
          idxMap.put(id, ref);
       }
 
       return idxMap;
    }


The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.

All that is left is to get all the files posted.

Thanks All,
IMarv


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Fri Feb 06, 2009 2:26 am 
IMarvinTPA wrote:
Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java

Sorry, I've been AFK and didn't see your posts. I could've told you that. :(

Quote:
The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.

What needed to be rearranged? The mv's were all of files, so they were not order-dependent.

Just curious in case I need to do something like this again some time. :)

And kudos, Marvin. This will be very useful!

Edit: Oh, I think I see. You swapped the position of the pathname with the checksum so that the index.gz had the original names. Hmm, that should probably work. I know there are situations where MT finds the entry in the AssetMap by calculating the checksum of the image. If it simply replaces the entry in the Map with a new value under some condition, it may point to a non-existent pathname as the value. But what the heck -- try it and see. But be sure to try it first and see if it works, then shutdown MT and empty your assetcache and try again.


Top
 Profile  
 
User avatar  Offline
Demigod
 
Joined: Sun Feb 25, 2007 11:53 am
Posts: 3726
Location: MD
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Fri Feb 06, 2009 3:28 am 
Any chance in the future that all of this could be done with a single click solution?

1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."

This would be a much more useful feature if it required less technical competency.


It would also be very nice to be able to see at a glance what assets in your library were on a repository.

_________________
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Fri Feb 06, 2009 7:29 pm 
Full Bleed wrote:
Any chance in the future that all of this could be done with a single click solution?

1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."

This would be a much more useful feature if it required less technical competency.

True. Short answer: yes, it can be done.

Quote:
It would also be very nice to be able to see at a glance what assets in your library were on a repository.

Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?


Top
 Profile  
 
User avatar  Offline
Demigod
 
Joined: Sun Feb 25, 2007 11:53 am
Posts: 3726
Location: MD
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Sat Feb 07, 2009 12:48 am 
Azhrei wrote:
Quote:
It would also be very nice to be able to see at a glance what assets in your library were on a repository.

Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?


How does it know when to look at a repository now for a particular asset?

_________________
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Sat Feb 07, 2009 1:02 am 
I think I misunderstood your question. :( Using the checksum, it is possible to determine which image came from which repository. It's not possible to know if it was actually downloaded from there or already existed on the local machine in the assetcache directory.

Background:

When you drag an image onto the map, the checksum is calculated and stored in a list. In addition, the image is stored in the assetcache directory under the same name as the checksum. Whenever the token needs to be displayed, the checksum is retrieved from the token. This is then used as the asset name and the file in the assetcache directory is opened and read into memory, stored in a BufferedImage object. (I'm not 100% positive on some of this because it's Trevor tricky code!)

Since the repository is remote, I'm thinking that when the campaign is loaded, so are the index.gz files from each repository URL. Those files have a list of checksums and relative pathnames on their server. Now when MapTool needs a particular image (i.e. checksum), it first searches in memory to see if the BufferedImage object is already loaded. If not, it looks in the local assetcache. Failing that, it looks through each of the index.gz files one at a time. The first one that has the checksum in it identifies the URL to use and the filename relative to the index.gz file on the server. That file is grabbed off the server and BufferedImage is created to represent it. At the same time, it's stored in the local assetcache so that the download can be avoided the next time the image is needed and there isn't a BufferedImage for it. This might happen when you change maps -- all image data for the current map is thrown away and the assets from the new map are loaded.

As I said, this is a fairly tricky section of code because the images are also sent asynchronously over the network to other hosts on an as-needed basis. As Trevor posted recently, the long-standing "?" bug was a result of this section of code.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Mon Sep 11, 2006 9:12 am
Posts: 468
 Post subject: Re: HOWTO: Put your images on a webserver
PostPosted: Sat Feb 07, 2009 5:29 pm 
I haven't had time to verify that all the files have been uploaded, nor if it works as intended yet, but the upload is reportedly complete. I'll verify when I have some time.

http://www.imarvintpa.com/Mapping/index.gz

IMarv


Top
 Profile  
 
Display posts from previous:  Sort by  
Reply to topic  [ 85 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 243 on Sun Nov 04, 2012 6:14 am

Users browsing this forum: No registered users and 1 guest





Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group

Style based on Andreas08 by Andreas Viklund

Style by Elizabeth Shulman