I don't disagree with you, was kinda just musing out loud (so to speak), thinking about how it could work right now. Something I will have to put on my to look at list but thats starting to get too long.Phergus wrote:
The users could just dump the contents of the assets directory from the exported repository file but asking the typical user to do this is asking for trouble.
Being able to import a exported repository file or being able to specify additional local local repositories would probably be good.
HOWTO: Put your images on a webserver (Repository)
Moderators: dorpond, trevor, Azhrei, giliath, jay, Mr.Ice
Forum rules
Please discuss all HOWTO topics in the main forum. They will be moved here by a moderator when appropriate.
Please discuss all HOWTO topics in the main forum. They will be moved here by a moderator when appropriate.
Re: HOWTO: Put your images on a webserver
Re: HOWTO: Put your images on a webserver
That will teach you to demonstrate competence.
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
Is there an online repository for the 2+Gig mapping_objects.zip for maptool?
If not, how do you calculate the MD5 values, just hash the files in-place?
(I'm contemplating uploading it to my site.)
Thanks,
IMarv
If not, how do you calculate the MD5 values, just hash the files in-place?
(I'm contemplating uploading it to my site.)
Thanks,
IMarv
Re: HOWTO: Put your images on a webserver
The Downloads link on the main site has a link to the torrent tracker and a link to the thread here on the forum that talks about it.IMarvinTPA wrote:Is there an online repository for the 2+Gig mapping_objects.zip for maptool?
Windows does not come with an MD5 checksum utility that is practical for renaming the files. Perl has an MD5 module that can calculate checksums and Perl runs on Windows. Other languages probably have similar tools.If not, how do you calculate the MD5 values, just hash the files in-place?
(I'm contemplating uploading it to my site.)
On Unix/Linux it's one command line:
Code: Select all
find . -type f | xargs -L 1 -I {} bash -c 'mv "{}" $(md5 -q "{}")'
After you enter that command, go get a BIG cup of coffee and watch a football game or something. I count 21,766 files in the mapping_objects.zip archive and almost 3GB when unpacked from the ZIP. That's a lot of data to be calculating a 128-bit checksum on...
(Edit: Just out of curiosity I think I'll do it. I won't be renaming the files, just figuring out the checksum. I'll let you know how long it takes. )
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
I'm trying to upload the whole of it. The plan is to put it at http://www.imarvintpa.com/Mapping
I just can't seem to manage to FTP a 3GB file, so I have to chunk it down to smaller zips.
Thank you for the commands, I'll see if they work once the files are up, this weekend...
IMarv
I just can't seem to manage to FTP a 3GB file, so I have to chunk it down to smaller zips.
Thank you for the commands, I'll see if they work once the files are up, this weekend...
IMarv
Re: HOWTO: Put your images on a webserver
Okay, minus the actual rename operation, calculating checksums came to just 7m05.562s. Not bad considering how many files there are...
Put the attached script on your Unix/Linux machine in the top-level directory of the extracted ZIP file. On my system, that's a directory called "Mapping Objects" (yeah, I like to keep things simple!). The forum won't let me attach a filename with a .sh extension, so I zipped it up. You'll need to unpack it either on your machine or on the Unix side. Post again if you need more info on this.
I generated a list of mv commands that accomplish the deed. So all you need to do is execute the script. There are four steps and a message is displayed as each one begins. Expect step two (the rename step) to take a while to finish. Step 1 is to create a subdirectory named assets. Step 2 is the rename operation. Step 3 is to create the index.gz file needed by MapTool. Step 4 is to find all directories that are now empty and remove them.
If you get any output other than the status messages, an error has occurred and the script has terminated. Cut/paste the text from the last "Step" message and anything following it to another post here and we can figure out what to do next.
You do not need to change the permissions on the script at all in order to run it:
Put the attached script on your Unix/Linux machine in the top-level directory of the extracted ZIP file. On my system, that's a directory called "Mapping Objects" (yeah, I like to keep things simple!). The forum won't let me attach a filename with a .sh extension, so I zipped it up. You'll need to unpack it either on your machine or on the Unix side. Post again if you need more info on this.
I generated a list of mv commands that accomplish the deed. So all you need to do is execute the script. There are four steps and a message is displayed as each one begins. Expect step two (the rename step) to take a while to finish. Step 1 is to create a subdirectory named assets. Step 2 is the rename operation. Step 3 is to create the index.gz file needed by MapTool. Step 4 is to find all directories that are now empty and remove them.
If you get any output other than the status messages, an error has occurred and the script has terminated. Cut/paste the text from the last "Step" message and anything following it to another post here and we can figure out what to do next.
You do not need to change the permissions on the script at all in order to run it:
Code: Select all
bash ./rename.sh
- Attachments
-
- rename.zip
- Script to rename images for repository use
- (568.97 KiB) Downloaded 225 times
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
The script is pretty cool, but I don't think you need to rename the files for the assets to work since the second column in the index file is simply the relative path to the file. I'll see if I can rework this into what I'm hoping for.
IMarv
IMarv
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
Question for Trevor,
How does the index file work when the file name contains spaces?
IE:
Thanks,
IMarv
How does the index file work when the file name contains spaces?
IE:
Code: Select all
6333b2e265f9843ecc4ff6c731281557 Compass Roses/compass_rose_silver_beve_bw.png
7a83efafe9fff8105b58c35c827aa0ee Enemies/Ant Spiders/antspiderumber_hybrid2_bw.png
8b1c663a7c0270cab3aba810def8cfc4 Enemies/Ant Spiders/antspiderumber_hybrid_bw.p
IMarv
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java:
The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.
All that is left is to get all the files posted.
Thanks All,
IMarv
Code: Select all
protected Map<String, String> parseIndex(List<String> index) {
Map<String, String> idxMap = new HashMap<String, String>();
for (String line : index) {
String id = line.substring(0, 32);
String ref = line.substring(33).trim();
idxMap.put(id, ref);
}
return idxMap;
}
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.
All that is left is to get all the files posted.
Thanks All,
IMarv
Re: HOWTO: Put your images on a webserver
Sorry, I've been AFK and didn't see your posts. I could've told you that.IMarvinTPA wrote:Ok, I found the answer to my own question in the code. It isn't a token at all, it is a fixed string length deal in AssetLoader.java
What needed to be rearranged? The mv's were all of files, so they were not order-dependent.The index file is at http://www.imarvintpa.com/Mapping/index.gz
I used the mv commands from the rename.sh file for the MD5s and paths. A little work in a spreadsheet program to rearrange the order and clean up the values a smidge and presto.
Just curious in case I need to do something like this again some time.
And kudos, Marvin. This will be very useful!
Edit: Oh, I think I see. You swapped the position of the pathname with the checksum so that the index.gz had the original names. Hmm, that should probably work. I know there are situations where MT finds the entry in the AssetMap by calculating the checksum of the image. If it simply replaces the entry in the Map with a new value under some condition, it may point to a non-existent pathname as the value. But what the heck -- try it and see. But be sure to try it first and see if it works, then shutdown MT and empty your assetcache and try again.
- Full Bleed
- Demigod
- Posts: 4736
- Joined: Sun Feb 25, 2007 11:53 am
- Location: FL
Re: HOWTO: Put your images on a webserver
Any chance in the future that all of this could be done with a single click solution?
1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."
This would be a much more useful feature if it required less technical competency.
It would also be very nice to be able to see at a glance what assets in your library were on a repository.
1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."
This would be a much more useful feature if it required less technical competency.
It would also be very nice to be able to see at a glance what assets in your library were on a repository.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."
Re: HOWTO: Put your images on a webserver
True. Short answer: yes, it can be done.Full Bleed wrote:Any chance in the future that all of this could be done with a single click solution?
1) Input your FTP info in Maptool.
2) Click "upload campaign/asset to repository."
This would be a much more useful feature if it required less technical competency.
Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?It would also be very nice to be able to see at a glance what assets in your library were on a repository.
- Full Bleed
- Demigod
- Posts: 4736
- Joined: Sun Feb 25, 2007 11:53 am
- Location: FL
Re: HOWTO: Put your images on a webserver
How does it know when to look at a repository now for a particular asset?Azhrei wrote:Good idea but the campaign doesn't know what the original asset name was. Tokens have name fields, but multiple tokens can contain the same asset ... so which name should be used?It would also be very nice to be able to see at a glance what assets in your library were on a repository.
Maptool is the Millennium Falcon of VTT's -- "She may not look like much, but she's got it where it counts."
Re: HOWTO: Put your images on a webserver
I think I misunderstood your question. Using the checksum, it is possible to determine which image came from which repository. It's not possible to know if it was actually downloaded from there or already existed on the local machine in the assetcache directory.
Background:
When you drag an image onto the map, the checksum is calculated and stored in a list. In addition, the image is stored in the assetcache directory under the same name as the checksum. Whenever the token needs to be displayed, the checksum is retrieved from the token. This is then used as the asset name and the file in the assetcache directory is opened and read into memory, stored in a BufferedImage object. (I'm not 100% positive on some of this because it's Trevor tricky code!)
Since the repository is remote, I'm thinking that when the campaign is loaded, so are the index.gz files from each repository URL. Those files have a list of checksums and relative pathnames on their server. Now when MapTool needs a particular image (i.e. checksum), it first searches in memory to see if the BufferedImage object is already loaded. If not, it looks in the local assetcache. Failing that, it looks through each of the index.gz files one at a time. The first one that has the checksum in it identifies the URL to use and the filename relative to the index.gz file on the server. That file is grabbed off the server and BufferedImage is created to represent it. At the same time, it's stored in the local assetcache so that the download can be avoided the next time the image is needed and there isn't a BufferedImage for it. This might happen when you change maps -- all image data for the current map is thrown away and the assets from the new map are loaded.
As I said, this is a fairly tricky section of code because the images are also sent asynchronously over the network to other hosts on an as-needed basis. As Trevor posted recently, the long-standing "?" bug was a result of this section of code.
Background:
When you drag an image onto the map, the checksum is calculated and stored in a list. In addition, the image is stored in the assetcache directory under the same name as the checksum. Whenever the token needs to be displayed, the checksum is retrieved from the token. This is then used as the asset name and the file in the assetcache directory is opened and read into memory, stored in a BufferedImage object. (I'm not 100% positive on some of this because it's Trevor tricky code!)
Since the repository is remote, I'm thinking that when the campaign is loaded, so are the index.gz files from each repository URL. Those files have a list of checksums and relative pathnames on their server. Now when MapTool needs a particular image (i.e. checksum), it first searches in memory to see if the BufferedImage object is already loaded. If not, it looks in the local assetcache. Failing that, it looks through each of the index.gz files one at a time. The first one that has the checksum in it identifies the URL to use and the filename relative to the index.gz file on the server. That file is grabbed off the server and BufferedImage is created to represent it. At the same time, it's stored in the local assetcache so that the download can be avoided the next time the image is needed and there isn't a BufferedImage for it. This might happen when you change maps -- all image data for the current map is thrown away and the assets from the new map are loaded.
As I said, this is a fairly tricky section of code because the images are also sent asynchronously over the network to other hosts on an as-needed basis. As Trevor posted recently, the long-standing "?" bug was a result of this section of code.
- IMarvinTPA
- Dragon
- Posts: 467
- Joined: Mon Sep 11, 2006 10:12 am
- Contact:
Re: HOWTO: Put your images on a webserver
I haven't had time to verify that all the files have been uploaded, nor if it works as intended yet, but the upload is reportedly complete. I'll verify when I have some time.
http://www.imarvintpa.com/Mapping/index.gz
IMarv
http://www.imarvintpa.com/Mapping/index.gz
IMarv