Making cached structures dynamic (Load BIG forms FAST)

Discuss macro implementations, ask for macro help (to share your creations, see User Creations, probably either Campaign Frameworks or Drop-in Resources).

Moderators: Azhrei, dorpond, trevor, giliath, jay, Mr.Ice

Post Reply
User avatar
wolph42
Deity
Posts: 9718
Joined: Fri Mar 20, 2009 5:40 am
Location: Netherlands
Contact:

Making cached structures dynamic (Load BIG forms FAST)

Post by wolph42 »

I've recently discovered (thanks to jfrazierjr) that it pays well off to cache complex forms. For example a weapons list, containing 258 weapons, stored into a json object that looks like this:

Code: Select all

{"Aegis-Redback":{"Category":"Ranged","Group":"Exotic","Class":"Heavy","Range":80, etc.},
"Assault Cannon":{"Category":"Ranged", etc.  

will take about 70 :shock: seconds to build into a form. So if you instead save the content of the form (encoded) in a (lib)token property, you only need to load the whole structure and put it in a frame command.
'Caching a form' is covered in a wiki article by 'CoveredInAquaCreatures', together with an example of how to cache a structure.

Now, charactersheets you can store locally onto a token and weapons lists like the one above you can store centrally on a lib:token.

However the problem with the latter is that you can't customize it for a specific token. For example, a traits list with all properties is stored in a json object on a lib:token, the traits chosen by the token are stored in a json object locally onto the token.
Now you have a function 'Pick new Traits' where you put up a form (which you cached centrally) where you can select traits for the token. You would however ALSO like to show on that form which traits the token already has... which would mean you have to store it locally, but that would make your tokens rather data heavy.
So I came up with the idea to create a form centrally with %{variables} for which you can later use strformat to fill them in. Its VERY fast and it makes a centrally cached construct customizable.

There are basically only two things the cached form needs in the code, for the macrolink to work you will need:

Code: Select all

[h:me = "%{me}"]
and
<
form id="pickTrait" method="json" action="[r:macroLinkText('[email protected]:DH', 'all',"",me)]">
  

and for the e.g. checkboxes to be checked when you have the trait you need

Code: Select all

<td align="center" valign="top">[r:"%{"+Trait+"}"]</td>  


The whole lot would then for example look like this:
EXAMPLE: CONSTRUCTION ROUTINE

Code: Select all

[H: '<!-- createTraitsCache -->']
[
h:TraitLibrary = getLibProperty("Traits", "Lib:Compendium")]
[
h:TraitList = listSort(json.fields(TraitLibrary), "A+")]
[
h:me = "%{me}"]

<
html>
<
head>
<
link rel='stylesheet' type='text/css' href='[email protected][r:getMacroLocation()]'></link>
</
head>
<
body>
<
form id="pickTrait" method="json" action="[r:macroLinkText('[email protected]:DH', 'all',"",me)]">
<
table id="Traits" width="100%" bgcolor="white">
<
tr>
<
td class="head" valign="bottom" align="center"><b>Select</b></td>
<
td class="head" valign="bottom"><b>Taken</b></td>
<
td class="head" valign="bottom"><b>Traits</b></td>
<
td class="head" valign="bottom"><b>Human</b></td>
<
td class="head" valign="bottom"><b>Prerequisite</b></td>
<
td class="head" valign="bottom"><b>Description</b></td>
<
td class="head" valign="bottom"><b>Source</b></td>
<
td class="head" valign="bottom"><b>Auto</b></td>

</
tr>
[
h:class='oddRow']
[foreach(
Trait,TraitList,""),CODE:
{
    [h: TraitDetails = json.get(TraitLibrary, Trait)]

    [h:varList=json.fields(TraitDetails)]
    [h,foreach(var,varList),CODE:{
         [value = json.get(TraitDetails,var)]
         [set(var,value)]
    }]

    [h, if(Human):Human = "Yes";Human = ""]
    [h: Source = source +"("+page+")"]
   <tr class='[r:class]'>
   <td align="center" valign="top"><input type="checkbox" name="[r:Trait]"></input></td>
   <td align="center" valign="top">[r:"%{"+Trait+"}"]</td>
   <td valign="top"><b>[r:Trait]</b></td>
   <td valign="top">[r: Human]</td>
   <td valign="top">[r: Prerequisite]</td>
   <td valign="top">
        [r: Description]
        [r:if(Groups != "-", "<br><b>Groups</b>: "+Groups, "")]
   </td>
   <td valign="top">[r: Source]</td>
   <td valign="top">[r: if(embedded, "Yes","")]</td>
   </tr>
   [h: class = if(class=="oddRow", "evenRow", "oddRow")]
}]
</
table>
<
input type="hidden" name="token" value='[r:me]'></input>
<
input type="submit" name="doWhat" value="Add Traits"></input>

</
form>
</
body>
</
html>  


This will construct the input for the form.

To make this form 'dynamical' when you call this routine means that EVERY variable needs to be defined, which you can achieve simply by:

Code: Select all

[h,foreach(var,AllTraitsList):set(var,"")]  

To make the checkboxes checked for the traits that the token has you can simply use:

Code: Select all

[h,foreach(var,tokenTraits):set(var,"CHECKED")]  

Then finally you can use strformat to substitute all the variables in the construct:

Code: Select all

[dialog("Choose Traits", "input=1; width=800; height=620; temporary=0"):{
    [r:strformat(decode(traitsCache))]
}]
  


Put everything together and you get the whole routine:
EXAMPLE: SHOW CUSTOMIZED FORM ROUTINE

Code: Select all

[H: '<!-- PickTraits -->']
[
h:me=getName()]
[
h,if(me==""):abort(0)]

[
h:switchToken(me)]
[
h,if(getProperty("S")==""):abort(0)]
[
h, if(argCount() > 1): rebuild = arg(1); rebuild = 0]

[
h:traitsCache = getLibProperty('traitsCache','lib:Compendium')]
[
h, if(traitsCache=="" || rebuild), CODE:{
    [traitsCache = encode(createTraits())]
    [setLibProperty("traitsCache",traitsCache,"lib:Compendium")]
}]

[
h:TraitLibrary = getLibProperty("Traits", "Lib:Compendium")]
[
h:TraitList = listSort(json.fields(TraitLibrary), "A+")]
[
h:tokenTraits = json.fields(Traits)]

[
h,foreach(var,TraitList):set(var,"")]
[
h,foreach(var,tokenTraits):set(var,"CHECKED")]

[
dialog("Choose Traits", "input=1; width=800; height=620; temporary=0"):{
    [r:strformat(decode(traitsCache))]
}]
  


And that's it.
Let me know if you find ways to simplify or improve on this!!
Last edited by wolph42 on Mon Dec 05, 2011 5:58 am, edited 3 times in total.

User avatar
aliasmask
Deity
Posts: 8582
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by aliasmask »

Instead of encode/decode you could just assign it to a json array

Code: Select all

[h: traitsCache = json.append("",createTraits())]
[r: strformat(json.toList(traitsCache,""))]

I like building my output in json arrays that way I can add stuff to the top or bottom after certain processes have already run. I don't know if its faster, but I think it would be.

Do you have a problem with setting the "me" variable. Does the UDF have to set newScope to false? Quotes seem off with the action in form. Seems to me you would need to do an eval at some point. Does the decode do that for you? Oh think I see. When it writes it to the dialog, then it's executed as macro code.

In your example you use charTraits in one location and tokenTraits in another... may be confusing to the reader, but overall, pretty clever. Another trick, but this won't work for the complex json object you have is to convert the json to a strprop and define them all with a Wiki: varsFromStrProp() instead of using a [foreach:] loop. It's faster when it can be used.

User avatar
wolph42
Deity
Posts: 9718
Joined: Fri Mar 20, 2009 5:40 am
Location: Netherlands
Contact:

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by wolph42 »

aliasmask wrote:Instead of encode/decode you could just assign it to a json array

Code: Select all

[h: traitsCache = json.append("",createTraits())]
[r: strformat(json.toList(traitsCache,""))]

I like building my output in json arrays that way I can add stuff to the top or bottom after certain processes have already run. I don't know if its faster, but I think it would be.

Do you have a problem with setting the "me" variable. Does the UDF have to set newScope to false? Quotes seem off with the action in form. Seems to me you would need to do an eval at some point. Does the decode do that for you? Oh think I see. When it writes it to the dialog, then it's executed as macro code.

In your example you use charTraits in one location and tokenTraits in another... may be confusing to the reader, but overall, pretty clever. Another trick, but this won't work for the complex json object you have is to convert the json to a strprop and define them all with a Wiki: varsFromStrProp() instead of using a [foreach:] loop. It's faster when it can be used.


> the biggest json structure in my framework, Weapons, counts 24 variables and lists 258 weapons. It loads with this method in about 1 second (creating the cache takes 70s :shock: ). So your method might be faster, but I don't think you're going to notice it :D.

> no problems with 'me' actually its the most vital reason why a dynamic approach is needed, even without setting the 'CHECKED' variable, you still need to set the selected token

> Thanks for pointing out the typo, fixed it in the example!

> nice trick indeed, but again building a form with 6216 json variables in about one second, I don't think 'faster' will be noticeable.

User avatar
aliasmask
Deity
Posts: 8582
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by aliasmask »

I've been experimenting with some big lists, json object with 621 fields with some string data. The problem I ran in to was with the Wiki: json.get() where each took an average of 20ms to put the variable in the string where the whole process was about 12 seconds to print the whole list.

Code: Select all

[H: spellLinks = am.sd.glp("spell_links")]
[
H: output = ""]
[
H, foreach(spellPropName,spellLinks), code: {
   [H: link = json.get(spellLinks,spellPropName)]
   [H: output = json.append(output,link)]
}]
[
dialog("Spell List"): {
   [R: json.toList(output,"<br>")]
}]
 

With a bunch of experimentation I think I found a good dynamic solution:

Code: Select all

[H: spellLinks = am.sd.glp("spell_links")]
[
H: spellLinks = replace(spellLinks,";","&#59")]
[
H: spellProps = json.toStrProp(spellLinks)]
[
H: varsFromStrProp(spellProps,"UNSUFFIXED")]
[
H: output = ""]
[
H, foreach(link,spellLinks): output = json.append(output,strformat("[r:%{link}]"))]
[
H: output = json.evaluate(output)]
[
dialog("Spell List"): {
   [R: json.toList(output,"<br>")]
}]
 

This takes about 3 seconds with no need to rebuild 621 spells. Of course, the smaller the output list, the quicker, but this would be for the full db.

User avatar
JonathanTheBlack
Dragon
Posts: 544
Joined: Mon Jun 28, 2010 12:12 pm
Location: I'm the worm...

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by JonathanTheBlack »

There is a simpler solution. Make your players do the work for you! :D

I'm creating a derivative of D&D 4e and instead of creating a huge json object database of powers, I'm letting each player create their own powers that are added to a Lib:Compendium token. Each power is stored by itself and its name is added to a PowerList property stored on the Compendium token as well. Another macro (yet to be created) will allow players to filter powers by power source and sort them alphabetically or by use (at-will, encounter, etc).

User avatar
aliasmask
Deity
Posts: 8582
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by aliasmask »

This is actually a part of a spell manager I'm working on for the community.

The list is actually dynamic and usually a lot smaller. It just pulls from a very large master list. I was just testing worse case scenario. Too bad the json.get is so slow pulling from a large object. I was going to test which was faster, creating two arrays, doing a json.indexOf in one array and json.get in the other array. I imagine it's about the same speed and that would be too much trouble to maintain.

User avatar
JonathanTheBlack
Dragon
Posts: 544
Joined: Mon Jun 28, 2010 12:12 pm
Location: I'm the worm...

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by JonathanTheBlack »

I'd thought about making one big json object or something, but in the end, decided it was easier just to store each power individually as a json object on the Lib:Compendium token. That way when an attack macro is called, it's only pulling data from a single token property and using that as is instead of sorting through a json object to find what it needs.

User avatar
aliasmask
Deity
Posts: 8582
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by aliasmask »

JonathanTheBlack wrote:I'd thought about making one big json object or something, but in the end, decided it was easier just to store each power individually as a json object on the Lib:Compendium token. That way when an attack macro is called, it's only pulling data from a single token property and using that as is instead of sorting through a json object to find what it needs.

That's a good idea if your json has multiple layers... that's what I did. But, for special lists that I may access a lot, it's good to combine your data in to one location. Each spell has a single layer object (so no objects in my object) and has it's own property. Each spell object has a field called "link" which is the text needed to output the link. It takes around 6ms to getLibProperty and another 4-12ms to use json.get on an object (1-4ms on array) to get the value needed. Repeated over 600 times and that adds up.

By only doing one getLibProperty, some string replacements and a json.evaluate cuts time significantly for large data sets. I think I calculated if I need to get more than 4 links, that this method is preferred just because of the getLibProperty and json.get time. Ideally, if the final value was in the property then that would be faster, but that would be over 600 properties just for link text and would make setting token values slower. It takes 4ms for the getLibProperty and averages 5ms for the above method. json.get at it's fastest would only tie the above method.

Merudo
Giant
Posts: 228
Joined: Wed Jun 05, 2019 7:06 am

Re: Making cached structures dynamic (Load BIG forms FAST)

Post by Merudo »

I love this article! It really speed up my character sheets!

One this I'd like emphasized is that as of maptool 1.5.2, token IDs are reset on server start. This will make cached ids no longer valid and the links on the character sheet will be broken. Possible solutions are to redo the cache if there is a token ID change detected, or pass the token ID as a variable %{} to be replaced by strFormat().


Post Reply

Return to “Macros”