Macro Best Practices

Discuss macro implementations, ask for macro help (to share your creations, see User Creations, probably either Campaign Frameworks or Drop-in Resources).

Moderators: Azhrei, dorpond, trevor, giliath, jay, Mr.Ice

User avatar
wyrmwood
Dragon
Posts: 458
Joined: Thu May 22, 2008 12:15 am

Re: Macro Best Practices

Post by wyrmwood »

Craig wrote:Well you did say something special is going on to create the reference that is different from primitives so its was only "technically" pass by value :) Which is the bit that was incorrect.

and I still say something "special" is going on, but I'll admit, whether the behavior is "special" in which language is debatable...

Here's my lousy attempt at an explanation

User avatar
wyrmwood
Dragon
Posts: 458
Joined: Thu May 22, 2008 12:15 am

Re: Macro Best Practices

Post by wyrmwood »

Rumble wrote:The upshot of all of this being: macros can go real slow if you use excessive JSONs. :P

Yeah, so back to my original question...
It appears that passing a single property is indeed more efficient than passing a large JSON, Why? Because the a new JSON gets copied each time we pass it as a parameter? Can we pass a JSON array or a JSON object into a macro and have the original be modified? or are users just passing around data JSONs to be used and discarded? (and things like loops are killing performance when you really only needed a couple of properties?)

User avatar
Rumble
Deity
Posts: 6235
Joined: Tue Jul 01, 2008 7:48 pm

Re: Macro Best Practices

Post by Rumble »

wyrmwood wrote:
Rumble wrote:The upshot of all of this being: macros can go real slow if you use excessive JSONs. :P

Yeah, so back to my original question...
It appears that passing a single property is indeed more efficient than passing a large JSON, Why? Because the a new JSON gets copied each time we pass it as a parameter? Can we pass a JSON array or a JSON object into a macro and have the original be modified? or are users just passing around data JSONs to be used and discarded? (and things like loops are killing performance when you really only needed a couple of properties?)



Can't answer the former, but loops appear to hit performance regardless. I did a simple loop over token states, and then modified it to directly manipulate the special "state.Whatever" variables, and the second was much faster, even though the loops involved no JSONs or any passed variables.

Craig
Great Wyrm
Posts: 2107
Joined: Sun Jun 22, 2008 7:53 pm
Location: Melbourne, Australia

Re: Macro Best Practices

Post by Craig »

wyrmwood wrote:
Craig wrote:Well you did say something special is going on to create the reference that is different from primitives so its was only "technically" pass by value :) Which is the bit that was incorrect.

and I still say something "special" is going on, but I'll admit, whether the behavior is "special" in which language is debatable...

Here's my lousy attempt at an explanation


Please reread my posts, your understanding is flawed. If you want equivalent code then you should change your C++ code to.

Code: Select all

{    
      Person *Bob = new Person();
      Bob->m_nHP = 20;
      cout << "Bob has " << Bob->m_nHP << " HP" << endl;
 
      HurtMeByValue(Bob, 5);
...

void HurtMeByValue(Person *a, int nDamage)
{
      a->m_nHP -= nDamage;
}


Because in Java
Person p;
is the same as the C++
Person *p;
Do that and then tell me they are doing something different.

edit for further explanation

In Java
Person p = new Person() is an object allocated in heap space

In C++
Person *p = new Person() is an object allocated in heap space
Person p is a temporary object allocated on the stack and has no equivalent in Java.

You compare apples from one store with oranges from a different store (instead of the seconds stores apples) and since you see a difference you are attributing that to something special that the second store is doing to its oranges so that they no longer look like apples.
Last edited by Craig on Mon Sep 28, 2009 6:02 pm, edited 2 times in total.

Craig
Great Wyrm
Posts: 2107
Joined: Sun Jun 22, 2008 7:53 pm
Location: Melbourne, Australia

Re: Macro Best Practices

Post by Craig »

wyrmwood wrote:
Rumble wrote:The upshot of all of this being: macros can go real slow if you use excessive JSONs. :P

Yeah, so back to my original question...
It appears that passing a single property is indeed more efficient than passing a large JSON, Why? Because the a new JSON gets copied each time we pass it as a parameter?

I already explained this is not the case. Please reread earlier posts.

wyrmwood wrote:Can we pass a JSON array or a JSON object into a macro and have the original be modified? or are users just passing around data JSONs to be used and discarded? (and things like loops are killing performance when you really only needed a couple of properties?)

Are you wanting mutable JSON objects or pass by reference? Pass by reference will make things even slower, mutable JSON objects is possible but then breaks how everything currently works causing people to rewrite all their macros.

Craig
Great Wyrm
Posts: 2107
Joined: Sun Jun 22, 2008 7:53 pm
Location: Melbourne, Australia

Re: Macro Best Practices

Post by Craig »

I just want to add to this thread token properties are always going to be faster than using JSON objects -- for 1.3 at least. This is because of the following reasons.

  • JSON objects under certain conditions (reading/writing from/to token properties) requires conversion from one internal representation to another.
  • MapTool supports storing JSON objects as string representations -- this is one of the main points behind JSON -- and strings are immutable in MapTool script, therefore JSON objects need to be immutable.

It has nothing to do with pass by value/references.When using [macro(): ] calls passing a JSON object (or property, or number) is essentially the following code in the MapTool.

Code: Select all

    newScope.put("macro.args", args);

Which creates a new variable in the new scope.

If you have your JSON object inside of a token property (say for example WeaponsList) and do the following

Code: Select all

      [c(100), code: {
           [macro("[email protected]:token"): WeponList]
      }]


You will take a internal conversion hit from token property representation to internal variable representation on each and every time through the loop. You could easily speed this up by doing the following for large JSON object.

Code: Select all

      [wList = WeaponList]
      [c(100), code: {
           [macro("[email protected]:token"): wList]
      }]



User defined functions occur differently. User defined functions in reality are not function calls. What they do is create a new string that is passed to the parser, so when you have a large argument this parsing takes extra time -- this is not to be confused with copying on pass by value because this is not what it is. So there are two solutions to this problem, keep your arguments small or use [macro(): ], but again be careful about "invisible" conversions when placing the token property directly in the call.


While allowing JSON objects to be mutable would solve several of the speed issues with modifying JSON objects it is not possible to make them mutable in all cases which means you end up with a large list of exceptions where JSON objects would still be immutable. Also supporting mutable JSON objects withing user defined functions would require a lot more processing than is currently done and significantly slow down calls to user defined functions.


In 1.4 there is a large opportunity for scripting performance and the two things that will provide the greatest performance are

JavaScript
This is not because JavaScript is an inherently faster language, but because of the way that parsing currently occurs in MapTool. The JavaScript engine will parse all input once and generate an intermediate code that it can process a lot faster.

The way MapTool currently works there is a Parser module (not part of the MapTool code) that parses all text and generates intermediate code that is faster to process, but this only parses/executes the text that MapTool passes to it. The MapTool input scanner breaks up macros and sends then essentially line by line to the Parser module, so when you loop you re parse the same text over and over and over.

JSON improvements
This has nothing to do with the implementation of JSON in JavaScript itself, but in 1.4 hopefully the parts of MapTool that expect strings and only strings to be stored in token properties can be fixed so that they wont bomb out when something other than a string is stored, this way there will be no conversion hit when moving JSON objects to and from token properties.

User avatar
Rumble
Deity
Posts: 6235
Joined: Tue Jul 01, 2008 7:48 pm

Re: Macro Best Practices

Post by Rumble »

Ahhhh...very useful. I guess my next project will be evaluating my framework's loops and make sure I'm not requiring a representation change on every iteration.

Craig, if nobody else says it, thanks - it's a huge help to have someone who knows the intricacies of the parser and the scripting language lay some knowledge on us.

User avatar
wyrmwood
Dragon
Posts: 458
Joined: Thu May 22, 2008 12:15 am

Re: Macro Best Practices

Post by wyrmwood »

Rumble wrote:Craig, if nobody else says it, thanks - it's a huge help to have someone who knows the intricacies of the parser and the scripting language lay some knowledge on us.

copy that

User avatar
toyrobots
Dragon
Posts: 278
Joined: Sat Apr 12, 2008 4:17 pm

Re: Macro Best Practices

Post by toyrobots »

Ah, I remember the days when this thread contained a discussion of macro best practices. ;)

User avatar
Mrugnak
Dragon
Posts: 745
Joined: Mon Jul 21, 2008 7:38 pm

Re: Macro Best Practices

Post by Mrugnak »

On the subject of Best Practices and JSON objects, just to be sure I'm currently on the right page, and I may not be...

I'm currently storing some JSON arrays of JSON objects in token properties. I don't pass them as macro arguments, I just use them directly as needed.
I haven't noticed a significant memory hit so far, except once when I just left Maptools alone and open all night with three frames open (by accident) and the memory used (As per the little bar) ended up creeeeeping. No macro activity, so I'm assuming that's related to something in Maptools directly that I may have triggered by creating the FRAMEs.

EDIT: Helps if I finish the question...

When working with the token property containing something JSON, is the best practice that:
I should access the token property as few times as possible, and instead work with a local variable where possible and "commit" my changes back to the token property at the end?

User avatar
CoveredInFish
Demigod
Posts: 3104
Joined: Mon Jun 29, 2009 10:37 am
Location: Germany
Contact:

Re: Macro Best Practices

Post by CoveredInFish »

Mrugnak wrote:When working with the token property containing something JSON, is the best practice that:
I should access the token property as few times as possible, and instead work with a local variable where possible and "commit" my changes back to the token property at the end?


Yes. Each time you read/store a json object from/to a property it is transformed into/created by a simple string. This conversion is performance-critical. As long as you use a variable containing a json object there are no conversions.

User avatar
Mrugnak
Dragon
Posts: 745
Joined: Mon Jul 21, 2008 7:38 pm

Re: Macro Best Practices

Post by Mrugnak »

CoveredInFish wrote:Yes. Each time you read/store a json object from/to a property it is transformed into/created by a simple string. This conversion is performance-critical. As long as you use a variable containing a json object there are no conversions.


OK! That's thankfully pretty easy to adjust the way I've got my macros set up so far (glad I caught this before getting too involved though!)

User avatar
aliasmask
Deity
Posts: 8579
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: Macro Best Practices

Post by aliasmask »

I found setting the property to a json object takes longer than getting it from a property. Maybe it's my imagination.

Barrodin
Kobold
Posts: 16
Joined: Thu Aug 18, 2011 10:51 am

Re: Macro Best Practices

Post by Barrodin »

After reading over some of the info in this thread about user defined functions, macros, and JSON objects, I had a couple questions I'm hoping someone can answer.

1.) JSON objects/arrays are slower to load from and save to string properties, but are there any speed advantages in working with JSON objects/arrays once they are loaded or as temporary variables? Is json.get() faster or slower than listGet(), for example?

2.) It was mentioned that passing a JSON object/array to a user defined function in a loop would be particularly slow, because each loop would require translating the object to and then from a string; am I correct in thinking that, in a situation like that, it would be considerably faster to use a function defined with newScope set to false to access the JSON object/array, instead of passing it as an argument?

i.e. instead of this:

Code: Select all

[h: defineFunction(exampleFunc, exampleMacro)]

...
 later when the function is called ...

[
h: someJsonObject = json.set("{}", "Key1", Value1, "Key2", Value2, ... , "KeyN", ValueN)]
[
h, count(someLoopCounter), CODE:
{
  [h: exampleFunc(someJsonObject)]
}]
 


something like this:

Code: Select all

[h: defineFunction(exampleFunc, exampleMacro, 0, 0)]

...
 later when the function is called ...

[
h: someJsonObject = json.set("{}", "Key1", Value1, "Key2", Value2, ... , "KeyN", ValueN)]
[
h: targetObject = "someJsonObject"]
[
h, count(someLoopCounter), CODE:
{
  [h: exampleFunc(targetObject)]
}]
 

Barrodin
Kobold
Posts: 16
Joined: Thu Aug 18, 2011 10:51 am

Re: Macro Best Practices

Post by Barrodin »

Barrodin wrote:2.) It was mentioned that passing a JSON object/array to a user defined function in a loop would be particularly slow, because each loop would require translating the object to and then from a string; am I correct in thinking that, in a situation like that, it would be considerably faster to use a function defined with newScope set to false to access the JSON object/array, instead of passing it as an argument?


Did a bit of testing, and it turns out that it is much, much faster to use a local scope function to pass and return a JSON Object by name, rather than passing the actual object itself. Since this involves using eval() to retrieve the value, it is absolutely not suitable for anything where you're dealing with uncertain input, but I think it could be a very useful trick for speeding up common function calls where you can be sure of the values passed.

Passing an object like this:

Code: Select all

[h: testObject = json.set("{}", "Key 1", value1, "Key 2", value2, ... , "Key N", valueN)]
[
h: test_function("testObject")]

...
 Inside test_function ...
[
h: local_testObject = eval(arg(0))] 


...instead of this:

Code: Select all

[h: testObject = json.set("{}", "Key 1", value1, "Key 2", value2, ... , "Key N", valueN)]
[
h: test_function(testObject)]

...
 Inside test_function ...
[
h: local_testObject = arg(0)] 


...was around 4x as fast, and returning JSON objects from a local function using a similar method was even faster (~16x as fast, but it was hard to be sure, because at that point each test was only about 1 second from start to finish).

Keep in mind that any variables in a function will overwrite functions in the macro from which it is called, so be conservative creating new variables and always use a distinctive name (I tend to add the prefix local_ to the variable names inside such functions, as a reminder). If you do decide to use this trick it is probably best to create as few variables inside the fucntion as possible, to avoid the possibility of collisions.

EDIT: On a side note, some sort of built in timestamp function would be enourmously helpful in tests like this.

Post Reply

Return to “Macros”