Macro Performance Tuning

[midnightdl]

I've been investigating why a macro I wrote is so terribly slow and tuned it but also tested out and profiled performance improvements in macro functions. These changes let me run a macros up to 5 times faster (this is mostly in the str functions codebase that requires O(n) for many operations where it doesn't have to).

I'm not sure though whether my example is just a pathological case. Maybe it's simply a 'special' case. Maybe I should push a PR.

QUESTION: Does anyone else have macros that are simply taking too long to run? I'd like to see if my changes help for those and can profile them as well.

nmeier

[aliasmask]

I would be interested in seeing your macro before and after to see what changes you made. You can also turn on debug mode and check the mt.log file and it will show the timestamps for execution which can reveal slow ups. There are certain functions like getTokens() that are very dependent on the number of tokens on the map that affect performance. Certain uses of eval() can also cause performance issues.

[midnightdl]

My plan was to update tokens spread across multiple maps with latest state after editing, kind of a refresh from original, as every token copied starts its own life, even though I'm updating e.g. handout info centrally.

Looping over many tokens, finding a 'master token' I started with assembling all tokens

Code: Select all

[h: maps = getAllMapNames(",")]

[h: maptokens = ""]
[h, FOREACH(map, maps), code:{
	[h: tokenids = getTokens(",", json.set("{}", "layer", json.append("[]","TOKEN"), "mapName", map ))] 
	[h, FOREACH(tokenid, tokenids, ","): maptokens = listAppend(listAppend(maptokens, map), tokenid))]
}]

Couple of angles
1. Overall I switched to a select-token-run-refresh-from-original approach since that reduces the number of tokens looked at overall - I can update tokens on a per map basis
2. Profiling showed a lot of MapToolLineParser/Expression instantiation which seems to scale with the number of bracket expressions - I've modded MapTool to work with a single parser per macro to gain 3-5x speed
3. Profiling also showed very bad performance on Zone.resolveToken, if one works with lots of token guids in a list (O(n^2)) - I've modded that to optimize for cases where GUIDs are provided
4. strlist functions don't have a best case performance since all lists are completely parses into a list for every append/get/count etc. - I've modded the strlistfunction code and wrote unit tests, that gives 4-5x speed. Making lists, and popping off elements is something very functional-ish that should be fast imo.

So it might be kinda specific to my use case of macros. Maybe I'm expecting too much from it. I started with 30ish seconds of runtime down to 5s for my worst case run through all tokens and do something with them approach. Again, might be too much to ask for.

Nils

[aliasmask]

Sounds like something to be submitted for review on github. I know of a couple of attempts to speed up getTokens() in general. If your changes don't break anything and improve performance I think the community as a whole could benefit from your changes.

[Merudo]

Any performance improvement to the parser or MapTool in general would be most welcomed!

Please open an issue on github and submit a PR, I will be happy to help review it.

Also, if you want to ask technical questions and get answers quickly, I recommend you visit the Discord channel.

[midnightdl]

ok, will put together a PR - I'm on discord (nmeier), wasn't sure if convo like this can be handled in chat.

One thing aside from string/list performance I realized for anyone reading is that a if() roll option doesn't mean what I thought it would if combined with foreach: for example this if/foreach option still means that the foreach code loops over all elements in masterToken, even though lookhere=false.

Code: Select all

[h, if(lookhere), foreach(m, masterTokens), code: {