Love it or Leave it: Current Macro Language

[Rumble]

If I'm not mistaken, that would fall closer to "exposing the internals" than a "designed" API.

Hmmm... I should clarify. My main concern is not LOSING any functionality over MTScript.

My point was just that I think that's not the way the Powers What Is want to go with this; it sounds like they'd rather step back and rebuild it "right," even if that means losing functionality in the short term (or, rather, not having javascript until they have it rebuilt to their satisfaction AND have the functionality to their satisfaction).

If I'm understanding the code properly, with the .eval(some javascript) approach, you'd have to go through the MapTool line parser to make it work anyway, so it wouldn't gain anything (I'm not entirely sure any of the MTScript functions actually exist in a Javascript-usable way).

[JonathanTheBlack]

I'm reluctant to grow the API organically like that. The current MTscript has a bunch of overlapping functionality that really should be cleaned up. But it can't be because people could be using it. :?

Clean it up anyway. Once there is a stable version, the people unwilling to re-code their frameworks or re-do campaigns can keep using the stable version.

Switching to JS is our chance to "do it right" so I would prefer to see it done carefully and thoughtfully -- not just adding some function "because it's there now". That's not good enough in my book.

The "switch to JavaScript" version should be 100% dedicated to doing just that and doing it properly. Freeze features and stick to bugfixes until after that's done.

[jfrazierjr]

If I'm not mistaken, that would fall closer to "exposing the internals" than a "designed" API.

Hmmm... I should clarify. My main concern is not LOSING any functionality over MTScript.

My point was just that I think that's not the way the Powers What Is want to go with this; it sounds like they'd rather step back and rebuild it "right," even if that means losing functionality in the short term (or, rather, not having javascript until they have it rebuilt to their satisfaction AND have the functionality to their satisfaction).

If I'm understanding the code properly, with the .eval(some javascript) approach, you'd have to go through the MapTool line parser to make it work anyway, so it wouldn't gain anything (I'm not entirely sure any of the MTScript functions actually exist in a Javascript-usable way).

My guess is that the javascript implementation would require a subclass of certain objects which will act as both a proxy as well as a security "gatekeeper". ie, instead of exposing the Zone object directly (which contains WAY to much stuff that does not need to be called from JS), there would be a wrapper object (say ZoneImpl) which would have a few methods(I expect all methods would need to be implemented and stubbed out... those not "allowed" don't call super(), so do nothing) and those would perhaps call super() from the parent Zone object. The jsEngine would then .put() this object so that it's available to scripting. The scripter may do something like:

Code: Select all

var map = ZoneImpl.getCurrentZone()

and then use the map variable to run some subset of the Zone object's methods which have been made available.

Of course, this is just one possible approach.

[jfrazierjr]

One point I should note, the switch to an OSGI implementation would potentially make it easier for someone to create a new UI. The whole point of OSGI is for a plug in architecture that could be modified at run time. This means that if someone so desired, 2 or more UI's could be implemented and switched back and forth as needed. The gating factor here is that a) the OSGI framework would have to be added to the code first, and b) the existing UI would need to be refactored into one or more bundles.

Now that I think about it, this could be a good thing for permissions. Imagine having a server setting that not only disabled the FOW/VBL tools, but removed them totally from the client while the client was connected. Because OSGI components can be turned on and off at runtime, any UI items(buttons,menus, right click actions, panels, etc) they submitted are removed when they are turned off.

Of course, as Azhrei said, its still a big topic to deal with and lot's of work.

[Rumble]

My guess is that the javascript implementation would require a subclass of certain objects which will act as both a proxy as well as a security "gatekeeper". ie, instead of exposing the Zone object directly (which contains WAY to much stuff that does not need to be called from JS), there would be a wrapper object (say ZoneImpl) which would have a few methods(I expect all methods would need to be implemented and stubbed out... those not "allowed" don't call super(), so do nothing) and those would perhaps call super() from the parent Zone object. The jsEngine would then .put() this object so that it's available to scripting. The scripter may do something like:

Code: Select all

var map = ZoneImpl.getCurrentZone()

and then use the map variable to run some subset of the Zone object's methods which have been made available.

Of course, this is just one possible approach.

Ah, I think I see. I'm not really clear on how this all works (I also know roughly nothing about javascript, which doesn't help my understanding significantly).

So...making something available to javascript in that fashion would involve both writing a wrapper to handle the methods that could be accessible from, say, ZoneRenderer, and then going through all the MTScript function classes and finding which functions call that method and what they return and making sure an equivalent exists in the wrapper...

Wow. Good luck with that, guys. Sounds hard.

[bstrdsmkr]

If I'm understanding the code properly, with the .eval(some javascript) approach, you'd have to go through the MapTool line parser to make it work anyway, so it wouldn't gain anything (I'm not entirely sure any of the MTScript functions actually exist in a Javascript-usable way).

While it's possible to be set up that way, probably through a [js:] roll option, at least for the first run, I don't think chat window scripts are the way to go. Since JS is pretty much self contained, I think a better way to go about it is more of a slash command, ie:

/js mycustomscript arg1 arg2 arg3

instead of typing js into the chat window or having your script feed lines to the chat window. I think js should only output to the chat window when you want to display something to the users. It should be able to do the rest internally. If you want to DISPLAY an attack roll, the chat.output('I attack and roll [1d20+5]') and let the parser deal with the [1d20+5] part. If you want to calculate an attack roll then do it inside javascript and when you're ready to display it then chat.ouput('I attack and roll '+d20roll+' plus '+strbonus+'for strength!')

instead of exposing the Zone object directly (which contains WAY to much stuff that does not need to be called from JS),

My question for this is, is the Zone object exposed to MTScript now? if not then why does it need to be exposed to Javascript? Now if we decide that it's a good idea at some point to expose the Zone object to user scripting in general then that's a different matter and the discussion of what parts to expose should get their own discussion separate from this one. I think the same for each object being considered for exposition.

[bstrdsmkr]

So...making something available to javascript in that fashion would involve both writing a wrapper to handle the methods that could be accessible from, say, ZoneRenderer, and then going through all the MTScript function classes and finding which functions call that method and what they return and making sure an equivalent exists in the wrapper...

I think only the latter half of this is necessary. The upper half should only come into play when you're talking about exposing new stuff. Since javascript can directly call Java methods (and vice versa) you could actually just call the same functions MTScript uses to call those methods. When you want to expose a Java method/function/object to JS, just .put() it into the engine.

Now when exposing stuff that isn't already exposed to MTScript, if you don't want to expose the entire object, then you'd have to wrap it with something that only exposes the desired methods, but for stuff that's already there, just call the same function that the parser calls when it encounters say "getToken()" inside a macro.

[jfrazierjr]

My question for this is, is the Zone object exposed to MTScript now? if not then why does it need to be exposed to Javascript? Now if we decide that it's a good idea at some point to expose the Zone object to user scripting in general then that's a different matter and the discussion of what parts to expose should get their own discussion separate from this one. I think the same for each object being considered for exposition.

It's not directly. Zone = a Map. So, if you think about it, all of the Map functions in MTScript operate in some way on a Zone object... it's just abstracted out... for example, getCurrentMapName() is the same as map.getName(). Likewise, instead of calling the moveTokenToMap() function, you might do:

Code: Select all

var zones = Maptool.getZones();
var map = zones["Grasslands"];
map.putToken(token, x,y);

Again, this just one possible way... I tend to prefer that if we are going to utilize JS as an integrated scripting language, we should be using objects... but that just my approach and I am not the one who will be doing the work(well.. at least not the majority of it).

[jfrazierjr]

Now when exposing stuff that isn't already exposed to MTScript, if you don't want to expose the entire object, then you'd have to wrap it with something that only exposes the desired methods, but for stuff that's already there, just call the same function that the parser calls when it encounters say "getToken()" inside a macro.

The problem with this approach is that all of the existing functions return strings(well... it's not a problem directly), and I personally would want to work with objects in an OO way via JS rather than in a functional programming model.

[Rumble]

Now when exposing stuff that isn't already exposed to MTScript, if you don't want to expose the entire object, then you'd have to wrap it with something that only exposes the desired methods, but for stuff that's already there, just call the same function that the parser calls when it encounters say "getToken()" inside a macro.

The problem with this approach is that all of the existing functions return strings(well... it's not a problem directly), and I personally would want to work with objects in an OO way via JS rather than in a functional programming model.

They all return strings, and require strings as arguments, IIRC, and do a lot of heavy lifting to turn the strings they receive into arguments usable by the methods they eventually call. So while there is a method called when the parser encounters "getTokens()" inside a macro - the childEvaluate() method in, uh, FindTokenFunctions.java, there's a lot of chewing that goes on to decide ultimately how to get the tokens that are requested.

So (just peeking at the code at the moment) to call getTokens() means really calling zone.getTokensFiltered() and instantiating a new zone filter (and knowing which filter to instantiate) and perhaps pulling down the results of getIdentity() from the current MapToolFrame instance (if you wanted getImpersonated() instead) and so forth.

However, if you created an interface or proxy or similar way of accessing it, you might be able to use getTokens() without the ugly internal stuff, and preserve some of the familiar functions from the MTScript we have now (edit: while gaining the speed and efficiency improvements jfraz mentions in the next post).

[jfrazierjr]

Now when exposing stuff that isn't already exposed to MTScript, if you don't want to expose the entire object, then you'd have to wrap it with something that only exposes the desired methods, but for stuff that's already there, just call the same function that the parser calls when it encounters say "getToken()" inside a macro.

The problem with this approach is that all of the existing functions return strings(well... it's not a problem directly), and I personally would want to work with objects in an OO way via JS rather than in a functional programming model.

They all return strings, and require strings as arguments, IIRC, and do a lot of heavy lifting to turn the strings they receive into arguments usable by the methods they eventually call. So while there is a method called when the parser encounters "getTokens()" inside a macro - the childEvaluate() method in, uh, FindTokenFunctions.java, there's a lot of chewing that goes on to decide ultimately how to get the tokens that are requested.

So (just peeking at the code at the moment) to call getTokens() means really calling zone.getTokensFiltered() and instantiating a new zone filter (and knowing which filter to instantiate) and perhaps pulling down the results of getIdentity() from the current MapToolFrame instance (if you wanted getImpersonated() instead) and so forth.

Exactly, and bypassing that as much as possible will result in even more speed increases without having to split the string arguments up and assign them to the appropriate object in the childEvaluate() method.

[Craig]

Yikes!
Since I have been threatened with a dungeon I better make some sort of reply in an attempt to avert it.

I apologise in advance for the size of this post but it covers a lot of ground.


I really haven't put much thought -- nearly none at all -- into what I think
JavaScript macros might look like in 1.4 so I will just try to cover random
parts of posts in this thread with a "off the top of my head this is what I
think". As I am more or less coming up with what I think needs to be done as
I type this I am sure I will miss something, or think of a better way to do
something just after I have posted this. Hopefully this this will in the end
make some sort of sense and not become a rambling mess, it should at least
serve to set some expectations on just how much thought and effort needs to
go into such an endeavor.

For starters I noticed a couple of posts about how it looks pretty easy just
take "this snippet of code" and it will work. This is far from the case, that
is about 1% of the work that needs to be done. Also that is the Java 6 way of
doing things, it may be preferable to do it the Java 5 ways as its not that
much more code, and you don't want a handful of code to force you to have to
adopt Java 6, I know people will be screaming but 1.4 will be Java 6 only!!!
That could quite possibly -- even probably -- be true, but then again the
biggest driver for this would be the vision code, you never know since OpenGL
offers even more that could be the way to go instead, so no longer would Java 6
be absolutely needed, but that possibility is a completely different discussion.
There are other reasons that doing it the Java way is beneficial as it requires
no set up by the user at all, where as the Java 6 method will in the case of
some non Oracle JVMs, while the scripting interface is required for Java 6 the
JavaScript implementation is not.

I know that the Java 5 method of including JavaScript is specific to the
JavaScript engine and does not apply to other scripting engines, unlike the
Java 6 method which is similar code for many other languages. I do not see
this as a problem, as its very unlikely that there will be more than one
"officially supported" scripting language for macros -- thats right no ruby,
no python, only JavaScript. This is not because anyone has something against
these other languages or I know something Trevor has said that no one else does,
but simply because the amount of work. Regardless what most people think,
adding another language is not simple, JavaScript and Java don't always see
eye to eye and some coercion is required. Add another language into the mix,
and suddenly you have another set of coercion required, whats worse a bridge
between the 2 languages, after all if you pick up a library for say DnD of
macros, do you want to find that it was written in Lisp so suddenly your
Python macros suddenly need to know how to manipulate Lisp data structures?
My guess is no you don't, the more languages in the mix the worse it gets,
the more support the RPTools team has to do for that other languages and the
testing for 2 languages is probably at least 3 times that of 1, and it gets
much worse from there. Hence no matter what language you like the most its
best to have one standard macro language. If 1.4 supports plugins then I am
sure someone will add X language to the macro language, but then thats for
them to sort out, not necessarily something that the RPTools Team would have
tool maintain.



Ugh thats a lot of typing so far and I haven't even addressed a small fraction
of the comments.

So first off to ally any fears, there will be a lot of functions to do many
of the things that you can currently do. At a guess though I would say that
you will find some largish differences. This is because apart from no overall
design for the current macro language it is for a large part heavily influenced
by the limitations in place. For example since the current macro language has
no null value or way of capturing an error there are several functions that
can be used to check for potential error conditions or nulls, these most likely
would have no analog as there is no need for them. Another property of the
current macro language is it really "feels" like an interpreted language, there
are several functions that do a lot of processing inside the function which
are quite possible to do in the macro language, but would be a lot less
efficient -- look at the getToken() function for example! A well designed API
for a non interpreted language -- such as JavaScript in our case -- would tend
to have more functions that try to do less things, I am a big fan of the Unix
philosophy of "write programs that do one thing and do it well", and this
equally applies to methods/functions, especially in an API. So the API should
contain smaller functions that work together to achieve what the current
API does with large complex functions. For the non programmers out there this
may sound scary because you are thinking "oh no I have to remember more
function names to do the same things?", well there will be more functions but
they should be consistently named -- so we wont be using Visual Basic as a
model -- which will make it clear which one you need to use without having
to remember the rather esoteric syntax required for the parameters of some of
the current functions.

The next thing to consider is security and permissions. There are two sides
of this to look at, and when thinking about these questions do not think
about them as the group that knows each other well and always plays together,
think about the GM/players in one off pickup games because they have the
highest security requirements.
1) As GM how do I limit what players can do?
2) As a player how do I limit what the GM can do on my machine?

To address the first one there are 2 options,
a) The trusted frame work is pushed down a level into the JavaScript macros
where each trusted JavaScript method checks to see if the macro is
running as trusted and fails if required.
b) JavaScript macros are GM only, players can not run their own JavaScript,
instead they run simple expressions -- think of anything you could
currently run in a { }. JavaScript could then be used to add callable
functions that players can run if they tweak their own macros.

I favor option b over option a because its cleaner, easier to implement, and
consistent, these are all good things when talking about security. JavaScript
objects would be able to export certain methods -- similar to how user
defined functions work now -- a exported function could be exported as
trusted only so that any call to it from an untrusted macro would fail.

The second you may be thinking "wait players limiting what the GM can do",
I am not talking about what the GM can do to players tokens, or maps or what
ever. What I am talking about is other things, like the GM shouldn't just be
able to open files on the users PCs, download arbitrary files outside of
MapTool and all kinds of other things. So MapTool has to defines some pretty
strict limits to what JavaScript macros can do, and that means limiting
what Java classes it can access. My preference would be to err on the side
of being more strict than less, so limit it to

• java.lang excluding things like System, Runtime, SecurityManager, Process etc.
• java.util
• java.util.regex
• java.math
• java.text

From the standard Java packages


And
net.rptools.maptool.macro.api (or what ever package the API lives in)
From MapTool

Everything -- out side of those core Java ones -- needs an API hook sitting
inside net.rptools.macro.api. Now this is obviously going to be slightly
different if OSGI is being used, but the concept will generally be the same,
macros will only be able to access those core Java classes and other plugins
would export a small API that could be access via JavaScript macros.

I am aware that I left out all of GUI creation classes, really any GUI frames/dialogs
should be created though the API similar to what happens now. Anything that requires
really fine grained control of how GUIs etc are created should probably be a plugin
and not a macro. So since everything that a JavaScript macro does to interact with
MapTool anyway requires an API to be written it starts to give you an idea of how
big a job it will be.

The API will more than likely be split into two parts, part of it will be in Java,
this will be the part that exposes certain parts of MapTool, and then there will
be the other part which is a JavaScript wrapper around certain areas that will
do the coercion between Java and JavaScript objects, rather than people writing
macros having do deal with this as it can get messy. Its also possible that standard
higher level routines are build using the lower level API to make certain things
easier while still retaining the possibility for people to use the lower level ones.

Its not a good idea to just expose the current API used by MTScript for the JavaScript
API, as the MTScript API is significantly hobbled by the fact that there is no built in
support for data structures -- and getting your data structures right is the most
important part of programing . So the JavaScript macros have a great scope for
potential improvement.

Another thing that has been discussed a lot is the ability for macros to be triggered
on events. To support this a we need to come up with a framework for macros to resgister
them selves when they want to be informed about events and for MapTool to send them those
events. This brings up another issue, concurrency and synchronisation. JavaScript macros
should never need to worry about either, which of course means that the implementation
would have to take care of any potential issues so that macro writers do not have to.

At the moment all macros run on the GUI thread, this is more of an accident of the fact
that macros were originally only intended to very limited things. Now that macros
do a lot and can run for quite a while the GUI thread is not really the best place for
them to run, while a macro runs it stops the GUI from doing any processing. Using
JavaScript macros this is going to get worse as people write more and more complex macros.
Also you can see some of the side effects of this decision for the need to use deferred
macros for some changes, or how some changes just seem to overwrite others. Jfrazierjr
ran into another of the disadvantages of this design when doing some targeting stuff,
he was unable to pop up a message saying something like "choose your targets" and then
let the user choose targets and continue executing after this, this was because no
GUI processing like responding to mouse events while a macro is running.

Some events -- timers for example -- wont even happen on the GUI thread so the possibility
that a macro might be fired off from somewhere other than the GUI would exist. So how
can you handle this? The GUI thread is actually a pretty good thing to mimic in this case.
Have a single dispatch thread that runs all macros, be the fired off by user typing,
pushing a button, or any event. Any macros that are run at added to a queue, and the
macro thread then processes this queue and runs any macros on it. When I say run macros
I also mean parse macros and anything between [ ] or { } not just JavaScript macros.
This of course would require careful planing on the MapTool side to deal with the
concurrency issues, but this is far preferable to making macro writers perform this task.
Some times you wont be able to get away with doing something on the GUI thread, but
that is ok, the API can handle that, pausing the macro while it waits for the GUI
thread to do the work -- if required -- and most importantly keeping the amount of work
done at a time on the GUI thread small.

So above there is a basic overview of the internal frame work required, but it still
really does describe what JavaScript macros will look like to the people using them.
Thats what my next post will be for

[Craig]

So now the question is "but what does it look like"

Lest start by looking at the current macro setup

It could be described as working something like this

• Player types/pushes button.
• Text gets sent to a "MacroRunner".
• The text is checked to see if it starts with a slash command, if not /say is silently preappended.
• The Java object for this command is retrieved, and checked to see if []/{} substation happens.
• If there is text to be substituted then its sent to MapToolLineParser
• MapToolLineParser then searches for []/{} blocks, and when it find the pass the text inside to the Parser -- its slightly different if there are loop constructs but lets not worry about that.
• The Parser takes this code and then runs it, produces a result and returns a string.
• MapToolLineParser gets this string and replaces the []/{} block it sent to the Parser with it. After all blocks have been dealt with this new string is returned to the "MacroRunner".
• "MacroRunner" passes the string to the java object for the slash command, which then decides what to do with it (e.g. Pre appends SoAndSo says :, and then sends it to the chat output).

So [] and {} essentially mean "run this code and place the result here", [] of course may
do apply extra formating to it. I don't see the meaning of [] or {} changing at all when
it comes to 1.4. There is the strong possibility that some looping constructs may disappear
from [] though. Its also possible that the result is something more structured than a
string to support conditional formatting, but that is neither here nor there.

So the new method of running things will actually look a lot like the old one, where does
JavaScript come in then? As mentioned in the previous post you wouldn't run JavaScript
directly from within [] and {}, an example might help, say you have the following
quite useless JavaScript snippet

Code: Select all

var myD20 = new MTMacroObject("Really bad D20 roller example")
myD20.roll = function(num) {
    var val
    for (i = 0; i < num; i++) 
        val += diceLib.roll(20)

    return val
}

MapTool.Parser.registerFunction("rollD20", myD20.roll, myD20, MapTool.Parser.UNTRUSTED);

Lets just say that that creates the rollD20() parser function as an untrusted function.
It probably wouldn't look exactly like that as I can see several problems with it already,
but it will do.

So when some one types in

Code: Select all

This is my roll of 6 dice! [myD20(6)]

It will call the new function which will return the result and insert that into the text
that is sent to the chat output a lot like it happens now.

There are some cases where the GM will want to use just some plain old JavaScript this
could be achieved with {{ }} or some thing similar, I am not sure I like the idea of
[j: ] as I would want it made clear that any JavaScript code can not have any of the
[e, t ] etc formatting options applied to them. It would really be more useful for testing
your macros, or quickly improvising something rather than your every day macros. This would
only be able to run if the macro is trusted so players can not insert their own JavaScript.

Now we come to another problem, how do we find the JavaScript Object we want to interact
with them, and the directly related question of scope and where we store all these objects.
For the Parser finding the object it needs is easy, every defined parser function provides
the exact location that the Parser needs to call. Scope thought is another issue, really
all scripts should have their own scope, this separates them all from each other and saves
us all from a lot of debugging induced heart burn. This has no effect on the Parser, but
what about JavaScript libraries? What if I want to write a library of functions for others
to use, having things in different scopes that can not see each other is obviously going
to cause us some problems. Using the functions that are export to the Parser is not a
viable method for calling another JavaScript object either. There are ways to solve this
by using closures and creating functions to retrieve "library objects", I don't want to
go into all the of the details here as this is getting long enough, and such a discussion
requires more thought first, but keep in mind its something that needs to be solved, and
effects how MapTool imports JavaScript.

I don't think you will ever see a way to run an external JavaScript command using a /js
slash command. Giliath commented that he didn't like it, I know Trevor before hand has
said he wants everything to remain with in the campaign file, and I completely agree
with both of their sentiments. So the question becomes how do we make it as easy as
possible to add JavaScript to your campaign? By using methods that already exist, its
probably no surprise to any one that the asset manager in MapTool will be expanded to
hold more than just images -- this is going based on many posts that have been made
about future functionality -- to support things like hand outs, so there is no reason
it can not contain JavaScript scripts, which could be added via drag and drop or the
resource library the way that tokens are currently handled. I am assuming that the
resource library would be expanded to support other objects if the asset manager is,
I don't think this is a wild assumption. This would allow you to edit the JavaScript
macros in your favorite editor and easily drop them into your campaign -- or update
them from a repository etc...

I know that this doesn't meet a lot of peoples desires to have every part of the GUI
be script able, but macros should facilitate running your game, we are not creating an
eclipse like platform here For things that are beyond the macro language
plugins should be considered, I think its ok to say to people if you want to create
windows with great control over how they look/behave or spawn background threads or
support different types of media then you would need to write a plugin rather than
a set of macros.

[JML]

Reserved, yes I know you are now thinking "oh no there is more?"

Well, I'm glad there's going to be more

Thank you for the detailed insight Craig, and for taking time to do it.

[Hawke]

What sort of speed/stability increase would going to JS give us? Right now with macros that is my greatest concern.