d20 StatBlock Importer: Pathfinder/DnD3/3.5

Show off your RPG maps, campaigns, and tokens. Share your gaming experiences and stories under General Discussion and save this forum for things you've created that may help others run their own games. Use the subforums when appropriate.

Moderators: dorpond, trevor, Azhrei, Gamerdude

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

I have seen other text parsers to pull stats out of text, but mostly for 4e (From WotC's DDi online resources), or designed for specific sources or frameworks. I thought I'd try to make a generic one for Pathfinder / DnD 3e / 3.5, seeing as I still have a bunch of old material lying around, and being able to copy-paste is so much easier than entering data manually! And, it was a rewarding way to learn regex :P. I've been tweaking this for a while, but I think it's ready to put out there for others to use and contribute.

Features & notes:
  • includes a 'Help' macro, which produces a frame with reference instructions and macro links to try it out.
  • "Preview & Map" UI (not shown in demo) to display parsed values before applying them, and also map fields to destination Properties on the tokens. These settings are used by the 'Basic' Processing macro included in the Lib: token, but can easily be changed or ignored by a custom framework interpreter.
    • Until a 'property map' is saved, this will try to match property names on the token with field names produced by the parser. If no match is found, the field will not be applied to any property. Once a 'map' is saved, fields will be assigned to the token property specified in the map (or none).
  • If a Name is parsed, it is used to set the GM Name (not the Token Name).
  • The original text input is appended to the end of the GM Notes (for future reference).
  • Modular structure to facilitate adding support for additional sources, or customizing the processing for a particular framework (see below).

It currently has parsers for the following sources:

* The pdf parsers are a bit buggy, and pdf text encoding and inconsistent formats sometimes cause trouble with a few of the properties, however, all the basic stats (Ability Scores, saves, etc.) are pretty reliable.

I don't have one for Pathfinder PDFs yet, mostly because I haven't found an example to test (I only have the core material, none of the accessories written for the PF system). It would be good to know if the Pathfinder parser can handle pdf sources as well as the online ones.
There is also a small section that can try to "Auto-detect" the source, based on the string contents (certain labels specific to the different StatBlock formats), but a specific source can always be selected, just to be sure.

Download:
Feel free to try it out, and share anything you think would improve it.
Lib-jaw-StatBlockImporter-1.2-1.3b56.rptok
Lib:Token v1.2. Includes a basic processing macro (with mapping UI), and an example of a processing macro for my homebrew campaign, if you want to see something more involved.
(57.06 KiB) Downloaded 484 times


How to Install & Use
1. Download the Lib:token linked above, and drag the .rptok file onto a map in your campaign.
2. select the Lib:token, and click on the "onCampaignLoad" macro to define the functions. You only need to do this the first time you drag the Lib:token into your campaign file (it runs automatically when you open a campaign file with the lib:token in it).
3. Drag the macro named "Import StatBlock text", in the "Campaign Macros" group, over to your campaign macros (or token macros or wherever you can access it while selecting multiple tokens).
3.1 Alternatively, impersonate the Lib:Token so you can access the "Import StatBlock text" macro while selecting other tokens.

4. To use this tool, select the tokens to which you want to apply statBlock values, and then activate the "Import StatBlock text" macro (from wherever you dragged it to, or directly from the Lib:Token as above). This generates the input dialog where you can paste in your Stat Block text, choose the source, preview the parsed values & assign them to token properties, or apply the values directly.
The "Import StatBlock" macro (in the the "Campaign Properties" group) is really the only macro you need to use, but you have to select the tokens to which you want to apply the statblock values. The trick is being able to select the target tokens, and activate this single macro at the same time.

How do I customize this for my favourite framework?
see this post for more details.

Demo:
Figure 1. Empty token and interface ready for input.
empty token, ready for input
empty token, ready for input
Screen shot 2010-06-20 at 10.00.22 PM.png (277.94 KiB) Viewed 14281 times

Figure 2. Text Input, and available options for the source. The Auto-detect should work fine for this text from pfsrd.com
Input text, from www.d20pfsrd.org.  Available options are seen, but the Auto-detection should work fine.
Input text, from www.d20pfsrd.org. Available options are seen, but the Auto-detection should work fine.
Screen shot 2010-06-20 at 10.00.58 PM.png (254.03 KiB) Viewed 14281 times

Figure 3. Values applied. Note the token has automatically changed size, based on the value pulled from the statBlock.
New Values Applied!
New Values Applied!
Screen shot 2010-06-20 at 10.01.19 PM.png (356.48 KiB) Viewed 14281 times



Concept & Structure:
I divided the entire import process into 3 main sections / macros:
  1. Main Interface & coordination:Generates interface, hands data off to other macros and sends the results somewhere appropriate.
  2. Parsing: pulls stats out of the text and assembles a json object of the values with sensible labels (keys), using a separate macro for each possible source.
  3. Processing ("Framework Interpreter") : takes the json object produced in the previous step and converts it to an object with property names and values. This is the part that is specific to a given framework. The Lib:token for download includes a basic version, and an example for my homebrew framework, if you want to see something fancier.
The json object produced at the end is used to apply values to properties of the selected tokens. i.e. values can be applied to multiple tokens in one fell swoop.

This way, I figure it makes it easier to add more parsers for different sources, and customize the processing for your preferred framework, while keeping the code neatly organized.
Last edited by biodude on Sun Jun 27, 2010 3:35 am, edited 8 times in total.
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

User avatar
Azhrei
Site Admin
Posts: 12058
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: StatBlock Importer: Pathfinder/DnD3/3.5

Post by Azhrei »

Very cool, biodude.

If you search the forum for statblock2token you'll find another such tool, although I don't recall off-hand who wrote it. :(

The other tool works pretty well but the regex doesn't handle skills with spaces in their names very well nor does it handle the subskills correctly when used in Lindsay's D&D/PF framework. (Lindsay stores subskills differently from the others, IIRC. I looked into fixing this but didn't want to spend the time on it. I need to have myself cloned. Twice. ;))

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

Thanks for the reminder, Azhrei. Funny thing is, I had bookmarked that thread, but never went back to it. I think what frustrated me with that one was that it was specific to lmarkus' Framework, and I wanted something that was easier to ... customize. Given the different sources and potential for different frameworks, the major 'novel' contribution here is the way the code is divided up, although mine also supports DnD v3/3.5 style statblocks as well :D.
The core of the parsing, as usual, is regex, however, so I should definitely have a closer look at statblock2token and see what we can learn from each other.
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

neofax
Great Wyrm
Posts: 1694
Joined: Tue May 26, 2009 8:51 pm
Location: Philadelphia, PA
Contact:

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by neofax »

If you need some Pathfinder PDF sources, Paizo.com has at least 4 for free that I know of. They are for low level characters though, but at least you can test your program.

User avatar
aliasmask
Deity
Posts: 8624
Joined: Tue Nov 10, 2009 6:11 pm
Location: Bay Area

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by aliasmask »

Would the 3.0 or 3.5 importer work for http://www.dinglesgames.com/ monster/npc generator? Do you also dump the stat block in to the GM notes?

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

aliasmask wrote:Would the 3.0 or 3.5 importer work for http://www.dinglesgames.com/ monster/npc generator?

Not yet, because the labels and delimiters they use don't correspond to 'official' DnD statblocks. Fortunately, this is why I structured it the way I did: it should relatively easy to write a parser for it, using the existing ones as a template and "plug it in". Sometimes, I find it easier to write regex specific to certain output, than try to write one set to handle all the myriad possibilities.
You are welcome, and encouraged to try to write one yourself (feel free to download and copy the macro code). I might be willing to try, although my time is rather limited: I certainly won't get to it before the weekend, but I might have some time then.

Eventually, I expect it should be also possible to have parsers that can handle other software output (PC-Gen, Crystal Ball). This seems to me to be the easiest way to transfer data into MapTool from these sources.

Do you also dump the stat block in to the GM notes?

Why yes, it all gets appended to the end of the GM Notes (nothing is overwritten).
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

User avatar
Azhrei
Site Admin
Posts: 12058
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by Azhrei »

biodude wrote:I expect it should be also possible to have parsers that can handle other software output (PC-Gen, Crystal Ball).

PCGen can output a character in any text-based format you choose using a user-defined template.

mosat has produced a template that ships with PCGen now. It generates the XML for a token -- again, using Lindsay's FW for properties and such.

I suppose if you're going to make it FW-agnostic, you'll need a UDF for each portion of the statblock. Each UDF would be called with parameters to indicate the type of data and the value to be stored in it. So something like bio.setSkill(name, value) and bio.setWeapon(name, bonus, damage, critrange, ...)

That way a different FW plugin can be provided as a set of UDFs and your main regex monster doesn't have to change. And picking good names for the UDFs will self-document the functionality of said monster as well. ;)

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

Azhrei wrote:
biodude wrote:I expect it should be also possible to have parsers that can handle other software output (PC-Gen, Crystal Ball).

PCGen can output a character in any text-based format you choose using a user-defined template.

mosat has produced a template that ships with PCGen now. It generates the XML for a token -- again, using Lindsay's FW for properties and such.

Sadly, framework-specific, which is my main complaint :( . With the system I posted here, as long as you can get PCGen to output a 'standard' PF/DnD statBlock, this tool can read it just fine. I just figure that translating data into a framework set-up should be done within MapTool: people who build frameworks are obviously versed in MapTool scripting, not necessarily coercing output from other programs. So, the only thing one really needs is a translator to get the data into a format that MapTool can work with, then framework developers can do whatever they want with it. Once again, regex saves the day (well, I thought it was pretty cool, anyway).
Azhrei wrote:I suppose if you're going to make it FW-agnostic, you'll need a UDF for each portion of the statblock. Each UDF would be called with parameters to indicate the type of data and the value to be stored in it. So something like bio.setSkill(name, value) and bio.setWeapon(name, bonus, damage, critrange, ...)

That way a different FW plugin can be provided as a set of UDFs and your main regex monster doesn't have to change. And picking good names for the UDFs will self-document the functionality of said monster as well. ;)


I tried to make it as FW-agnostic as I could: the parsed values are stored in a json object, which is passed to a generic "StatBlock: Process" macro: you can just replace the contents of this macro with an 'interpreter' for your favourite framework, and process the values however you want. I didn't bother defining it as a UDF, since it didn't seem necessary: it is called directly within the main macro, once the input dialog is submitted.
An example for my homebrew franken-framework (in progress) is included on the lib:token for download, if you want to have a look. It essentially does what you describe: sets skills, using the data format in my framework, and sets the appropriate token properties, which are not necessarily the same as the json object used to store all the parsed values from the statBlock. The parser currently does not do anything with attack text, but FullBleed is convincing me to try to tackle that next...

In theory, I could add the option to support multiple "processing" macros for different frameworks, but I figured if one is going to use this within a campaign, it was more likely for a user to switch input sources, rather than framework destinations. And, it is relatively simple to set it up for one framework, and then it's good to go for that campaign.
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

User avatar
Azhrei
Site Admin
Posts: 12058
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by Azhrei »

biodude wrote:the parsed values are stored in a json object, which is passed to a generic "StatBlock: Process" macro: you can just replace the contents of this macro with an 'interpreter' for your favourite framework, and process the values however you want.

Ah. That sounds like a good plan. It makes the connection a little fragile between the parser and the called macro (changes to the format of the JSON object could break all implementations of the called macros), but it's probably the fastest way to move the data around.

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

Azhrei wrote:
biodude wrote:the parsed values are stored in a json object, which is passed to a generic "StatBlock: Process" macro: you can just replace the contents of this macro with an 'interpreter' for your favourite framework, and process the values however you want.

Ah. That sounds like a good plan. It makes the connection a little fragile between the parser and the called macro (changes to the format of the JSON object could break all implementations of the called macros), but it's probably the fastest way to move the data around.


Good point. I wouldn't mind some feedback on the structure of said JSON object. I would really suck to have an interpreter break because the parser changed.

Using the PF Giant Eagle as an example, it looks like this:
[spoiler=Parsed JSON structure]

Code: Select all


 
{ "Sex":"",
"Alignment":"NG",
"Size":"Large",
"Space":"",
"Reach":" 5 ft.",
"CR":3,
"XP":800,
"Initiative":3,
"Senses":" low-light vision; Perception +15",
"Movement":" 10 ft., fly 80 ft. (average)",
"Strength":18,
"Dexterity":17,
"Constitution":12,
"Intelligence":10,
"Wisdom":15,
"Charisma":11,
"BAB":4,
"CMB":9,
"CMD":22,
"HP":26,
"HD":4,
"AC":15,
"AC_touch":12,
"AC_flat":12,
"Armor.natural":3,
"Fortitude":5,
"Reflex":7,
"Will":3,
"SR":"",
"SpecialAbilities":"",
"SpecialQualities":" evasion",
"Skills":" Fly +8, Perception +15, Sense Motive +4; Racial Modifiers +4 Perception",
"Feats":" Alertness, Flyby Attack",
"BasicAttack":" 2 claws +7 (1d8+4), bite +7 (1d6+4)",
"FullAttack":"",
"Languages":" Auran (cannot speak)",
"":"",
"Type":"magical beast",
"Skills.totals": { 
    
"Fly":8,
    "Perception":15,
    "Sense Motive":4,
    "Racial Modifiers":
    
}
}
 
[/spoiler]
Feel free to suggest better names for the fields. Is anything obvious missing? Is this a useful format for people to convert to their own frameworks (if you are json-averse, it can easily be converted to a StrProp and values accessed that way, except for the nested stuff, which probably works better in a json structure anyway)?
Note that some fields are empty and actually there for DnD statBlocks. Conversely, the "CMB" field would also be present for the product of a DnD StatBlock, it would just be empty.
Although I do parse the Skills out individually, the original text is still there. I intend to do much the same thing with attacks, if I ever figure out how to parse those fully: a separate entry would contain the parsed values, but the raw text would still be available.
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

Zandel
Giant
Posts: 133
Joined: Thu May 13, 2010 1:52 pm

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by Zandel »

Does this Macro assume you have created all the appropriate Token properties ahead of time, or does it create them?

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

Zandel wrote:Does this Macro assume you have created all the appropriate Token properties ahead of time, or does it create them?


Depends what you mean by 'create'...
Short answer: Token properties are indeed set, but they won't necessarily appear in the statsheet, or the MapTool UI, until they are defined in the Campaign Properties.

Long answer:
The "Processing" macro is intended to be specific to a framework / set of Campaign Properties. I included a very basic one that takes the parsed values and sets properties of the same name to those values. It uses [wfunc]setProperty[/wfunc], so those property names are 'created' on the token. Nevertheless, this does not mean that they automatically appear in the pop-up statsheet, or are accessible in the MapTool UI under "Properties" when you double-click a token to edit them: for these things to happen, the properties of the same name must be defined in the Campaign Properties (Frameworks often have a set of Campaign Properties for convenience and integration with the MapTool UI, but this is not always necessary). I don't know how, or even if it is possible, to set Campaign Properties via a macro (other than switching the Property Type, but that's not quite the same thing).

Ultimately, the "Processing" macro is where the parsed values are converted and applied to a token, according to the desired Framework. Each framework expects, and manages, it's own set of token properties. The focus of this importer is just on the parsing: I intended for people to develop "Processing" macros to 'interpret' the parsed values into a data structure appropriate for specific frameworks. This makes the process more modular, and makes it easier to adapt this tool to any framework.

Does this make sense? Should I write an interpreter for a popular framework as an example to make it more useful?
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

Zandel
Giant
Posts: 133
Joined: Thu May 13, 2010 1:52 pm

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by Zandel »

It makes perfect sense. I don't use a framework, but I do have a lot of custom macros that use properties I have defined and there is of course overlap, in many cases I may have used a different name for the same property. So as I understand it, I can either rename my properties to match the output of the parser which would require me to modify all of my custom macro's that use those properties, or go into your macro and change the output to my property names correct? Unless there is a mapping UI

User avatar
biodude
Dragon
Posts: 444
Joined: Sun Jun 15, 2008 2:40 pm
Location: Montréal, QC

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by biodude »

Zandel wrote:It makes perfect sense. I don't use a framework, but I do have a lot of custom macros that use properties I have defined and there is of course overlap, in many cases I may have used a different name for the same property. So as I understand it, I can either rename my properties to match the output of the parser which would require me to modify all of my custom macro's that use those properties, or go into your macro and change the output to my property names correct?

Oh god, pick #2! Yes, that's the idea: change the "Processing" macro to do whatever you want with the parsed data. In the lib:token for download, there is a basic version provided for functionality, as well as the one I use for my framework, if you want to see something fancier.
Basically, that macro just has to create a json object with field names matching the token property you want the values to go into (The Main macro does the rest, although I may change this if people want). So, you just have to get the 'processing' macro to make a json with your property names.
You can also just return an empty json and do all the property-setting yourself, or do other things (apply states, change the size, etc.) beyond just setting property values: it all depends on how you want to represent the data in your system.

Zandel wrote:Unless there is a mapping UI

This is certainly possible, but I figured that the possible data structures for frameworks could exceed what I could imagine and expect. I figure anyone savvy enough to write their own macros should be able to convert the data produced by this parser to fit their system.
Hm, I might make one anyway, at least for the 'basic' interpreter: framework developers can always replace it with their own. The mapping UI would also make a useful reference for developers wanting to write their own custom interpreter.
Thanks for the idea.
"The trouble with communicating is believing you have achieved it"
[ d20 StatBlock Importer ] [ Batch Edit Macros ] [ Canned Speech UI ] [ Lib: Math ]

Zandel
Giant
Posts: 133
Joined: Thu May 13, 2010 1:52 pm

Re: d20 StatBlock Importer: Pathfinder/DnD3/3.5

Post by Zandel »

How do you actually view the macros that are included with the Library token? I see how to run them, but how do I view the code?

Post Reply

Return to “User Creations”