DDi Compendium API and Java Classes?

Progress reports and musings from the developers on the current gaming tools.

Moderators: dorpond, trevor, Azhrei

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

DDi Compendium API and Java Classes?

Post by Blakey »

Does anyone know if anyone has written a Java class to grab stuff from the DDi API? I'm specifically after grabbing monsters from the Compendium and don't want to write my own class if someone has already done all the work for me...

Cheers
Blakey
The guy in the green hat.

neofax
Great Wyrm
Posts: 1694
Joined: Tue May 26, 2009 8:51 pm
Location: Philadelphia, PA
Contact:

Re: DDi Compendium API and Java Classes?

Post by neofax »

I know VCC uses JAVA and pulls the Monsters out of the compendium using a Firefox extension.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Any clues as to what VCC is? I looked in google and found:

VCC - Wikipedia, the free encyclopedia VCC is a three-letter acronym that may mean: Common-collector voltage (VCC)
The Veteran Car Club of Great Britain
The Veteran Cycle Club (V-CC) home page
Vancouver Community College - B.C.'s No. 1 College
Voice for the Child in Care (VCC)

None of these seemed likely to have written a DDi Java interface... :D
The guy in the green hat.

neofax
Great Wyrm
Posts: 1694
Joined: Tue May 26, 2009 8:51 pm
Location: Philadelphia, PA
Contact:

Re: DDi Compendium API and Java Classes?

Post by neofax »

Here it is: VCC

User avatar
jstgtpaid
Giant
Posts: 142
Joined: Sun Jun 22, 2008 1:23 am
Location: Tampa, FL

Re: DDi Compendium API and Java Classes?

Post by jstgtpaid »

That sounds very interesting. It would be awesome if we could use the DDI content to easily create monster tokens and such. There is a thread about parsing a PDF and also DDI text and creating a sheet from that http://forums.rptools.net/viewtopic.php?f=8&t=13237. So I would imagine some of the heavy lifting is already done.

The fella at the VCC project indicates that he wrote an extension for firefox that yanks the data from DDI. I would imagine that code is available. A little bit of cutting and pasting and you may be closer than you think...
When the boogeyman goes to bed everynight, he checks his closet for Chuck Norris.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

I'm doing this more as an intellectual exercise and for practicing Java than anything else, so I'll probably not be stealing other people's code, even though I could. I might steal it for peeking at though and seeing how they do stuff.

I've so far figured out how to run a query on the compendium to get an XML file with a set of results in it - so I can ask it to give me a Goblin Blackblade and I'll get an XML file back which has the monster's ID number. This can then be sent as a query to get back the HTML page for that particular monster. At the moment this doesn't quite work as I'm hitting a wall trying to get past the DDi authentication. I'm currently looking into how to login programatically to get around this.

Once that is done I should be able to tell my java class 'get me a goblin blackblade' and have it return the HTML code for the monster. I'll then have to parse that into something which makes sense to MapTool but that is another set of issues entirely as I have to start thinking about frameworks at that stage.

Simultaneously with this, as part of the same package, I'm writing a java parser to parse the Character Builder save files and build a PC object. That is coming along nicely too, although there is a problem getting the details of the actual powers as these are not kept in the save file. But I'm thinking I might be able to pop off to the Compendium and grab the HTML for the power once I have the power name. We shall see.

Maybe I'll make a new thread where I can track my progress. Would anyone be interested in that?

Cheers
Blakey

P.S. This will be slow and steady work and I'm liable to just stop working on it at any moment so don't expect anything from me on this effort at all!! :D
The guy in the green hat.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

On this subject, can anyone figure out how to effectively pass your login details to the compendium on the URL line of a browser? I have figured this much so far:

If I want to go to a given power (say Divine Armor, which is power number 959), and also login in the same breath, I use this URL:

Code: Select all

http://www.wizards.com/dndinsider/compendium/login.aspx?page=power&id=959


That will take me to the login page and then when I have logged in, it will forward me to the power page and select power 959. However I really want to pass my login and password details in here too, so that it takes me straight to the page without asking for any input, regardless of whether you are logged in or not - effectively guaranteeing a login as it goes. Something like this:

Code: Select all

http://www.wizards.com/dndinsider/compendium/[email protected]&password=myPassWord&page=power&id=959


I just don't know what varibales I need to set up to pass through to the login page automatically. Can anyone help?

Cheers
Blakey
The guy in the green hat.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:
  1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
  2. Get the cookie data from the response object passed back from the site.
  3. Store the cookie data in a global value for future use
  4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
  5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

jfrazierjr wrote:The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:
  1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
  2. Get the cookie data from the response object passed back from the site.
  3. Store the cookie data in a global value for future use
  4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
  5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"


Thanks.

That makes sense but unless I'm missing something, it is step 1 that I don't know how to do. How do you login to the web page (www.wizards.com/.../login.aspx) with your username and password unless you use "www.wizards.com/.../[email protected]&password=pw" - and I don't know what parameters to pass in here as I can't get this to work by hand.

Or am I getting more confused than I think I am?

Cheers for the help!
The guy in the green hat.

User avatar
aku
Dragon
Posts: 856
Joined: Wed Nov 15, 2006 9:03 am
Contact:

Re: DDi Compendium API and Java Classes?

Post by aku »

I think you're confusing yourself. What i understand JF to be saying is basically, manually login to the site, and get the cookie (mmm, cooooookkkkieeeeee) that that login generates.

Thanlis
Giant
Posts: 240
Joined: Tue Mar 24, 2009 3:34 pm

Re: DDi Compendium API and Java Classes?

Post by Thanlis »

I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way. :)
Reed (halfling sorcerer P3) // Collin (human fighter P2) // Cine (eladrin psion H2)
Sirath (deva shaman H1) // Alesk (dragonborn cleric H3) // Kevin (halfling barbarian H1)

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

aku wrote:I think you're confusing yourself. What i understand JF to be saying is basically, manually login to the site, and get the cookie (mmm, cooooookkkkieeeeee) that that login generates.


Err, I can login but how do I generate a cookie from that? Sorry for the stupid question!!!
The guy in the green hat.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

Blakey wrote:
jfrazierjr wrote:The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:
  1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
  2. Get the cookie data from the response object passed back from the site.
  3. Store the cookie data in a global value for future use
  4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
  5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"


Thanks.

That makes sense but unless I'm missing something, it is step 1 that I don't know how to do. How do you login to the web page (http://www.wizards.com/.../login.aspx) with your username and password unless you use "www.wizards.com/.../[email protected]&password=pw" - and I don't know what parameters to pass in here as I can't get this to work by hand.

Or am I getting more confused than I think I am?

Cheers for the help!


Ok... First, you have to go to the login page in the site and view the html source to find the username/password(which may not be called that!!!!!) fields as well as the enclosing <form> element(so you can get the action and method). In many cases, sites may ONLY allow credentials to be passed in via POST method and not GET (thats what you are doing when you add the things to the end of a URL).

Once you have that information, you can then begin your code to login programmaticlly.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Thanlis wrote:I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way. :)


I can easily figure out how to do a POST in java. The question is what do I need to post?

The login.aspx file labels the two fields 'email' and 'password'. Is that all I need to pass in?

Cheers
Blakey
The guy in the green hat.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

Blakey wrote:
Thanlis wrote:I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way. :)


I can easily figure out how to do a POST in java. The question is what do I need to post?

The login.aspx file labels the two fields 'email' and 'password'. Is that all I need to pass in?

Cheers
Blakey

You should pass EVERY form element on the page. Example:

Code: Select all

<input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" /> 

should be: InsiderSignin=Sign In


The BEST way to do this is to use Java to do a GET on the login page itself. Then in the response document, interate though the page's form elements and grab each <input> tags name and value. Then, post those name/value pairs to the login.aspx page. When you get the response from this page, pull the cookie values out of the header and store for later use.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

Post Reply

Return to “Developer Notes”