DDi Compendium API and Java Classes?

[Blakey]

Does anyone know if anyone has written a Java class to grab stuff from the DDi API? I'm specifically after grabbing monsters from the Compendium and don't want to write my own class if someone has already done all the work for me...

Cheers
Blakey

[neofax]

I know VCC uses JAVA and pulls the Monsters out of the compendium using a Firefox extension.

[Blakey]

Any clues as to what VCC is? I looked in google and found:

VCC - Wikipedia, the free encyclopedia VCC is a three-letter acronym that may mean: Common-collector voltage (VCC)
The Veteran Car Club of Great Britain
The Veteran Cycle Club (V-CC) home page
Vancouver Community College - B.C.'s No. 1 College
Voice for the Child in Care (VCC)

None of these seemed likely to have written a DDi Java interface...

[neofax]

Here it is: VCC

[jstgtpaid]

That sounds very interesting. It would be awesome if we could use the DDI content to easily create monster tokens and such. There is a thread about parsing a PDF and also DDI text and creating a sheet from that http://forums.rptools.net/viewtopic.php?f=8&t=13237. So I would imagine some of the heavy lifting is already done.

The fella at the VCC project indicates that he wrote an extension for firefox that yanks the data from DDI. I would imagine that code is available. A little bit of cutting and pasting and you may be closer than you think...

[Blakey]

I'm doing this more as an intellectual exercise and for practicing Java than anything else, so I'll probably not be stealing other people's code, even though I could. I might steal it for peeking at though and seeing how they do stuff.

I've so far figured out how to run a query on the compendium to get an XML file with a set of results in it - so I can ask it to give me a Goblin Blackblade and I'll get an XML file back which has the monster's ID number. This can then be sent as a query to get back the HTML page for that particular monster. At the moment this doesn't quite work as I'm hitting a wall trying to get past the DDi authentication. I'm currently looking into how to login programatically to get around this.

Once that is done I should be able to tell my java class 'get me a goblin blackblade' and have it return the HTML code for the monster. I'll then have to parse that into something which makes sense to MapTool but that is another set of issues entirely as I have to start thinking about frameworks at that stage.

Simultaneously with this, as part of the same package, I'm writing a java parser to parse the Character Builder save files and build a PC object. That is coming along nicely too, although there is a problem getting the details of the actual powers as these are not kept in the save file. But I'm thinking I might be able to pop off to the Compendium and grab the HTML for the power once I have the power name. We shall see.

Maybe I'll make a new thread where I can track my progress. Would anyone be interested in that?

Cheers
Blakey

P.S. This will be slow and steady work and I'm liable to just stop working on it at any moment so don't expect anything from me on this effort at all!!

[Blakey]

On this subject, can anyone figure out how to effectively pass your login details to the compendium on the URL line of a browser? I have figured this much so far:

If I want to go to a given power (say Divine Armor, which is power number 959), and also login in the same breath, I use this URL:

Code: Select all

http://www.wizards.com/dndinsider/compendium/login.aspx?page=power&id=959

That will take me to the login page and then when I have logged in, it will forward me to the power page and select power 959. However I really want to pass my login and password details in here too, so that it takes me straight to the page without asking for any input, regardless of whether you are logged in or not - effectively guaranteeing a login as it goes. Something like this:

Code: Select all

http://www.wizards.com/dndinsider/compendium/[email protected]&password=myPassWord&page=power&id=959

I just don't know what varibales I need to set up to pass through to the login page automatically. Can anyone help?

Cheers
Blakey

[jfrazierjr]

The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:

1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
2. Get the cookie data from the response object passed back from the site.
3. Store the cookie data in a global value for future use
4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"

[Blakey]

The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:

1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
2. Get the cookie data from the response object passed back from the site.
3. Store the cookie data in a global value for future use
4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"

Thanks.

That makes sense but unless I'm missing something, it is step 1 that I don't know how to do. How do you login to the web page (www.wizards.com/.../login.aspx) with your username and password unless you use "www.wizards.com/.../[email protected]&password=pw" - and I don't know what parameters to pass in here as I can't get this to work by hand.

Or am I getting more confused than I think I am?

Cheers for the help!

[aku]

I think you're confusing yourself. What i understand JF to be saying is basically, manually login to the site, and get the cookie (mmm, cooooookkkkieeeeee) that that login generates.

[Thanlis]

I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way.

[Blakey]

I think you're confusing yourself. What i understand JF to be saying is basically, manually login to the site, and get the cookie (mmm, cooooookkkkieeeeee) that that login generates.

Err, I can login but how do I generate a cookie from that? Sorry for the stupid question!!!

[jfrazierjr]

The way many websites work is by passing a cookie back in the response. You then have to pass that cookie back in your next request. Flow is something like this:

1. Open connection to Site, passing in username/password(typically to the login page URL of the site)
2. Get the cookie data from the response object passed back from the site.
3. Store the cookie data in a global value for future use
4. create a new connection to the next page you wish to go to, passing the cookie data in as part of your request headers.
5. Repeat step 4 for each page you wish to access.

Google "java pass cookie to URL"

Thanks.

That makes sense but unless I'm missing something, it is step 1 that I don't know how to do. How do you login to the web page (http://www.wizards.com/.../login.aspx) with your username and password unless you use "www.wizards.com/.../[email protected]&password=pw" - and I don't know what parameters to pass in here as I can't get this to work by hand.

Or am I getting more confused than I think I am?

Cheers for the help!

Ok... First, you have to go to the login page in the site and view the html source to find the username/password(which may not be called that!!!!!) fields as well as the enclosing <form> element(so you can get the action and method). In many cases, sites may ONLY allow credentials to be passed in via POST method and not GET (thats what you are doing when you add the things to the end of a URL).

Once you have that information, you can then begin your code to login programmaticlly.

[Blakey]

I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way.

I can easily figure out how to do a POST in java. The question is what do I need to post?

The login.aspx file labels the two fields 'email' and 'password'. Is that all I need to pass in?

Cheers
Blakey

[jfrazierjr]

I have some perl code that scrapes the Compendium. The sign in process uses POST rather than GET, so you can't pass the parameters via the URL. I don't know how you'd send a POST request in Java, but I'm sure there's a way.

I can easily figure out how to do a POST in java. The question is what do I need to post?

The login.aspx file labels the two fields 'email' and 'password'. Is that all I need to pass in?

Cheers
Blakey

You should pass EVERY form element on the page. Example:

Code: Select all

<input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" /> 

should be: InsiderSignin=Sign In


The BEST way to do this is to use Java to do a GET on the login page itself. Then in the response document, interate though the page's form elements and grab each <input> tags name and value. Then, post those name/value pairs to the login.aspx page. When you get the response from this page, pull the cookie values out of the header and store for later use.