DDi Compendium API and Java Classes?

Progress reports and musings from the developers on the current gaming tools.

Moderators: dorpond, trevor, Azhrei

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

If you refresh the page to the __VIEWSTATE and/or __EVENTVALIDATION values change? most likely it's sending you a random string that you need to pass back to authenticate yourself. If that's the case if you do nothing but refresh the page, that should be the only thing that changes, then you'll have your culprit. You may need to open two windows side by side and only refresh one so you can compare the two.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Still hacking...

A question: on a POST, does the order of the fields matter?

Cheers
The guy in the green hat.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

I just can't figure this out. This is what I'm now doing:

1. Open the powers.aspx?id-805 page (Divine Challenge).
2. Open a Reader to that URL - this gets me redirected to the login.aspx page of course.
3. Get the Cookies from that URL (note: I only see 1 cookie here, where FF seems to see 2).
4. Read in the content of the HTML page I got redirected to.
5. Scan that source for all the input tags and build my POST data.
6. Scan that source for the action= tag to see where we need to POST to (login.aspx?page=power&id=805)
7. Drop the read connection.
8. Open a write connection to the new URL (login.aspx?page=power&id=805)
9. Add the cookies to the connection.
10. Write the POST data to the connection.
11. Open a reader on the same connection.
12. Read the connection - expecting to see the content of the power, but find I am STILL getting the content of the login.aspx page.

Arggh!!!!

Questions arrising:

1. Is this the right set of steps?
2. I think the cookie situation is weird - Java is reporting less cookies than FF. Am I doing something wrong here?
3. Should I give up?!?!?


For those that care, here is the complete Java Class: Note that the init() method does all the work really.

Code: Select all

package info.rodinia.tokenmaker;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;

// a simple class which models a compendium entry
public class CompendiumEntry {

    private String myHTMLSource;
    private String myURL;
    private boolean loggedIn = false;
    private static final String email = "[email protected]";
    private static final String password = "password=myPasswd";
    private static final String baseURL = "http://www.wizards.com/dndinsider/compendium/";

    // constructor takes a URL and builds an entry from it.
    public CompendiumEntry(String stringUrl) {
	myURL = stringUrl;
	init();
    }

    // give back the source
    public String getHTML() {
	return myHTMLSource;
    }

    private void init() {
	try {
	    // Open a connection to the page
	    URL url = new URL(myURL);
	    URLConnection conn = url.openConnection();

	    // Open a reader to the page - this will actually be the redirected
	    // login page
	    BufferedReader reader = new BufferedReader(new InputStreamReader(
		    conn.getInputStream()));

	    // Get the Cookies from the Page
	    String cookies = null;
	    String headerName = null;
	    for (int i = 1; (headerName = conn.getHeaderFieldKey(i)) != null; i++) {
		if (headerName.equals("Set-Cookie")) {
		    cookies += conn.getHeaderField(i);
		}
	    }
	    System.out.println("Cookies List: " + cookies);

	    // read the contents of the page to get the formData we need to push back
	    String content = null;
	    String inputLine;
	    while ((inputLine = reader.readLine()) != null) {
		content += inputLine + "\n";
	    }
	    reader.close();
	    System.out.println("content: " + content);

	    // parse the page for <input> and "action=" tags ready for our POST phase
	    String formData = parseFormData(content);
	    formData += email + "&" + password;
	    formData = URLEncoder.encode(formData, "UTF-16");
	    String actionUrl = parseAction(content);
	    actionUrl = baseURL+actionUrl;
	    System.out.println("formData: " + formData);
	    System.out.println("actionUrl: " + actionUrl);

	    // now drop the connection to the old page and connect up to the new one, POSTing our data and setting out cookies.
	    url = new URL(actionUrl);
	    conn = url.openConnection();
	    
	    // Add the required cookies to the writer connection
	    conn.setRequestProperty("Cookie", cookies);
	    conn.setDoOutput(true);	// let us write
	    
	    // Open a writer to the redirected page
	    BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
		    conn.getOutputStream()));
	    
	    // POST our data
	    writer.write(formData);
	    writer.flush();
	    writer.close();
	    
	    // connect the reader back to the new connection
	    reader = new BufferedReader(new InputStreamReader(
		    conn.getInputStream()));
	    
	    // read the contents of the new page - which should be what we wanted in the first place!
	    content = null;
	    while ((inputLine = reader.readLine()) != null) {
		content += inputLine + "\n";
	    }
	    reader.close();
	    System.out.println("content: " + content);
	    
	    
	} catch (Exception e) {
	    System.err.println("Error initializing HTML page: " + e);
	}
    }

    // Take a string which contains the source of an HTML file and parse it for
    // the <input> tags that need to be used to perform a POST on a page.
    private String parseFormData(String source) {
	String formData = null;
	while (source.contains("<input")) {
	    // First grep out the <input> tag.
	    int start = source.indexOf("<input");
	    String input = source.substring(start);
	    int end = input.indexOf('>');
	    input = input.substring(0, end + 1);
	    source = source.substring(start + end);
	    // System.out.println("Input: " +input);
	    // Now grep the name= attribute from the input tag
	    start = input.indexOf("name=");
	    String name = input.substring(start + 6);
	    end = name.indexOf("\"");
	    name = name.substring(0, end);
	    if (name.equals("email") || name.equals("password"))
		continue; // special case dealt with outside this code.
	    // System.out.println("Name: " +name);
	    // Now grep the value= attribute from the input tag
	    start = input.indexOf("value=");
	    String value = input.substring(start + 7);
	    end = value.indexOf("\"");
	    value = value.substring(0, end);
	    // System.out.println("Value: " +value);
	    if (formData != null)
		formData += "&";
	    formData += name + "=" + value;
	}
	return formData;
    }

    // Take a string which contains the source of an HTML file and parse it for
    // the action="file" tag.
    private String parseAction(String source) {
	int start = source.indexOf("action=");
	String action = source.substring(start + 8);
	int end = action.indexOf("\"");
	return action.substring(0, end);
    }

}

The guy in the green hat.

Thanlis
Giant
Posts: 240
Joined: Tue Mar 24, 2009 3:34 pm

Re: DDi Compendium API and Java Classes?

Post by Thanlis »

Blakey wrote:I just can't figure this out. This is what I'm now doing:

1. Open the powers.aspx?id-805 page (Divine Challenge).
2. Open a Reader to that URL - this gets me redirected to the login.aspx page of course.
3. Get the Cookies from that URL (note: I only see 1 cookie here, where FF seems to see 2).
4. Read in the content of the HTML page I got redirected to.
5. Scan that source for all the input tags and build my POST data.
6. Scan that source for the action= tag to see where we need to POST to (login.aspx?page=power&id=805)
7. Drop the read connection.
8. Open a write connection to the new URL (login.aspx?page=power&id=805)
9. Add the cookies to the connection.
10. Write the POST data to the connection.
11. Open a reader on the same connection.
12. Read the connection - expecting to see the content of the power, but find I am STILL getting the content of the login.aspx page.
Hm -- OK, I think I maybe see it.

You've opened the login page, parsed it, and gotten all the cookies and form fields. Then you post it. In steps 8-10, you're actually executing the login. So then for step 12, read the connection and snag the cookies /again/. The cookies you get in that step are the session cookies; those are the ones that tell the server you're authenticated. Step 13: send a request, with your new cookies, to the powers.aspx?id-805 page. Read it, and you should have your data.
Reed (halfling sorcerer P3) // Collin (human fighter P2) // Cine (eladrin psion H2)
Sirath (deva shaman H1) // Alesk (dragonborn cleric H3) // Kevin (halfling barbarian H1)

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Thanlis wrote: Hm -- OK, I think I maybe see it.

You've opened the login page, parsed it, and gotten all the cookies and form fields. Then you post it. In steps 8-10, you're actually executing the login. So then for step 12, read the connection and snag the cookies /again/. The cookies you get in that step are the session cookies; those are the ones that tell the server you're authenticated. Step 13: send a request, with your new cookies, to the powers.aspx?id-805 page. Read it, and you should have your data.

Just tried that. Curiously the cookies appear empty once I connect the reader to the connection in step 12.
The guy in the green hat.

User avatar
jstgtpaid
Giant
Posts: 142
Joined: Sun Jun 22, 2008 1:23 am
Location: Tampa, FL

Re: DDi Compendium API and Java Classes?

Post by jstgtpaid »

My experience is with javascript, firefox and greasemonkey. I made an autoplayer for a facebook game. Anyway, I don't know if this would be helpful, but we frequently used a firefox plugin called firebug to see all the data being sent and received. You can see what is posted with that tool.

If posting or whatever got to complicated, I would take the easy route and fill in the buttons and programmatically 'click' on the submit button. It would work very well in the autoplayer type environment, but I must admit I don't know how much that would apply in the java environment you are working in.

In any case, using firebug you may see the post details to give you the information you want. In addition, if you are filling out forms and such you may want to try creating a simple greasemonky javascript fill out the fields and click the button just to ensure that everything is working, plus you can view the details with firebug to see if there is something you may be missing.

I hope that helps in some way. I am cheering for ya! This would be a game changer feature, IMHO.
When the boogeyman goes to bed everynight, he checks his closet for Chuck Norris.

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

maybe it'd help to know what you're trying to get from the website. I don't actually have a DDI account so i don't know really what's there. sounds like you're just trying to get descriptions so you can display them in MT?

User avatar
Hawke
Great Wyrm
Posts: 2261
Joined: Sat Apr 21, 2007 12:12 am
Location: Albuquerque, NM

Re: DDi Compendium API and Java Classes?

Post by Hawke »

The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

Hawke wrote:The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.
Holy cow! I had no idea that this was available. This will make it FAR EASIER.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Hawke
Great Wyrm
Posts: 2261
Joined: Sat Apr 21, 2007 12:12 am
Location: Albuquerque, NM

Re: DDi Compendium API and Java Classes?

Post by Hawke »

Woohoo! I've been hoping someone had the java knowledge to make this a reality =D

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

OMG how did I not see that? -_- at any rate, make sure to post back when you get it working as I'm sure someone else will have the same question

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Hawke wrote:The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.
Yeah, I know about that thread and lots of other similar ones. That stuff is really useful and from that I am already able to get lists of monsters (etc) that match patterns and return some rudimentary data for them. The searching facilities provided by the API are great and very powerful but the info they give back is limited. You get the monster name, ID (which is vital), it's role, level and a few other stats. No real game info though. You need to get to the main HTML page for each monster to get those more detailed stats.

I've knocked up a starting program to allow you to search for monsters from the Compendium. The idea being when you click on a monster and press the Build button it will go off and build you a MapTool token. It currently looks like this: The data is 'real world' data scanned from the Compendium.
MonsterMaker in use.
MonsterMaker in use.
monstermaker.jpg (43.56 KiB) Viewed 2348 times
Trouble that I'm having is taking this info (which includes a ID, so I can easily make a full URL of the page I want) and then actually logging into the site to get HTML data for the monster. If I can crack that hurdle then I can easily scan the HTML and get the actual monster stats out for a given monster.

Unfortunately there is no (published) way to get all a monster's details even if you know it's ID without looking up and scanning the HTML. Part 2 of that blog said he was going to cover that in Part 3 but he never wrote Part 3 (yet). Talking to the author of Masterplan, he is using the same method to get monster data from the Compendium - although he hasn't explained how to get past the authentication hurdle I'm having.

I'm new to Java and to programming HTTP requests, and my knowledge of POST/GET and cookies is limited so I'm fighting a bit of a battle!! :D I'll get there though.


Cheers
Blakey
The guy in the green hat.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

bstrdsmkr wrote:maybe it'd help to know what you're trying to get from the website. I don't actually have a DDI account so i don't know really what's there. sounds like you're just trying to get descriptions so you can display them in MT?
Yeah, exactly that. Or in actual case rather than display them I want to parse the HTML and build a Monster object, with a collection of Power object that it knows. From that base I'm planning on taking Monsters and turning them into MT tokens although I've not had the first thought about how I'm actually going to do that yet. One step at a time.
The guy in the green hat.

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

If you're stronger in another language, you might try writing a quick app in that language until you understand everything that's required to pass authentication, but that's Dependant.

Also, I would print out every variable every time it changes, get a cookie, print it, get html, print it, etc, it'll be a huge amount to comb through but, if you print EVERY variable every time it changes, it should make is easier to spot the purple duck lol

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: DDi Compendium API and Java Classes?

Post by Azhrei »

I'm not familiar with the authentication implementation that DDI uses. So just some overview stuff here..

You'll first need to understand how authentication works in the browser. When you visit a site that uses authentication for a particular realm, that information is requested from the browser by the server. If the browser doesn't have it, it pops up a simple authentication dialog for the username/password for that realm. Now whenever the server requests the username/password for that realm the browser will use what it has cached, not prompting the user again. (It'll use that information for every page requested from the server which is part of that realm. Most servers define the realm as some sub-hierarchy of their domain address space.)

That technique isn't very flexible when it needs to be controlled by the server, however. So another technique is to put a form up as the HTML page and the user types in username/password information. The server creates a session-only cookie out of the random data (and usually factors in IP address and other values so that the same cookie won't/can't be used on other computers). That cookie is then automatically provided by the browser whenever it requests a page whose URL prefix matches the PATH field of the cookie. (Similar to the realm approach, above.)

The second technique is much more common as it (a) doesn't require web server support and (b) gives more control to the application regarding cookie lifetime and such.

To determine what's needed in the second version, you can set up a proxy server that records all data traveling in each direction. There are proxies designed for debugging web page that do exactly this. Now you can see the requests and responses going back and forth. Without the proxy it's a more manual process: get the input fields from the HTML, send a request with that data filled in, check out any cookies created by the server, make further requests and send the cookies.

On Unix systems the wget command can do all of the above for you, even recording the cookies and using them in future requests. It wouldn't integrate well with your application, but it's free software so you could see how they do things if you want.

Post Reply

Return to “Developer Notes”