RPTools.net

Discussion and Support

Skip to content

It is currently Sat Aug 19, 2017 3:36 pm 






Reply to topic  [ 64 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

Previous topic | Next topic 

  Print view

Author Message
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Thu Jun 17, 2010 4:17 pm 
If you refresh the page to the __VIEWSTATE and/or __EVENTVALIDATION values change? most likely it's sending you a random string that you need to pass back to authenticate yourself. If that's the case if you do nothing but refresh the page, that should be the only thing that changes, then you'll have your culprit. You may need to open two windows side by side and only refresh one so you can compare the two.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 3:45 am 
Still hacking...

A question: on a POST, does the order of the fields matter?

Cheers

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 5:00 am 
I just can't figure this out. This is what I'm now doing:

1. Open the powers.aspx?id-805 page (Divine Challenge).
2. Open a Reader to that URL - this gets me redirected to the login.aspx page of course.
3. Get the Cookies from that URL (note: I only see 1 cookie here, where FF seems to see 2).
4. Read in the content of the HTML page I got redirected to.
5. Scan that source for all the input tags and build my POST data.
6. Scan that source for the action= tag to see where we need to POST to (login.aspx?page=power&id=805)
7. Drop the read connection.
8. Open a write connection to the new URL (login.aspx?page=power&id=805)
9. Add the cookies to the connection.
10. Write the POST data to the connection.
11. Open a reader on the same connection.
12. Read the connection - expecting to see the content of the power, but find I am STILL getting the content of the login.aspx page.

Arggh!!!!

Questions arrising:

1. Is this the right set of steps?
2. I think the cookie situation is weird - Java is reporting less cookies than FF. Am I doing something wrong here?
3. Should I give up?!?!?


For those that care, here is the complete Java Class: Note that the init() method does all the work really.

Code:
package info.rodinia.tokenmaker;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLEncoder;

// a simple class which models a compendium entry
public class CompendiumEntry {

    private String myHTMLSource;
    private String myURL;
    private boolean loggedIn = false;
    private static final String email = "email=myemail@hotmail.com";
    private static final String password = "password=myPasswd";
    private static final String baseURL = "http://www.wizards.com/dndinsider/compendium/";

    // constructor takes a URL and builds an entry from it.
    public CompendiumEntry(String stringUrl) {
   myURL = stringUrl;
   init();
    }

    // give back the source
    public String getHTML() {
   return myHTMLSource;
    }

    private void init() {
   try {
       // Open a connection to the page
       URL url = new URL(myURL);
       URLConnection conn = url.openConnection();

       // Open a reader to the page - this will actually be the redirected
       // login page
       BufferedReader reader = new BufferedReader(new InputStreamReader(
          conn.getInputStream()));

       // Get the Cookies from the Page
       String cookies = null;
       String headerName = null;
       for (int i = 1; (headerName = conn.getHeaderFieldKey(i)) != null; i++) {
      if (headerName.equals("Set-Cookie")) {
          cookies += conn.getHeaderField(i);
      }
       }
       System.out.println("Cookies List: " + cookies);

       // read the contents of the page to get the formData we need to push back
       String content = null;
       String inputLine;
       while ((inputLine = reader.readLine()) != null) {
      content += inputLine + "\n";
       }
       reader.close();
       System.out.println("content: " + content);

       // parse the page for <input> and "action=" tags ready for our POST phase
       String formData = parseFormData(content);
       formData += email + "&" + password;
       formData = URLEncoder.encode(formData, "UTF-16");
       String actionUrl = parseAction(content);
       actionUrl = baseURL+actionUrl;
       System.out.println("formData: " + formData);
       System.out.println("actionUrl: " + actionUrl);

       // now drop the connection to the old page and connect up to the new one, POSTing our data and setting out cookies.
       url = new URL(actionUrl);
       conn = url.openConnection();
      
       // Add the required cookies to the writer connection
       conn.setRequestProperty("Cookie", cookies);
       conn.setDoOutput(true);   // let us write
      
       // Open a writer to the redirected page
       BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
          conn.getOutputStream()));
      
       // POST our data
       writer.write(formData);
       writer.flush();
       writer.close();
      
       // connect the reader back to the new connection
       reader = new BufferedReader(new InputStreamReader(
          conn.getInputStream()));
      
       // read the contents of the new page - which should be what we wanted in the first place!
       content = null;
       while ((inputLine = reader.readLine()) != null) {
      content += inputLine + "\n";
       }
       reader.close();
       System.out.println("content: " + content);
      
      
   } catch (Exception e) {
       System.err.println("Error initializing HTML page: " + e);
   }
    }

    // Take a string which contains the source of an HTML file and parse it for
    // the <input> tags that need to be used to perform a POST on a page.
    private String parseFormData(String source) {
   String formData = null;
   while (source.contains("<input")) {
       // First grep out the <input> tag.
       int start = source.indexOf("<input");
       String input = source.substring(start);
       int end = input.indexOf('>');
       input = input.substring(0, end + 1);
       source = source.substring(start + end);
       // System.out.println("Input: " +input);
       // Now grep the name= attribute from the input tag
       start = input.indexOf("name=");
       String name = input.substring(start + 6);
       end = name.indexOf("\"");
       name = name.substring(0, end);
       if (name.equals("email") || name.equals("password"))
      continue; // special case dealt with outside this code.
       // System.out.println("Name: " +name);
       // Now grep the value= attribute from the input tag
       start = input.indexOf("value=");
       String value = input.substring(start + 7);
       end = value.indexOf("\"");
       value = value.substring(0, end);
       // System.out.println("Value: " +value);
       if (formData != null)
      formData += "&";
       formData += name + "=" + value;
   }
   return formData;
    }

    // Take a string which contains the source of an HTML file and parse it for
    // the action="file" tag.
    private String parseAction(String source) {
   int start = source.indexOf("action=");
   String action = source.substring(start + 8);
   int end = action.indexOf("\"");
   return action.substring(0, end);
    }

}


_________________
The guy in the green hat.


Top
 Profile  
 
 Offline
Dragon
 
Joined: Tue Mar 24, 2009 2:34 pm
Posts: 859
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 6:54 am 
Blakey wrote:
I just can't figure this out. This is what I'm now doing:

1. Open the powers.aspx?id-805 page (Divine Challenge).
2. Open a Reader to that URL - this gets me redirected to the login.aspx page of course.
3. Get the Cookies from that URL (note: I only see 1 cookie here, where FF seems to see 2).
4. Read in the content of the HTML page I got redirected to.
5. Scan that source for all the input tags and build my POST data.
6. Scan that source for the action= tag to see where we need to POST to (login.aspx?page=power&id=805)
7. Drop the read connection.
8. Open a write connection to the new URL (login.aspx?page=power&id=805)
9. Add the cookies to the connection.
10. Write the POST data to the connection.
11. Open a reader on the same connection.
12. Read the connection - expecting to see the content of the power, but find I am STILL getting the content of the login.aspx page.


Hm -- OK, I think I maybe see it.

You've opened the login page, parsed it, and gotten all the cookies and form fields. Then you post it. In steps 8-10, you're actually executing the login. So then for step 12, read the connection and snag the cookies /again/. The cookies you get in that step are the session cookies; those are the ones that tell the server you're authenticated. Step 13: send a request, with your new cookies, to the powers.aspx?id-805 page. Read it, and you should have your data.

_________________
Reed (halfling sorcerer P3) // Collin (human fighter P2) // Cine (eladrin psion H2)
Sirath (deva shaman H1) // Alesk (dragonborn cleric H3) // Kevin (halfling barbarian H1)


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 7:24 am 
Thanlis wrote:
Hm -- OK, I think I maybe see it.

You've opened the login page, parsed it, and gotten all the cookies and form fields. Then you post it. In steps 8-10, you're actually executing the login. So then for step 12, read the connection and snag the cookies /again/. The cookies you get in that step are the session cookies; those are the ones that tell the server you're authenticated. Step 13: send a request, with your new cookies, to the powers.aspx?id-805 page. Read it, and you should have your data.



Just tried that. Curiously the cookies appear empty once I connect the reader to the connection in step 12.

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Giant
 
Joined: Sun Jun 22, 2008 12:23 am
Posts: 231
Location: Tampa, FL
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 9:45 am 
My experience is with javascript, firefox and greasemonkey. I made an autoplayer for a facebook game. Anyway, I don't know if this would be helpful, but we frequently used a firefox plugin called firebug to see all the data being sent and received. You can see what is posted with that tool.

If posting or whatever got to complicated, I would take the easy route and fill in the buttons and programmatically 'click' on the submit button. It would work very well in the autoplayer type environment, but I must admit I don't know how much that would apply in the java environment you are working in.

In any case, using firebug you may see the post details to give you the information you want. In addition, if you are filling out forms and such you may want to try creating a simple greasemonky javascript fill out the fields and click the button just to ensure that everything is working, plus you can view the details with firebug to see if there is something you may be missing.

I hope that helps in some way. I am cheering for ya! This would be a game changer feature, IMHO.

_________________
When the boogeyman goes to bed everynight, he checks his closet for Chuck Norris.


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 12:07 pm 
maybe it'd help to know what you're trying to get from the website. I don't actually have a DDI account so i don't know really what's there. sounds like you're just trying to get descriptions so you can display them in MT?


Top
 Profile  
 
User avatar  Offline
Great Wyrm
 
Joined: Fri Apr 20, 2007 11:12 pm
Posts: 2317
Location: Albuquerque, NM
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 3:56 pm 
The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.


Top
 Profile  
 
User avatar  Offline
Deity
 
Joined: Tue Sep 11, 2007 6:31 pm
Posts: 5410
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 4:04 pm 
Hawke wrote:
The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.


Holy cow! I had no idea that this was available. This will make it FAR EASIER.

_________________
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..


Top
 Profile  
 
User avatar  Offline
Great Wyrm
 
Joined: Fri Apr 20, 2007 11:12 pm
Posts: 2317
Location: Albuquerque, NM
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 4:18 pm 
Woohoo! I've been hoping someone had the java knowledge to make this a reality =D


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Fri Jun 18, 2010 11:14 pm 
OMG how did I not see that? -_- at any rate, make sure to post back when you get it working as I'm sure someone else will have the same question


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sat Jun 19, 2010 1:16 am 
Hawke wrote:
The Compendium API might provide some help here...

http://community.wizards.com/wotc_frazi ... pi__part_2

I think Iv'e seen a few threads on EN that explain some of the latest Compendium API updates and how some of the older info out there doesn't work quite as much now.


Yeah, I know about that thread and lots of other similar ones. That stuff is really useful and from that I am already able to get lists of monsters (etc) that match patterns and return some rudimentary data for them. The searching facilities provided by the API are great and very powerful but the info they give back is limited. You get the monster name, ID (which is vital), it's role, level and a few other stats. No real game info though. You need to get to the main HTML page for each monster to get those more detailed stats.

I've knocked up a starting program to allow you to search for monsters from the Compendium. The idea being when you click on a monster and press the Build button it will go off and build you a MapTool token. It currently looks like this: The data is 'real world' data scanned from the Compendium.

Attachment:
File comment: MonsterMaker in use.
monstermaker.jpg
monstermaker.jpg [ 43.56 KiB | Viewed 1530 times ]


Trouble that I'm having is taking this info (which includes a ID, so I can easily make a full URL of the page I want) and then actually logging into the site to get HTML data for the monster. If I can crack that hurdle then I can easily scan the HTML and get the actual monster stats out for a given monster.

Unfortunately there is no (published) way to get all a monster's details even if you know it's ID without looking up and scanning the HTML. Part 2 of that blog said he was going to cover that in Part 3 but he never wrote Part 3 (yet). Talking to the author of Masterplan, he is using the same method to get monster data from the Compendium - although he hasn't explained how to get past the authentication hurdle I'm having.

I'm new to Java and to programming HTTP requests, and my knowledge of POST/GET and cookies is limited so I'm fighting a bit of a battle!! :D I'll get there though.


Cheers
Blakey

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sat Jun 19, 2010 1:20 am 
bstrdsmkr wrote:
maybe it'd help to know what you're trying to get from the website. I don't actually have a DDI account so i don't know really what's there. sounds like you're just trying to get descriptions so you can display them in MT?


Yeah, exactly that. Or in actual case rather than display them I want to parse the HTML and build a Monster object, with a collection of Power object that it knows. From that base I'm planning on taking Monsters and turning them into MT tokens although I've not had the first thought about how I'm actually going to do that yet. One step at a time.

_________________
The guy in the green hat.


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sat Jun 19, 2010 1:33 am 
If you're stronger in another language, you might try writing a quick app in that language until you understand everything that's required to pass authentication, but that's Dependant.

Also, I would print out every variable every time it changes, get a cookie, print it, get html, print it, etc, it'll be a huge amount to comb through but, if you print EVERY variable every time it changes, it should make is easier to spot the purple duck lol


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12078
Location: Tampa, FL
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sat Jun 19, 2010 11:58 am 
I'm not familiar with the authentication implementation that DDI uses. So just some overview stuff here..

You'll first need to understand how authentication works in the browser. When you visit a site that uses authentication for a particular realm, that information is requested from the browser by the server. If the browser doesn't have it, it pops up a simple authentication dialog for the username/password for that realm. Now whenever the server requests the username/password for that realm the browser will use what it has cached, not prompting the user again. (It'll use that information for every page requested from the server which is part of that realm. Most servers define the realm as some sub-hierarchy of their domain address space.)

That technique isn't very flexible when it needs to be controlled by the server, however. So another technique is to put a form up as the HTML page and the user types in username/password information. The server creates a session-only cookie out of the random data (and usually factors in IP address and other values so that the same cookie won't/can't be used on other computers). That cookie is then automatically provided by the browser whenever it requests a page whose URL prefix matches the PATH field of the cookie. (Similar to the realm approach, above.)

The second technique is much more common as it (a) doesn't require web server support and (b) gives more control to the application regarding cookie lifetime and such.

To determine what's needed in the second version, you can set up a proxy server that records all data traveling in each direction. There are proxies designed for debugging web page that do exactly this. Now you can see the requests and responses going back and forth. Without the proxy it's a more manual process: get the input fields from the HTML, send a request with that data filled in, check out any cookies created by the server, make further requests and send the cookies.

On Unix systems the wget command can do all of the above for you, even recording the cookies and using them in future requests. It wouldn't integrate well with your application, but it's free software so you could see how they do things if you want.


Top
 Profile  
 
Display posts from previous:  Sort by  
Reply to topic  [ 64 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 243 on Sun Nov 04, 2012 6:14 am

Users browsing this forum: No registered users and 1 guest





Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group

Style based on Andreas08 by Andreas Viklund

Style by Elizabeth Shulman