DDi Compendium API and Java Classes?

Progress reports and musings from the developers on the current gaming tools.

Moderators: dorpond, trevor, Azhrei

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

This is hugely frustrating.

I've discovered that I'd missed a cookie - there is a page on the top level wizards.com site that sets a cookie up which I'd not noticed. So I now get that cookie and the one that login.aspx sets. All good. I can get all the <input> tags from login.aspx too.

So I go into the login.aspx page with my 2 cookies set, and I post the 5 parts of data that I need to post (2 hidden fields, email, password and the submit button) and I always get passed back to the login page with no new session cookies given to me.

If I do the same things in Firefox I can trace exactly what pages are accessed and what cookies are set up and what data is POSTed and I really think I'm replicating it exactly but I can't get past the authentication. Argh!

Maybe I need to talk to the guys at VCC and see if they can help?

Blakey
The guy in the green hat.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Can any of you java experts help me? I've built a single class/app that shows my problem connecting to the Compendium API. When I try and connect it always takes me back to the login form without the two sesison cookies I'm expecting to have enabled.

Code: Select all

package info.rodinia.tokenmaker;

import java.io.*;
import java.net.*;
import java.util.*;

public class Login {

    public static void main(String[] args) {
	try {
	    // Set up Cookie Manager
	    CookieManager cookieManager = new CookieManager();
	    cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
	    CookieHandler.setDefault(cookieManager);
	    CookieStore cookieJar = cookieManager.getCookieStore();

	    // Get the cookie from favicon.ico
	    URL url = new URL("http://www.wizards.com/favicon.ico");
	    URLConnection conn = url.openConnection();
	    Object obj = conn.getContent();

	    // Connect to login.aspx
	    url = new URL(
		    "http://www.wizards.com/dndinsider/compendium/login.aspx");
	    conn = url.openConnection();

	    // set up connection params
	    conn.setDoOutput(true);
	    conn.setUseCaches(false);
	    conn.setRequestProperty("Content-Type",
		    "application/x-www-form-urlencoded");
	    // set up the data we need to post to the login.aspx page
	    String email = "[email protected]";
	    String password = "myPassword";
	    String postData = "email="
		    + email
		    + "&password="
		    + password
		    + "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
	    String encodedData = URLEncoder.encode(postData, "UTF-16");

	    // Send POST output.
	    BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
		    conn.getOutputStream()));
	    writer.write(encodedData);
	    writer.flush();
	    writer.close();

	    // Get final page.
	    BufferedReader reader = new BufferedReader(new InputStreamReader(
		    conn.getInputStream()));
	    String inputLine;
	    String content = null;
	    while ((inputLine = reader.readLine()) != null) {
		content += inputLine + "\n";
	    }
	    reader.close();
	    System.out.println("Final Source: " + content);

	    // print our cookies
	    List<HttpCookie> cookies = cookieJar.getCookies();
	    for (HttpCookie cookie : cookies) {
		System.out.println("Cookie: " + cookie);
	    }

	} catch (Exception e) {
	    System.err.println("Error Logging In: " + e);
	}
    }

}
This is what the output of running the code looks like:

Code: Select all

Final Source: null
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head><link type="text/css" href="styles/login.css" media="all" rel="stylesheet" /><title>

</title></head>
<body>
    <form name="form1" method="post" action="login.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=" />

<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN" />
		    <div id="content">
		        <h1>Save Time with Every Search&#151;Subscribe Now</h1>
		        <p>
		            Gain full access to the complete rules text for every race, class, paragon path, 
		            epic destiny, skill, feat, power, item, and ritual&#151;from every D&D rulebook 
		            and online magazine article. The D&DI Compendium offers you the fastest way 
		            to find what you're looking for&#151;and to discover even more.
		        </p>
		        <p>
		            <a target="_blank" href="http://www.wizards.com/default.asp?x=dnd/insider/subscription" title="Click to visit the 'Subscription' page.">
		                <b>D&D Insider&#151;Subscribe Now!</b>
		            </a>
		        </p>
		        <h1>Already a Subscriber?</h1>
		        <p/>
				<div id="loginForm">
	
					<div>
						<fieldset>
							<ol>
								<li>
									Email:  <input name="email" type="text" id="email" />
								</li>
								<li>
									Password:  <input name="password" type="password" id="password" />
								</li>
							</ol>
						</fieldset>
						<p>
							<a href="https://accounts.gleemax.com/UserRegistration/DobForm.aspx" title="Join Insider" target="_blank">Join Insider</a> - 
							<a href="https://accounts.gleemax.com/UserRegistration/PasswordRecoveryForm.aspx" title="Forgot Password?" target="_blank">Forgot Password?</a>
						</p>
						<p>
							<input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" />
						</p>
					</div>
				
</div>
				
		    </div>
    </form>
</body>
</html>

Cookie: BIGipServerWWWPool1=3809478922.20480.0000
Cookie: BIGipServerWWWCOMPPool1=722471178.20480.0000

As you can see, after connecting to login.aspx at the end of the program we have only 2 cookies. I should have 4 now - on a successful connection/login I should get an 'ASP.NET_SessionId' and a 'iPlanetDirectoryPro' cookie. I'm not getting either one.

Can anyone spot what I'm doing wrong?

Cheers
Blakey
The guy in the green hat.

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: DDi Compendium API and Java Classes?

Post by Azhrei »

Why are you hard-coding the two hidden fields?

They probably contain state information that is no longer valid. You'll likely need to read the values from the initial connection and then include those values when you submit your form data.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
The guy in the green hat.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

One possibility is that the pages are looking for a browser string or something else a browser might normally send in an HTTP POST request(such as content type, language, etc). thats just a guess as I have had the same problem you have with my playing around.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Azhrei
Site Admin
Posts: 12086
Joined: Mon Jun 12, 2006 1:20 pm
Location: Tampa, FL

Re: DDi Compendium API and Java Classes?

Post by Azhrei »

It's possible to fake the agent string using Java code, but I don't recall the technique off-hand. You might be able to specify the header prior to opening the connection and that might be enough...

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

Blakey wrote:Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
Hmmm... if they're not changing, probably not session related. I'm willing to bet that Az and Jfrazierjr are on the right track. Those strings are probably generated based on the headers in your connection or possibly some information the server side code has gathered about it. I think you're best bet is to look at the headers on each sent and received connection in FF and mimic them exactly. Sometimes they'll pass back something important in a weird header.
Blakey wrote:

Code: Select all

+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
       String encodedData = URLEncoder.encode(postData, "UTF-16");
also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

bstrdsmkr wrote:also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.
If I do that I get that HTTP 500 error. It seems I need to encode the whole string.

I haven't absolutely confirmed that the hidden fields aren't changing - I guess I'll have to do that. I'll also look into how I can set up a complete header in code. Will take me some time to figure out.

I'm amazed this isn't easier....
The guy in the green hat.

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html

Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
The guy in the green hat.

User avatar
jfrazierjr
Deity
Posts: 5176
Joined: Tue Sep 11, 2007 7:31 pm

Re: DDi Compendium API and Java Classes?

Post by jfrazierjr »

Blakey wrote:I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
Ok, so download it from apache.org. Then right click on your project at the top level and select Build Path->configure build path. On the Libraries Tab, click add external Jars. This makes the new jar you downloaded available for use in the project.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

bstrdsmkr wrote:ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html

Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?

Awesome!!!!! :D

i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D

I can now get monster HTML data out of the Compendium.

Thanks heaps and heaps and heaps to eveyone who helped me out here - I'm dead happy!! :D

Blakey, the Java mostly-newbie.
The guy in the green hat.

Thanlis
Giant
Posts: 240
Joined: Tue Mar 24, 2009 3:34 pm

Re: DDi Compendium API and Java Classes?

Post by Thanlis »

Blakey wrote: i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D
*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.
Reed (halfling sorcerer P3) // Collin (human fighter P2) // Cine (eladrin psion H2)
Sirath (deva shaman H1) // Alesk (dragonborn cleric H3) // Kevin (halfling barbarian H1)

User avatar
Blakey
Dragon
Posts: 778
Joined: Fri Mar 23, 2007 11:27 am
Location: Sussex, UK.

Re: DDi Compendium API and Java Classes?

Post by Blakey »

Thanlis wrote:
Blakey wrote: i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D
*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.
Dont get your hopes up! It will be a long time before I have anything worth sharing, if at all! But thanks for the help all the same. :D
The guy in the green hat.

bstrdsmkr
Cave Troll
Posts: 41
Joined: Sat Jun 05, 2010 8:58 pm

Re: DDi Compendium API and Java Classes?

Post by bstrdsmkr »

Awesome, glad you're up and running! =) If you want to post a working code sample as a generic example of how to access secured website info, I'm sure the next person that comes looking would appreciate it =)

Post Reply

Return to “Developer Notes”