Page 4 of 5
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 7:16 am
by Blakey
This is hugely frustrating.
I've discovered that I'd missed a cookie - there is a page on the top level wizards.com site that sets a cookie up which I'd not noticed. So I now get that cookie and the one that login.aspx sets. All good. I can get all the <input> tags from login.aspx too.
So I go into the login.aspx page with my 2 cookies set, and I post the 5 parts of data that I need to post (2 hidden fields, email, password and the submit button) and I always get passed back to the login page with no new session cookies given to me.
If I do the same things in Firefox I can trace exactly what pages are accessed and what cookies are set up and what data is POSTed and I really think I'm replicating it exactly but I can't get past the authentication. Argh!
Maybe I need to talk to the guys at VCC and see if they can help?
Blakey
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 9:54 am
by Blakey
Can any of you java experts help me? I've built a single class/app that shows my problem connecting to the Compendium API. When I try and connect it always takes me back to the login form without the two sesison cookies I'm expecting to have enabled.
Code: Select all
package info.rodinia.tokenmaker;
import java.io.*;
import java.net.*;
import java.util.*;
public class Login {
public static void main(String[] args) {
try {
// Set up Cookie Manager
CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(cookieManager);
CookieStore cookieJar = cookieManager.getCookieStore();
// Get the cookie from favicon.ico
URL url = new URL("http://www.wizards.com/favicon.ico");
URLConnection conn = url.openConnection();
Object obj = conn.getContent();
// Connect to login.aspx
url = new URL(
"http://www.wizards.com/dndinsider/compendium/login.aspx");
conn = url.openConnection();
// set up connection params
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
// set up the data we need to post to the login.aspx page
String email = "[email protected]";
String password = "myPassword";
String postData = "email="
+ email
+ "&password="
+ password
+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
String encodedData = URLEncoder.encode(postData, "UTF-16");
// Send POST output.
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
conn.getOutputStream()));
writer.write(encodedData);
writer.flush();
writer.close();
// Get final page.
BufferedReader reader = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String inputLine;
String content = null;
while ((inputLine = reader.readLine()) != null) {
content += inputLine + "\n";
}
reader.close();
System.out.println("Final Source: " + content);
// print our cookies
List<HttpCookie> cookies = cookieJar.getCookies();
for (HttpCookie cookie : cookies) {
System.out.println("Cookie: " + cookie);
}
} catch (Exception e) {
System.err.println("Error Logging In: " + e);
}
}
}
This is what the output of running the code looks like:
Code: Select all
Final Source: null
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head><link type="text/css" href="styles/login.css" media="all" rel="stylesheet" /><title>
</title></head>
<body>
<form name="form1" method="post" action="login.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=" />
<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN" />
<div id="content">
<h1>Save Time with Every Search—Subscribe Now</h1>
<p>
Gain full access to the complete rules text for every race, class, paragon path,
epic destiny, skill, feat, power, item, and ritual—from every D&D rulebook
and online magazine article. The D&DI Compendium offers you the fastest way
to find what you're looking for—and to discover even more.
</p>
<p>
<a target="_blank" href="http://www.wizards.com/default.asp?x=dnd/insider/subscription" title="Click to visit the 'Subscription' page.">
<b>D&D Insider—Subscribe Now!</b>
</a>
</p>
<h1>Already a Subscriber?</h1>
<p/>
<div id="loginForm">
<div>
<fieldset>
<ol>
<li>
Email: <input name="email" type="text" id="email" />
</li>
<li>
Password: <input name="password" type="password" id="password" />
</li>
</ol>
</fieldset>
<p>
<a href="https://accounts.gleemax.com/UserRegistration/DobForm.aspx" title="Join Insider" target="_blank">Join Insider</a> -
<a href="https://accounts.gleemax.com/UserRegistration/PasswordRecoveryForm.aspx" title="Forgot Password?" target="_blank">Forgot Password?</a>
</p>
<p>
<input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" />
</p>
</div>
</div>
</div>
</form>
</body>
</html>
Cookie: BIGipServerWWWPool1=3809478922.20480.0000
Cookie: BIGipServerWWWCOMPPool1=722471178.20480.0000
As you can see, after connecting to login.aspx at the end of the program we have only 2 cookies. I should have 4 now - on a successful connection/login I should get an 'ASP.NET_SessionId' and a 'iPlanetDirectoryPro' cookie. I'm not getting either one.
Can anyone spot what I'm doing wrong?
Cheers
Blakey
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 11:28 am
by Azhrei
Why are you hard-coding the two hidden fields?
They probably contain state information that is no longer valid. You'll likely need to read the values from the initial connection and then include those values when you submit your form data.
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 11:50 am
by Blakey
Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 11:56 am
by jfrazierjr
One possibility is that the pages are looking for a browser string or something else a browser might normally send in an HTTP POST request(such as content type, language, etc). thats just a guess as I have had the same problem you have with my playing around.
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 12:10 pm
by Azhrei
It's possible to fake the agent string using Java code, but I don't recall the technique off-hand. You might be able to specify the header prior to opening the connection and that might be enough...
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 1:02 pm
by bstrdsmkr
Blakey wrote:Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
Hmmm... if they're not changing, probably not session related. I'm willing to bet that Az and Jfrazierjr are on the right track. Those strings are probably generated based on the headers in your connection or possibly some information the server side code has gathered about it. I think you're best bet is to look at the headers on each sent and received connection in FF and mimic them exactly. Sometimes they'll pass back something important in a weird header.
Blakey wrote:Code: Select all
+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
String encodedData = URLEncoder.encode(postData, "UTF-16");
also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 1:40 pm
by Blakey
bstrdsmkr wrote:also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.
If I do that I get that HTTP 500 error. It seems I need to encode the whole string.
I haven't absolutely confirmed that the hidden fields aren't changing - I guess I'll have to do that. I'll also look into how I can set up a complete header in code. Will take me some time to figure out.
I'm amazed this isn't easier....
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 3:57 pm
by bstrdsmkr
ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html
Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 4:44 pm
by Blakey
I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
Re: DDi Compendium API and Java Classes?
Posted: Sun Jun 20, 2010 6:10 pm
by jfrazierjr
Blakey wrote:I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
Ok, so download it from apache.org. Then right click on your project at the top level and select Build Path->configure build path. On the Libraries Tab, click add external Jars. This makes the new jar you downloaded available for use in the project.
Re: DDi Compendium API and Java Classes?
Posted: Mon Jun 21, 2010 4:39 am
by Blakey
Awesome!!!!!
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
I can now get monster HTML data out of the Compendium.
Thanks heaps and heaps and heaps to eveyone who helped me out here - I'm dead happy!!
Blakey, the Java
mostly-newbie.
Re: DDi Compendium API and Java Classes?
Posted: Mon Jun 21, 2010 7:52 am
by Thanlis
Blakey wrote:
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.
Re: DDi Compendium API and Java Classes?
Posted: Mon Jun 21, 2010 7:55 am
by Blakey
Thanlis wrote:Blakey wrote:
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.
Dont get your hopes up! It will be a long time before I have anything worth sharing, if at all! But thanks for the help all the same.
Re: DDi Compendium API and Java Classes?
Posted: Mon Jun 21, 2010 8:50 am
by bstrdsmkr
Awesome, glad you're up and running! =) If you want to post a working code sample as a generic example of how to access secured website info, I'm sure the next person that comes looking would appreciate it =)