DDi Compendium API and Java Classes?
Moderators: dorpond, trevor, Azhrei
Re: DDi Compendium API and Java Classes?
This is hugely frustrating.
I've discovered that I'd missed a cookie - there is a page on the top level wizards.com site that sets a cookie up which I'd not noticed. So I now get that cookie and the one that login.aspx sets. All good. I can get all the <input> tags from login.aspx too.
So I go into the login.aspx page with my 2 cookies set, and I post the 5 parts of data that I need to post (2 hidden fields, email, password and the submit button) and I always get passed back to the login page with no new session cookies given to me.
If I do the same things in Firefox I can trace exactly what pages are accessed and what cookies are set up and what data is POSTed and I really think I'm replicating it exactly but I can't get past the authentication. Argh!
Maybe I need to talk to the guys at VCC and see if they can help?
Blakey
I've discovered that I'd missed a cookie - there is a page on the top level wizards.com site that sets a cookie up which I'd not noticed. So I now get that cookie and the one that login.aspx sets. All good. I can get all the <input> tags from login.aspx too.
So I go into the login.aspx page with my 2 cookies set, and I post the 5 parts of data that I need to post (2 hidden fields, email, password and the submit button) and I always get passed back to the login page with no new session cookies given to me.
If I do the same things in Firefox I can trace exactly what pages are accessed and what cookies are set up and what data is POSTed and I really think I'm replicating it exactly but I can't get past the authentication. Argh!
Maybe I need to talk to the guys at VCC and see if they can help?
Blakey
The guy in the green hat.
Re: DDi Compendium API and Java Classes?
Can any of you java experts help me? I've built a single class/app that shows my problem connecting to the Compendium API. When I try and connect it always takes me back to the login form without the two sesison cookies I'm expecting to have enabled.
This is what the output of running the code looks like:
As you can see, after connecting to login.aspx at the end of the program we have only 2 cookies. I should have 4 now - on a successful connection/login I should get an 'ASP.NET_SessionId' and a 'iPlanetDirectoryPro' cookie. I'm not getting either one.
Can anyone spot what I'm doing wrong?
Cheers
Blakey
Code: Select all
package info.rodinia.tokenmaker;
import java.io.*;
import java.net.*;
import java.util.*;
public class Login {
public static void main(String[] args) {
try {
// Set up Cookie Manager
CookieManager cookieManager = new CookieManager();
cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(cookieManager);
CookieStore cookieJar = cookieManager.getCookieStore();
// Get the cookie from favicon.ico
URL url = new URL("http://www.wizards.com/favicon.ico");
URLConnection conn = url.openConnection();
Object obj = conn.getContent();
// Connect to login.aspx
url = new URL(
"http://www.wizards.com/dndinsider/compendium/login.aspx");
conn = url.openConnection();
// set up connection params
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
// set up the data we need to post to the login.aspx page
String email = "[email protected]";
String password = "myPassword";
String postData = "email="
+ email
+ "&password="
+ password
+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
String encodedData = URLEncoder.encode(postData, "UTF-16");
// Send POST output.
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
conn.getOutputStream()));
writer.write(encodedData);
writer.flush();
writer.close();
// Get final page.
BufferedReader reader = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
String inputLine;
String content = null;
while ((inputLine = reader.readLine()) != null) {
content += inputLine + "\n";
}
reader.close();
System.out.println("Final Source: " + content);
// print our cookies
List<HttpCookie> cookies = cookieJar.getCookies();
for (HttpCookie cookie : cookies) {
System.out.println("Cookie: " + cookie);
}
} catch (Exception e) {
System.err.println("Error Logging In: " + e);
}
}
}
Code: Select all
Final Source: null
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head><link type="text/css" href="styles/login.css" media="all" rel="stylesheet" /><title>
</title></head>
<body>
<form name="form1" method="post" action="login.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=" />
<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN" />
<div id="content">
<h1>Save Time with Every Search—Subscribe Now</h1>
<p>
Gain full access to the complete rules text for every race, class, paragon path,
epic destiny, skill, feat, power, item, and ritual—from every D&D rulebook
and online magazine article. The D&DI Compendium offers you the fastest way
to find what you're looking for—and to discover even more.
</p>
<p>
<a target="_blank" href="http://www.wizards.com/default.asp?x=dnd/insider/subscription" title="Click to visit the 'Subscription' page.">
<b>D&D Insider—Subscribe Now!</b>
</a>
</p>
<h1>Already a Subscriber?</h1>
<p/>
<div id="loginForm">
<div>
<fieldset>
<ol>
<li>
Email: <input name="email" type="text" id="email" />
</li>
<li>
Password: <input name="password" type="password" id="password" />
</li>
</ol>
</fieldset>
<p>
<a href="https://accounts.gleemax.com/UserRegistration/DobForm.aspx" title="Join Insider" target="_blank">Join Insider</a> -
<a href="https://accounts.gleemax.com/UserRegistration/PasswordRecoveryForm.aspx" title="Forgot Password?" target="_blank">Forgot Password?</a>
</p>
<p>
<input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" />
</p>
</div>
</div>
</div>
</form>
</body>
</html>
Cookie: BIGipServerWWWPool1=3809478922.20480.0000
Cookie: BIGipServerWWWCOMPPool1=722471178.20480.0000
Can anyone spot what I'm doing wrong?
Cheers
Blakey
The guy in the green hat.
Re: DDi Compendium API and Java Classes?
Why are you hard-coding the two hidden fields?
They probably contain state information that is no longer valid. You'll likely need to read the values from the initial connection and then include those values when you submit your form data.
They probably contain state information that is no longer valid. You'll likely need to read the values from the initial connection and then include those values when you submit your form data.
Re: DDi Compendium API and Java Classes?
Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
The guy in the green hat.
- jfrazierjr
- Deity
- Posts: 5176
- Joined: Tue Sep 11, 2007 7:31 pm
Re: DDi Compendium API and Java Classes?
One possibility is that the pages are looking for a browser string or something else a browser might normally send in an HTTP POST request(such as content type, language, etc). thats just a guess as I have had the same problem you have with my playing around.
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Re: DDi Compendium API and Java Classes?
It's possible to fake the agent string using Java code, but I don't recall the technique off-hand. You might be able to specify the header prior to opening the connection and that might be enough...
Re: DDi Compendium API and Java Classes?
Hmmm... if they're not changing, probably not session related. I'm willing to bet that Az and Jfrazierjr are on the right track. Those strings are probably generated based on the headers in your connection or possibly some information the server side code has gathered about it. I think you're best bet is to look at the headers on each sent and received connection in FF and mimic them exactly. Sometimes they'll pass back something important in a weird header.Blakey wrote:Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.
also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.Blakey wrote:Code: Select all
+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In"; String encodedData = URLEncoder.encode(postData, "UTF-16");
Re: DDi Compendium API and Java Classes?
If I do that I get that HTTP 500 error. It seems I need to encode the whole string.bstrdsmkr wrote:also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.
I haven't absolutely confirmed that the hidden fields aren't changing - I guess I'll have to do that. I'll also look into how I can set up a complete header in code. Will take me some time to figure out.
I'm amazed this isn't easier....
The guy in the green hat.
Re: DDi Compendium API and Java Classes?
ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html
Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html
Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?
Re: DDi Compendium API and Java Classes?
I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
The guy in the green hat.
- jfrazierjr
- Deity
- Posts: 5176
- Joined: Tue Sep 11, 2007 7:31 pm
Re: DDi Compendium API and Java Classes?
Ok, so download it from apache.org. Then right click on your project at the top level and select Build Path->configure build path. On the Libraries Tab, click add external Jars. This makes the new jar you downloaded available for use in the project.Blakey wrote:I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..
Re: DDi Compendium API and Java Classes?
bstrdsmkr wrote:ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-cli ... mples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/0 ... based.html
Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?
Awesome!!!!!
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
I can now get monster HTML data out of the Compendium.
Thanks heaps and heaps and heaps to eveyone who helped me out here - I'm dead happy!!
Blakey, the Java mostly-newbie.
The guy in the green hat.
Re: DDi Compendium API and Java Classes?
*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.Blakey wrote: i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
Re: DDi Compendium API and Java Classes?
Dont get your hopes up! It will be a long time before I have anything worth sharing, if at all! But thanks for the help all the same.Thanlis wrote:*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.Blakey wrote: i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!!
The guy in the green hat.
Re: DDi Compendium API and Java Classes?
Awesome, glad you're up and running! =) If you want to post a working code sample as a generic example of how to access secured website info, I'm sure the next person that comes looking would appreciate it =)