RPTools.net

Discussion and Support

Skip to content

It is currently Mon Oct 23, 2017 3:09 am 






Reply to topic  [ 64 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

Previous topic | Next topic 

  Print view

Author Message
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 6:16 am 
This is hugely frustrating.

I've discovered that I'd missed a cookie - there is a page on the top level wizards.com site that sets a cookie up which I'd not noticed. So I now get that cookie and the one that login.aspx sets. All good. I can get all the <input> tags from login.aspx too.

So I go into the login.aspx page with my 2 cookies set, and I post the 5 parts of data that I need to post (2 hidden fields, email, password and the submit button) and I always get passed back to the login page with no new session cookies given to me.

If I do the same things in Firefox I can trace exactly what pages are accessed and what cookies are set up and what data is POSTed and I really think I'm replicating it exactly but I can't get past the authentication. Argh!

Maybe I need to talk to the guys at VCC and see if they can help?

Blakey

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 8:54 am 
Can any of you java experts help me? I've built a single class/app that shows my problem connecting to the Compendium API. When I try and connect it always takes me back to the login form without the two sesison cookies I'm expecting to have enabled.

Code:
package info.rodinia.tokenmaker;

import java.io.*;
import java.net.*;
import java.util.*;

public class Login {

    public static void main(String[] args) {
   try {
       // Set up Cookie Manager
       CookieManager cookieManager = new CookieManager();
       cookieManager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
       CookieHandler.setDefault(cookieManager);
       CookieStore cookieJar = cookieManager.getCookieStore();

       // Get the cookie from favicon.ico
       URL url = new URL("http://www.wizards.com/favicon.ico");
       URLConnection conn = url.openConnection();
       Object obj = conn.getContent();

       // Connect to login.aspx
       url = new URL(
          "http://www.wizards.com/dndinsider/compendium/login.aspx");
       conn = url.openConnection();

       // set up connection params
       conn.setDoOutput(true);
       conn.setUseCaches(false);
       conn.setRequestProperty("Content-Type",
          "application/x-www-form-urlencoded");
       // set up the data we need to post to the login.aspx page
       String email = "myemail@hotmail.com";
       String password = "myPassword";
       String postData = "email="
          + email
          + "&password="
          + password
          + "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
       String encodedData = URLEncoder.encode(postData, "UTF-16");

       // Send POST output.
       BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
          conn.getOutputStream()));
       writer.write(encodedData);
       writer.flush();
       writer.close();

       // Get final page.
       BufferedReader reader = new BufferedReader(new InputStreamReader(
          conn.getInputStream()));
       String inputLine;
       String content = null;
       while ((inputLine = reader.readLine()) != null) {
      content += inputLine + "\n";
       }
       reader.close();
       System.out.println("Final Source: " + content);

       // print our cookies
       List<HttpCookie> cookies = cookieJar.getCookies();
       for (HttpCookie cookie : cookies) {
      System.out.println("Cookie: " + cookie);
       }

   } catch (Exception e) {
       System.err.println("Error Logging In: " + e);
   }
    }

}


This is what the output of running the code looks like:

Code:
Final Source: null
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head><link type="text/css" href="styles/login.css" media="all" rel="stylesheet" /><title>

</title></head>
<body>
    <form name="form1" method="post" action="login.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=" />

<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN" />
          <div id="content">
              <h1>Save Time with Every Search&#151;Subscribe Now</h1>
              <p>
                  Gain full access to the complete rules text for every race, class, paragon path,
                  epic destiny, skill, feat, power, item, and ritual&#151;from every D&amp;D rulebook
                  and online magazine article. The D&amp;DI Compendium offers you the fastest way
                  to find what you're looking for&#151;and to discover even more.
              </p>
              <p>
                  <a target="_blank" href="http://www.wizards.com/default.asp?x=dnd/insider/subscription" title="Click to visit the 'Subscription' page.">
                      <b>D&amp;D Insider&#151;Subscribe Now!</b>
                  </a>
              </p>
              <h1>Already a Subscriber?</h1>
              <p/>
            <div id="loginForm">
   
               <div>
                  <fieldset>
                     <ol>
                        <li>
                           Email:&nbsp;&nbsp;<input name="email" type="text" id="email" />
                        </li>
                        <li>
                           Password:&nbsp;&nbsp;<input name="password" type="password" id="password" />
                        </li>
                     </ol>
                  </fieldset>
                  <p>
                     <a href="https://accounts.gleemax.com/UserRegistration/DobForm.aspx" title="Join Insider" target="_blank">Join Insider</a> -
                     <a href="https://accounts.gleemax.com/UserRegistration/PasswordRecoveryForm.aspx" title="Forgot Password?" target="_blank">Forgot Password?</a>
                  </p>
                  <p>
                     <input type="submit" name="InsiderSignin" value="Sign In" id="InsiderSignin" />
                  </p>
               </div>
            
</div>
            
          </div>
    </form>
</body>
</html>

Cookie: BIGipServerWWWPool1=3809478922.20480.0000
Cookie: BIGipServerWWWCOMPPool1=722471178.20480.0000



As you can see, after connecting to login.aspx at the end of the program we have only 2 cookies. I should have 4 now - on a successful connection/login I should get an 'ASP.NET_SessionId' and a 'iPlanetDirectoryPro' cookie. I'm not getting either one.

Can anyone spot what I'm doing wrong?

Cheers
Blakey

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 10:28 am 
Why are you hard-coding the two hidden fields?

They probably contain state information that is no longer valid. You'll likely need to read the values from the initial connection and then include those values when you submit your form data.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 10:50 am 
Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Deity
 
Joined: Tue Sep 11, 2007 6:31 pm
Posts: 5410
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 10:56 am 
One possibility is that the pages are looking for a browser string or something else a browser might normally send in an HTTP POST request(such as content type, language, etc). thats just a guess as I have had the same problem you have with my playing around.

_________________
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..


Top
 Profile  
 
User avatar  Offline
Site Admin
 
Joined: Mon Jun 12, 2006 12:20 pm
Posts: 12101
Location: Tampa, FL
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 11:10 am 
It's possible to fake the agent string using Java code, but I don't recall the technique off-hand. You might be able to specify the header prior to opening the connection and that might be enough...


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 12:02 pm 
Blakey wrote:
Yeah, I do that. I read the two hidden files from the login form before I post them. For what it's worth (as far as I ca see) they don't ever seem to change so the code above should still work.

Hmmm... if they're not changing, probably not session related. I'm willing to bet that Az and Jfrazierjr are on the right track. Those strings are probably generated based on the headers in your connection or possibly some information the server side code has gathered about it. I think you're best bet is to look at the headers on each sent and received connection in FF and mimic them exactly. Sometimes they'll pass back something important in a weird header.
Blakey wrote:
Code:
+ "&__VIEWSTATE=/wEPDwUKLTMxMzExNzE1NGRk5yeNnWQGSxW08LBMM/goQ8zo5Es=&__EVENTVALIDATION=/wEWBAL25N6oDwKyzcaDDQLyveCRDwK4+vrXBUx4g9s9PCt9gK77St3QWFU1ZunN&InsiderSignin=Sign In";
       String encodedData = URLEncoder.encode(postData, "UTF-16");

also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 12:40 pm 
bstrdsmkr wrote:
also, try not encoding this string and hard code a %2f for the space in "sign in" everything looks url safe there, but if your PW has not-url-safe characters, encode it on the same line you add it to the string.


If I do that I get that HTTP 500 error. It seems I need to encode the whole string.

I haven't absolutely confirmed that the hidden fields aren't changing - I guess I'll have to do that. I'll also look into how I can set up a complete header in code. Will take me some time to figure out.

I'm amazed this isn't easier....

_________________
The guy in the green hat.


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 2:57 pm 
ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-client-4.0.1/examples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/03/using-httpclient-to-make-form-based.html

Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 3:44 pm 
I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...

_________________
The guy in the green hat.


Top
 Profile  
 
User avatar  Offline
Deity
 
Joined: Tue Sep 11, 2007 6:31 pm
Posts: 5410
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Sun Jun 20, 2010 5:10 pm 
Blakey wrote:
I don't seem to have Apache installed by default - at least my Eclipse doesn't know about HttpClient class. Still looking...


Ok, so download it from apache.org. Then right click on your project at the top level and select Build Path->configure build path. On the Libraries Tab, click add external Jars. This makes the new jar you downloaded available for use in the project.

_________________
I save all my Campaign Files to DropBox. Not only can I access a campaign file from pretty much any OS that will run Maptool(Win,OSX, linux), but each file is versioned, so if something goes crazy wild, I can always roll back to a previous version of the same file.

Get your Dropbox 2GB via my referral link, and as a bonus, I get an extra 250 MB of space. Even if you don't don't use my link, I still enthusiastically recommend Dropbox..


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Mon Jun 21, 2010 3:39 am 
bstrdsmkr wrote:
ok, so i took a look around and found this:
http://hc.apache.org/httpcomponents-client-4.0.1/examples.html
but more specifically this:
http://anilsaldhana.blogspot.com/2007/03/using-httpclient-to-make-form-based.html

Maybe that lib will make it easier? everything in your code looks right, but maybe it's something simple we're all missing?



Awesome!!!!! :D

i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D

I can now get monster HTML data out of the Compendium.

Thanks heaps and heaps and heaps to eveyone who helped me out here - I'm dead happy!! :D

Blakey, the Java mostly-newbie.

_________________
The guy in the green hat.


Top
 Profile  
 
 Offline
Dragon
 
Joined: Tue Mar 24, 2009 2:34 pm
Posts: 859
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Mon Jun 21, 2010 6:52 am 
Blakey wrote:
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D


*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.

_________________
Reed (halfling sorcerer P3) // Collin (human fighter P2) // Cine (eladrin psion H2)
Sirath (deva shaman H1) // Alesk (dragonborn cleric H3) // Kevin (halfling barbarian H1)


Top
 Profile  
 
User avatar  Offline
Dragon
 
Joined: Fri Mar 23, 2007 10:27 am
Posts: 778
Location: Sussex, UK.
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Mon Jun 21, 2010 6:55 am 
Thanlis wrote:
Blakey wrote:
i did have apache installed already - I just had to add it to the build path. Then I tried this approach and IT WORKED!!!! :D


*cheer*! and thank you for the thread tracking your progress and so on; sharing your tool is awesome.


Dont get your hopes up! It will be a long time before I have anything worth sharing, if at all! But thanks for the help all the same. :D

_________________
The guy in the green hat.


Top
 Profile  
 
 Offline
Cave Troll
 
Joined: Sat Jun 05, 2010 7:58 pm
Posts: 41
 Post subject: Re: DDi Compendium API and Java Classes?
PostPosted: Mon Jun 21, 2010 7:50 am 
Awesome, glad you're up and running! =) If you want to post a working code sample as a generic example of how to access secured website info, I'm sure the next person that comes looking would appreciate it =)


Top
 Profile  
 
Display posts from previous:  Sort by  
Reply to topic  [ 64 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:

Who is online

In total there is 1 user online :: 0 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 243 on Sun Nov 04, 2012 6:14 am

Users browsing this forum: No registered users and 1 guest





Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group

Style based on Andreas08 by Andreas Viklund

Style by Elizabeth Shulman