Automatic form submission with PERL - debugging!

Question

Our client has a simple setup.

Page A has a form that submits to page B which displays the query results. Unfortunately, there is no other API or DB access to get the data.

Since we need to do this query often, we decided to automate this submission with Perl.

I've determined the form key value pairs of Page A with a sniffer and replicated the code. However, on running the program page B is throwing a HTTP 500 error with no additional meaningful explanation.

Any pointers to debug this code? Code in itself is simple:

use strict;
  use warnings;
  use LWP;
  my $browser = LWP::UserAgent->new;

  my $url = "targeturl.asp"
  my $response = $browser->post( $url,
    [
    "HisSort" => "1", 
    "RTsort" => "", 
    "chkHisRun" => "on", 
    "chkRTRun" => "on", 
    "optAdHoc" => "on", 
    "optHist" => "", 
    "optServer" => "servername", 
    "optStatus" => "", 
    "optWhat" => "H", 
    "txtEnd" => "", 
    "txtFields" => "1,0,10,17,11,18,24,19,21,25,1", 
    "txtHEnd" => "11/3/2010", 
    "txtHStart" => "11/1/2010", 
    "txtServer" => "", 
    "txtStart" => "",
    ]
  );

Note: I don't have access to the source of page A or page B

If you were sending an identical request, you would get an identical response. Therefore you need to find out where your request is different from the browser's. Are there cookies, for example? Your Perl script doesn't seem to try and set any headers to match the browsers, either - perhaps page B is failing to parse the User-Agent, say. — Matthew Wilson
– Matthew Wilson, Commented Nov 29, 2010 at 13:04
Also it can be UserAgent string ($browser->agent("...")) or some cookie received earlier... — gangabass
– gangabass, Commented Nov 29, 2010 at 13:20
Benoit - it is an intranet URL and was obfuscated when posting here. No issues there. — deepakkt
– deepakkt, Commented Nov 30, 2010 at 10:08

Dave Cross · Accepted Answer · 2010-11-29 14:23:00Z

1

Firstly, I suggest looking at WWW::Mechanize which is a friendlier wrapper around LWP. Secondly, if your HTTP client is getting 500 errors, then there should be something more meaningful in the web server error logs. And finally, as Matthew has mentioned, you need to closely examine the request being sent by the browser and work out how it differs from the request that your Perl program is sending.

answered Nov 29, 2010 at 14:23

Dave Cross

69.5k3 gold badges55 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

deepakkt Over a year ago

Matthew / Davorg - thanks for your inputs. I am constrained by what I can install on the server, but I will see if Mechanize is already there. I am sort of blindfolded in understanding what B is expecting. Is there anyway I can put something in between A and B to see the headers of A's output? All sniffer pages I have encountered seem to take a web page and display its headers and under the circumstances that doesn't help me much.

Matthew Wilson Over a year ago

Can you install Wireshark anywhere to snoop the traffic to B?

deepakkt · Accepted Answer · 2010-12-01 15:56:55Z

0

Team, This has been resolved. It eventually turned out that the problem was not with the headers but with the key value pairs I was sending. Page B wasn't doing validations on the fields and was plugging them into a query directly.

I had to try some brute force combinations (by testing with Page A) to get to what exactly page B was expecting.

Thanks to all who volunteered to help.

edited Dec 1, 2010 at 15:56

answered Dec 1, 2010 at 14:57

deepakkt

821 silver badge8 bronze badges

Collectives™ on Stack Overflow

Automatic form submission with PERL - debugging!

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related