perl - HTTPS

[peterchan75]

Hi all,

Previously, I was able to download using perl get http request. But now, the site has changed to https. So, I am getting "Failed: 401 Unauthorized" respond. I am still able to compose the https url and use IE to download. Previously, I composed the url and use perl get to download. Possible to use perl to download using IE ?

Thanks in advance for your tips.

 

[davidktw]

Hi all,

Previously, I was able to download using perl get http request. But now, the site has changed to https. So, I am getting "Failed: 401 Unauthorized" respond. I am still able to compose the https url and use IE to download. Previously, I composed the url and use perl get to download. Possible to use perl to download using IE ?

Thanks in advance for your tips.

401 is that the site is protected either with HTTP digest or basic. You can simulate any user agent you like as long as the server is not exploiting certain behaviour of the browser to determine the browser type.

Do you have the username and password to access the protected site?

 

[peterchan75]

401 is that the site is protected either with HTTP digest or basic. You can simulate any user agent you like as long as the server is not exploiting certain behaviour of the browser to determine the browser type.

Do you have the username and password to access the protected site?

Thanks.

There is no need for username and password. All I need to do is just to compose the url and paste it onto IE and hit enter. IE will respond with a CSV formated file and prompt me to save it. Instead of saving it, I need perl to grab this file.

 

[davidktw]

Thanks.

There is no need for username and password. All I need to do is just to compose the url and paste it onto IE and hit enter. IE will respond with a CSV formated file and prompt me to save it. Instead of saving it, I need perl to grab this file.

Use chrome and access, if you are also able to access, then get the curl command and try running it from the command line. If it also can be run, then perl wouldn't be an issue

 

[peterchan75]

Use chrome and access, if you are also able to access, then get the curl command and try running it from the command line. If it also can be run, then perl wouldn't be an issue

Both are alien to me. Slippery slop. I will let this one past.

Thanks.

 

[davidktw]

Both are alien to me. Slippery slop. I will let this one past.

Thanks.

How is this hard?
https://lornajane.net/posts/2013/chrome-feature-copy-as-curl

 

[peterchan75]

How is this hard?
https://lornajane.net/posts/2013/chrome-feature-copy-as-curl

Well this may be a snap you !
But I have learned all these stuffs for this one off stuff. Don't really add up.

I manage to get perl to launch IE and Navigate to the site.
IE prompt window... Open Save Cancel menu.

Anyway to grab it and store it to string ?

 

[davidktw]

Well this may be a snap you !
But I have learned all these stuffs for this one off stuff. Don't really add up.

I manage to get perl to launch IE and Navigate to the site.
IE prompt window... Open Save Cancel menu.

Anyway to grab it and store it to string ?

Then it will always be an one off stuffs for you. Life at the end of the route is death, it doesn't adds up too. Anyway I have shown you the way, your decision alright?

 

[peterchan75]

Then it will always be an one off stuffs for you. Life at the end of the route is death, it doesn't adds up too. Anyway I have shown you the way, your decision alright?

I some dudes over at StackOverflow talking about the some stuff.
No so straight forward implementation.

 

[davidktw]

I some dudes over at StackOverflow talking about the some stuff.
No so straight forward implementation.

Actually I don't know what is the difficulty you are talking about. You mean control the IE? If you want to control the IE from a script, use selenium with the IE webdriver.

 

[peterchan75]

Found the solution but it's in Excel VBA. Will attempt to convert to perl.

http://www.xlautomation.com.au/free-spreadsheets/yahoo-historical-price-extract

 

[u0206397]

Well this may be a snap you !
But I have learned all these stuffs for this one off stuff. Don't really add up.

I manage to get perl to launch IE and Navigate to the site.
IE prompt window... Open Save Cancel menu.

Anyway to grab it and store it to string ?

 

[peterchan75]

Is that so difficult that it stunt Mr.Bean ?

 

[u0206397]

Is that so difficult that it stunt Mr.Bean ?

I am just surprised you chose a more convoluted way of approaching the problem, than trying to do it the more straightforward way.

Controlling IE -- possibly through Win32::OLE -- to me is a more complicated way that may give you even more trouble as you go along.

Just my personal opinion.

Regarding why Perl cannot handle https, I am not sure if you are using LWP::UserAgent. CPAN mentions the following:

The libwww-perl core no longer bundles protocol plugins for SSL. You will need to install LWP:: Protocol::https separately to enable support for processing https-URLs.

This may or may not be the cause of your problem.

If you're running ActiveState ActivePerl on Windows, it may be just a module install that looks like this in the Command Prompt:

ppm install LWP::Protocol::https

If you are running on Linux/Unix, then do this in the terminal:

sudo cpan install LWP::Protocol::https

 

[peterchan75]

Thanks.
Manage to get it to work. The webserver needs a cookie.

$h = new HTTP::Headers();
$h->header('Cookie' => $kookies);
$user_agent = new LWP::UserAgent; 
$request = new HTTP::Request ('GET',$url,$h);
$response = $user_agent->request($request);

 

[u0206397]

You might find tools like Fiddler and Chrome DevTools useful for inspecting HTTP requests and responses in future.

 

[peterchan75]

I copy the idea from the link in post #11.
In the VBA code, it "request" a cookie from server and send https request with cookie back, only then the webserver with take it as a valid request. It works with IE or Firefox because the browser "soft of auto" provide cookie to the server.

That's what davidktw was telling me the use curl with chrome dev tool. Too many things to learn and I treat this as one-off. I give it a past as there is too many things to learn. One-off strategy.