Web Crawling?

Lasogette

Lasogette

Supremacy Member
Joined
Dec 20, 2012
Messages
8,190
Reaction score
4,543
Hi all, been away from web technologies for quite a while. Have a implementation which i want to check for feasibility.

I wish to get data from the below URL and then store the data into my own database.
https://secure.mas.gov.sg/dir/domesticinterestrates.aspx

Not sure if it is actually possible and if it what technique is it being called. Not waiting to get spoon fed but just need some info to at least do a google search. Thanks.
:)
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,550
Reaction score
1,302
PippingCafe said:
Hi all, been away from web technologies for quite a while. Have a implementation which i want to check for feasibility.

I wish to get data from the below URL and then store the data into my own database.
https://secure.mas.gov.sg/dir/domesticinterestrates.aspx

Not sure if it is actually possible and if it what technique is it being called. Not waiting to get spoon fed but just need some info to at least do a google search. Thanks.
:)
Click to expand...

Web scraping is what you wanted. No fixed approach, basically is HTML downloading and parsing. You can also run the javascript on it using various javascript engines. One possible set of technologies is Node.js + jsdom for headless javascript dom execution.
 
Last edited:
Lasogette

Lasogette

Supremacy Member
Joined
Dec 20, 2012
Messages
8,190
Reaction score
4,543
davidktw said:
Web scraping is what you wanted. No fixed approach, basically is HTML downloading and parsing. You can also run the javascript on it using various javascript engines. One possible set of technologies is Node.js + jsdom for headless javascript dom execution.
Click to expand...

Hi david, thanks for the reply. However i found out that webpage requires me to click on my desired check boxes and return me the interest rates. However the ASPX page rendered is the same as the query page. So can i say that there is no way for me to get to the 2nd page to use this technique of web scraping because the URL is still the same? Thanks.
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,550
Reaction score
1,302
PippingCafe said:
Hi david, thanks for the reply. However i found out that webpage requires me to click on my desired check boxes and return me the interest rates. However the ASPX page rendered is the same as the query page. So can i say that there is no way for me to get to the 2nd page to use this technique of web scraping because the URL is still the same? Thanks.
Click to expand...

Whether it's javascript and/or ajax, flash or any plugin, at the end of the day, it's mostly HTTP request and response that is sent and received by the browser.

For your case, use Chrome or Firefox, open the developer tab and monitor the network traffic of all requests and response. See what is fire off from the webpage when you click on what you want to see and click on the submit or any UI button.

It should response to one or more HTTP requests, you can always use any method from programmatic, or utility like wget or curl or so forth to send the http request over yourself and get back the intended response. What you want is the data, not how they are shown on the screen unless the rendering matters to you.

Just to give you an idea. I have this iphone sales script that run in my server in AWS when iphones are about the launch. The iphone sales page is quite interesting, but at the end of the day, what I'm interested in is to find out when Apple store opens for iPhone

Using firefox, I found out what evetually matters is the Ajax call of the following
Code: 
curl 'http://store.apple.com/sg/buyFlowVariationUpdate/IPHONE5S?option.dimensionColor=gold&option.dimensionCapacity=16gb&carrierPolicyType=UNLOCKED&node=home%2Fshop_iphone%2Ffamily%2Fiphone5s&step=select'
{"body":{"content":{"selected":{"productTitle":"iPhone 5s 16GB Gold Unlocked","productImage":"<img class=\"ir\" src=\"http://store.storeimages.cdn-apple.com/7390/as-images.apple.com/is/image/AppleInc/2013-iphone5s-gold?wid=110&hei=78&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" width=\"110\" height=\"78\" data-scale-params-2=\"wid=220&hei=156&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" />","purchaseOptions":{"financing":"\n\t\t\t\t\n\t\n\t\t\n\t\t\t\t\n\t\t            \t\n    \n    <a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Financing Available<\/a>\n\n<!--\n12 months interest-free<br>\ninstallment plan available.<br>\n<a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Learn more<\/a>\n-->\n\n\n    \n\n\n\t\t            \n\t\t\t\n\t\n\n\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t","price":"\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t<span class=\"current_price\">\n\t\t\t\t\n\t\t\t\t\t<span itemtype=\"http://schema.org/Offer\" itemprop=\"offers\" itemscope=\"itemscope\">\n\t\t\t\t\t\n\n\t\t\t\t\t\t<meta itemprop=\"priceCurrency\" content=\"SGD\"><\/meta>\n\t\t\t\t\t\t<span itemprop=\"price\">\n\t\t\t\t\t\t\tS$ 988.00\n\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/span>\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\n\t\t\n    \n\t\n\t\n\n\n\n\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t","shippingPrice":"\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<span class=\"free-shipping\">Free Shipping<\/span>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t","shippingLead":"\n\t\t\t\t\n    <span class=\"shipping-description\">\n    \t\n\t    \t\n\t        \tAvailable to ship<b>:<\/b><br/>\n\t        \n\t        <span class=\"customer_commit_display\">Currently unavailable<\/span>\n\t        \n\t        \n\t        \n\t        \n\t    \n    <\/span>\n\n\n\n\n\t\t\t","promotions":" ","isBuyable":false},"partNumber":"MF354ZA/A"},"socialSharingDisplay":"\n            \t\n    \n        \n\t<ul class=\"socialsharing\">\n\t\t\n\t\t\t<li class=\"facebook\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h450\" class=\"facebook\" id=\"facebook\" href=\"/sg/socialsharing/redirect?dn=sp&partner=facebook&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Share<\/span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Facebook\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/li>\n\t\t\n\t\t\t<li class=\"twitter\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h256\" class=\"twitter\" id=\"twitter\" href=\"/sg/socialsharing/redirect?dn=sp&partner=twitter&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Tweet<\/span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Twitter\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/li>\n\t\t\n\t<\/ul>\n\n    \n\n\n        \t"}},"head":{"status":"200","data":{}}}

Notice it's a bunch of text as response in JSON format. Parse the JSON and it gives me this
Code: 
{
    "body": {
        "content": {
            "selected": {
                "partNumber": "MF354ZA/A", 
                "productImage": "<img class=\"ir\" src=\"http://store.storeimages.cdn-apple.com/7390/as-images.apple.com/is/image/AppleInc/2013-iphone5s-gold?wid=110&hei=78&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" width=\"110\" height=\"78\" data-scale-params-2=\"wid=220&hei=156&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" />", 
                "productTitle": "iPhone 5s 16GB Gold Unlocked", 
                "purchaseOptions": {
                    "financing": "\n\t\t\t\t\n\t\n\t\t\n\t\t\t\t\n\t\t            \t\n    \n    <a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Financing Available</a>\n\n<!--\n12 months interest-free<br>\ninstallment plan available.<br>\n<a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Learn more</a>\n-->\n\n\n    \n\n\n\t\t            \n\t\t\t\n\t\n\n\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t", 
                    "isBuyable": false, 
                    "price": "\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t<span class=\"current_price\">\n\t\t\t\t\n\t\t\t\t\t<span itemtype=\"http://schema.org/Offer\" itemprop=\"offers\" itemscope=\"itemscope\">\n\t\t\t\t\t\n\n\t\t\t\t\t\t<meta itemprop=\"priceCurrency\" content=\"SGD\"></meta>\n\t\t\t\t\t\t<span itemprop=\"price\">\n\t\t\t\t\t\t\tS$ 988.00\n\t\t\t\t\t\t</span>\n\t\t\t\t\t</span>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</span>\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\n\t\t\n    \n\t\n\t\n\n\n\n\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t", 
                    "promotions": " ", 
                    "shippingLead": "\n\t\t\t\t\n    <span class=\"shipping-description\">\n    \t\n\t    \t\n\t        \tAvailable to ship<b>:</b><br/>\n\t        \n\t        <span class=\"customer_commit_display\">[COLOR="Red"][B]Currently unavailable[/B][/COLOR]</span>\n\t        \n\t        \n\t        \n\t        \n\t    \n    </span>\n\n\n\n\n\t\t\t", 
                    "shippingPrice": "\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<span class=\"free-shipping\">Free Shipping</span>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t"
                }
            }, 
            "socialSharingDisplay": "\n            \t\n    \n        \n\t<ul class=\"socialsharing\">\n\t\t\n\t\t\t<li class=\"facebook\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h450\" class=\"facebook\" id=\"facebook\" href=\"/sg/socialsharing/redirect?dn=sp&partner=facebook&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Share</span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Facebook\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t</span>\n\t\t\t\t\t\t</a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</li>\n\t\t\n\t\t\t<li class=\"twitter\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h256\" class=\"twitter\" id=\"twitter\" href=\"/sg/socialsharing/redirect?dn=sp&partner=twitter&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Tweet</span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Twitter\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t</span>\n\t\t\t\t\t\t</a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</li>\n\t\t\n\t</ul>\n\n    \n\n\n        \t"
        }
    }, 
    "head": {
        "data": {}, 
        "status": "200"
    }
}

What I'm interested in is the text in RED(scroll to the right). The existent of it certainly tell me what I want to know. This is web scraping. I pick out what I want to know. I don't need to follow the workflow of the webpage exactly. Get what I'm talking about ?
 
Lasogette

Lasogette

Supremacy Member
Joined
Dec 20, 2012
Messages
8,190
Reaction score
4,543
davidktw said:
Whether it's javascript and/or ajax, flash or any plugin, at the end of the day, it's mostly HTTP request and response that is sent and received by the browser.

For your case, use Chrome or Firefox, open the developer tab and monitor the network traffic of all requests and response. See what is fire off from the webpage when you click on what you want to see and click on the submit or any UI button.

It should response to one or more HTTP requests, you can always use any method from programmatic, or utility like wget or curl or so forth to send the http request over yourself and get back the intended response. What you want is the data, not how they are shown on the screen unless the rendering matters to you.

Just to give you an idea. I have this iphone sales script that run in my server in AWS when iphones are about the launch. The iphone sales page is quite interesting, but at the end of the day, what I'm interested in is to find out when Apple store opens for iPhone

Using firefox, I found out what evetually matters is the Ajax call of the following
Code: 
curl 'http://store.apple.com/sg/buyFlowVariationUpdate/IPHONE5S?option.dimensionColor=gold&option.dimensionCapacity=16gb&carrierPolicyType=UNLOCKED&node=home%2Fshop_iphone%2Ffamily%2Fiphone5s&step=select'
{"body":{"content":{"selected":{"productTitle":"iPhone 5s 16GB Gold Unlocked","productImage":"<img class=\"ir\" src=\"http://store.storeimages.cdn-apple.com/7390/as-images.apple.com/is/image/AppleInc/2013-iphone5s-gold?wid=110&hei=78&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" width=\"110\" height=\"78\" data-scale-params-2=\"wid=220&hei=156&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" />","purchaseOptions":{"financing":"\n\t\t\t\t\n\t\n\t\t\n\t\t\t\t\n\t\t            \t\n    \n    <a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Financing Available<\/a>\n\n<!--\n12 months interest-free<br>\ninstallment plan available.<br>\n<a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Learn more<\/a>\n-->\n\n\n    \n\n\n\t\t            \n\t\t\t\n\t\n\n\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t","price":"\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t<span class=\"current_price\">\n\t\t\t\t\n\t\t\t\t\t<span itemtype=\"http://schema.org/Offer\" itemprop=\"offers\" itemscope=\"itemscope\">\n\t\t\t\t\t\n\n\t\t\t\t\t\t<meta itemprop=\"priceCurrency\" content=\"SGD\"><\/meta>\n\t\t\t\t\t\t<span itemprop=\"price\">\n\t\t\t\t\t\t\tS$ 988.00\n\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/span>\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\n\t\t\n    \n\t\n\t\n\n\n\n\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t","shippingPrice":"\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<span class=\"free-shipping\">Free Shipping<\/span>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t","shippingLead":"\n\t\t\t\t\n    <span class=\"shipping-description\">\n    \t\n\t    \t\n\t        \tAvailable to ship<b>:<\/b><br/>\n\t        \n\t        <span class=\"customer_commit_display\">Currently unavailable<\/span>\n\t        \n\t        \n\t        \n\t        \n\t    \n    <\/span>\n\n\n\n\n\t\t\t","promotions":" ","isBuyable":false},"partNumber":"MF354ZA/A"},"socialSharingDisplay":"\n            \t\n    \n        \n\t<ul class=\"socialsharing\">\n\t\t\n\t\t\t<li class=\"facebook\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h450\" class=\"facebook\" id=\"facebook\" href=\"/sg/socialsharing/redirect?dn=sp&partner=facebook&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Share<\/span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Facebook\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/li>\n\t\t\n\t\t\t<li class=\"twitter\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h256\" class=\"twitter\" id=\"twitter\" href=\"/sg/socialsharing/redirect?dn=sp&partner=twitter&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Tweet<\/span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Twitter\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/li>\n\t\t\n\t<\/ul>\n\n    \n\n\n        \t"}},"head":{"status":"200","data":{}}}

Notice it's a bunch of text as response in JSON format. Parse the JSON and it gives me this
Code: 
{
    "body": {
        "content": {
            "selected": {
                "partNumber": "MF354ZA/A", 
                "productImage": "<img class=\"ir\" src=\"http://store.storeimages.cdn-apple.com/7390/as-images.apple.com/is/image/AppleInc/2013-iphone5s-gold?wid=110&hei=78&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" width=\"110\" height=\"78\" data-scale-params-2=\"wid=220&hei=156&fmt=jpeg&qlt=95&op_sharpen=0&resMode=bicub&op_usm=0.5,1.5,0,0&iccEmbed=0&layer=comp&.v=1378488417662\" />", 
                "productTitle": "iPhone 5s 16GB Gold Unlocked", 
                "purchaseOptions": {
                    "financing": "\n\t\t\t\t\n\t\n\t\t\n\t\t\t\t\n\t\t            \t\n    \n    <a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Financing Available</a>\n\n<!--\n12 months interest-free<br>\ninstallment plan available.<br>\n<a href=\"http://store.apple.com/sg/browse/campaigns/financing\" target=\"_blank\">Learn more</a>\n-->\n\n\n    \n\n\n\t\t            \n\t\t\t\n\t\n\n\n\t\t\t\t\n\t\t\t\t \n\t\t\t\t\n\t\t\t", 
                    "isBuyable": false, 
                    "price": "\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\n\t\n\t\n\t\t\n\t\t\t<span class=\"current_price\">\n\t\t\t\t\n\t\t\t\t\t<span itemtype=\"http://schema.org/Offer\" itemprop=\"offers\" itemscope=\"itemscope\">\n\t\t\t\t\t\n\n\t\t\t\t\t\t<meta itemprop=\"priceCurrency\" content=\"SGD\"></meta>\n\t\t\t\t\t\t<span itemprop=\"price\">\n\t\t\t\t\t\t\tS$ 988.00\n\t\t\t\t\t\t</span>\n\t\t\t\t\t</span>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</span>\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\n\t\t\n\t\t\n    \n\t\n\t\n\n\n\n\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t", 
                    "promotions": " ", 
                    "shippingLead": "\n\t\t\t\t\n    <span class=\"shipping-description\">\n    \t\n\t    \t\n\t        \tAvailable to ship<b>:</b><br/>\n\t        \n\t        <span class=\"customer_commit_display\">[COLOR="Red"][B]Currently unavailable[/B][/COLOR]</span>\n\t        \n\t        \n\t        \n\t        \n\t    \n    </span>\n\n\n\n\n\t\t\t", 
                    "shippingPrice": "\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t<span class=\"free-shipping\">Free Shipping</span>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t"
                }
            }, 
            "socialSharingDisplay": "\n            \t\n    \n        \n\t<ul class=\"socialsharing\">\n\t\t\n\t\t\t<li class=\"facebook\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h450\" class=\"facebook\" id=\"facebook\" href=\"/sg/socialsharing/redirect?dn=sp&partner=facebook&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Share</span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Facebook\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t</span>\n\t\t\t\t\t\t</a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</li>\n\t\t\n\t\t\t<li class=\"twitter\">\n\t\t\t\t\n\t\t\t\t    \n\t\t\t\t\t\t<a target=\"popup-w600-h256\" class=\"twitter\" id=\"twitter\" href=\"/sg/socialsharing/redirect?dn=sp&partner=twitter&uri=%252Fsg%252Fbuy-iphone%252Fiphone5s%253Fcid%253Diphone&uh=-1099047713\" data-evar1=\"AOS: \">\n\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<span>Tweet</span>\n\t\t\t\t\t\t    \n\t\t\t\t\t\t\t<span class=\"a11y\">\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tShare on Twitter\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t</span>\n\t\t\t\t\t\t</a>\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t</li>\n\t\t\n\t</ul>\n\n    \n\n\n        \t"
        }
    }, 
    "head": {
        "data": {}, 
        "status": "200"
    }
}

What I'm interested in is the text in RED(scroll to the right). The existent of it certainly tell me what I want to know. This is web scraping. I pick out what I want to know. I don't need to follow the workflow of the webpage exactly. Get what I'm talking about ?
Click to expand...

Hi David, I'll do some intense googling to try out first. Abit hard to visualize now. Thanks for pointing me in the correct direction.
 
davidktw

davidktw

Arch-Supremacy Member
Joined
Apr 15, 2010
Messages
13,550
Reaction score
1,302
PippingCafe said:
Hi David, I'll do some intense googling to try out first. Abit hard to visualize now. Thanks for pointing me in the correct direction.
Click to expand...

For your MAS thingie. There is the download button, it actually give you a CSV file.
The exact CURL command you need is the following if I will to select ALL the options.

General direction for you is go learn how to use the Chrome Developer Tool

Code: 
curl 'https://secure.mas.gov.sg/dir/domesticinterestrates.aspx' -H 'Origin: https://secure.mas.gov.sg' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Host: secure.mas.gov.sg' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: https://secure.mas.gov.sg/dir/domesticinterestrates.aspx' -H 'Cookie: ASP.NET_SessionId=gb2zwhv5edhecwxxxxxxxxx; MAScookie=xPk0LX0uID4TuGZDxFvD1otS0xxxxxxxxxxxxtKCJbwmt9tjJAYFmo9vlfwY26pkGV6ooOrtfnvw=' -H 'Connection: keep-alive' --data '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKMTYxNTA5MzI4Ng9kFgJmD2QWAgIDD2QWAgIBD2QWAgIFD2QWCAIBDw8WAh4HVmlzaWJsZWhkZAIDDw8WAh8AaGRkAgUPZBYEAgEPEGQPFxxxxxxxxxxIDAgQCBQIGAgcCCAIJAgoCCwIMAg0CDgIPAhACEQISAhMCFAIVAhYCFwIYAhkCGhYbEAUEMTk4NwUEMTk4N2cQBQQxOTg4BQQxOTg4ZxAFBDE5ODkFBDE5ODlnEAUEMTk5MAUEMTk5MGcQBQQxOTkxBQQxOTkxZxAFBDE5OTIFBDE5OTJnEAUEMTk5MwUEMTk5M2cQBQQxOTk0BQQxOTk0ZxAFBDE5OTUFBDE5OTVnEAUEMTk5NgUEMTk5NmcQBQQxOTk3BQQxOTk3ZxAFBDE5OTgFBDE5OThnEAUEMTk5OQUEMTk5OWcQBQQyMDAwBQQyMDAwZxAFBDIxxxxxxxxxxxxxxxxxQUEMjAwNWcQBQQyMDA2BQQyMDA2ZxAFBDIwMDcFBDIwMDdnEAUEMjAwOAUEMjAwOGcQBQQyMDA5BQQyMDA5ZxAFBDIwMTAFBDIwMTBnEAUEMjAxMQUEMjAxMWcQBQQyMDEyBQQyMDEyZxAFBDIwMTMFBDIwMTNnFgECGmQCAw8QZA8WG2YCAQICAgMCBAIFAgYCBwIIAgkCCgILAgwCDQIOAg8CEAIRAhICEwIUAhUCFgIXAhgCGQIaFhsQBQQxOTg3BQQxOTg3ZxAFBDE5ODgFBDE5ODhnEAUEMTk4OQUEMTk4OWcQBQQxOTkwBQQxOTkwZxAFBDE5OTEFBDE5OTFnEAUEMTk5MgUEMTk5MmcQBQQxOTkzBQQxOTkzZxAFBDE5OTQFBDE5OTRnEAUEMTk5NQUEMTk5NWcQBQQxOTk2BQQxOTk2ZxAFBDE5OTcFBDE5OTdnEAUEMTk5OAUEMTk5OGcQBQQxOTk5BQQxOTk5ZxAFBDIwMDAFBDIwMDBnEAUEMjAwMQUEMjAwMWcQBQQyMDAyBQQyMDAyZxAFBDIwMDMFBDIwMDNnEAUEMjAwNAUEMjAwNGcQBQQyMDA1BQQyMDA1ZxAFBDIwMDYFBDIwMDZnEAUEMjAwNwUEMjAwN2cQBQQyMDA4BQQyMDA4ZxAFBDIwMDkFBDIwMDlnEAUEMjAxMAUEMjAxMGcQBQQyMDExBQQyMDExZxAFBDIwMTIFBDIwMTJnEAUEMjAxMwUEMjAxM2dkZAIPDxBkZBYAZBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WDgUvY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRDb2x1bW5zQ2hlY2tCb3hMaXN0JDAFL2N0bDAwJENvbnRlbnRQbGFjZUhvbGRlcjEkQ29sdW1uc0NoZWNrQm94TGlzdCQxBS9jdGwwMCRDb250ZW50UGxhY2VIb2xkZXIxJENvbHVtbnNDaGVja0JveExpc3QkMgUvY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRDb2x1bW5zQ2hlY2tCb3hMaXN0JDMFL2N0bDAwJENvbnRlbnRQbGFjZUhvbGRlcjEkQ29sdW1uc0NoZWNrQm94TGlzdCQ0BS9jdGwwMCRDb250ZW50UGxhY2VIb2xkZXIxJENvbHVtbnNDaGVja0JveExpc3QkNQUvY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRDb2x1bW5zQ2hlY2tCb3hMaXN0JDYFL2N0bDAwJENvbnRlbnRQbGFjZUhvbGRlcjEkQ29sdW1uc0NoZWNrQm94TGlzdCQ3BS9jdGwwMCRDb250ZW50UGxhY2VIb2xkZXIxJENvbHVtbnNDaGVja0JveExpc3QkOAUvY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRDb2x1bW5zQ2hlY2tCb3hMaXN0JDkFMGN0bDAwJENvbnRlbnRQbGFjZUhvbGRlcjEkQ29sdW1uc0NoZWNrQm94TGlzdCQxMAUwY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRDb2x1bW5zQ2hlY2tCb3hMaXN0JDExBTBjdGwwMCRDb250ZW50UGxhY2VIb2xkZXIxJENvbHVtbnNDaGVja0JveExpc3QkMTIFMGN0bDAwJENvbnRlbnRQbGFjZUhvbGRlcjEkQ29sdW1uc0NoZWNrQm94TGlzdCQxMkHTw6KAmuLYQ2zqk4BYzj6FgNCk&__EVENTVALIDATION=%2FwEWYgLSsuSuBwL2jt%2BdBwL2jqv1AgL2jreeDQLLp6mxDgLLp7XaBgLLp8H%2BAQLLp%2B2jCALLp%2FlEAsunhekLAsunkZICAsunvbcNAsuniY4IAsunlbMDAo%2FE%2FMUGAo%2FEiO4BAo%2FElJMIAo%2FEoLQDAo%2FEzNgLAo%2FE2P0CAo%2FE5KYNAo%2FE8MsFAo%2FE3KIDAo%2FE6McLAuT92t4MAuT95oMHAuT98qQOAuT9nskGAq6MgpwOAq6M9vQLAq6M6p8EApOl9LAHApOl6NsPApOlnP8IApOlsKIBApOlpMUJApOl2OgCApOlzJMLApOl4LYEApOl1I8BApOlyLIKAtfGocQPAtfG1e8IAtfGyZIBAtfG%2FbUKAtfGkdkCAtfGhfwLAtfGuacEAtfGrcoMAtfGgaMKAtfGtcYCArz%2Fh98FArz%2Fu4IOArz%2Fr6UHArz%2Fw8gPApu%2B%2BcQDApq%2B%2BcQDApm%2B%2BcQDApi%2B%2BcQDAp%2B%2B%2BcQDAp6%2B%2BcQDAp2%2B%2BcQDAoy%2B%2BcQDAoO%2B%2BcQDApu%2BuccDApu%2BtccDApu%2BsccDApmcr7UFApicr7UFApucr7UFApqcr7UFAp2cr7UFApycr7UFAp%2Bcr7UFAo6cr7UFAoGcr7UFApmc77YFApmc47YFApmc57YFAvXq%2F5EBAuHq%2F5EBAuvq%2F5EBAt7q%2F5EBAoDiyWMCg%2BLJYwL16MXRAwLa%2F6e8CQK%2FloqnDwKkreyRBQKJxM78CgLu2rBnAtPxktIGAriI9bwMAs2ytPsEArLJluYKAtr%2F59MLAtr%2F0%2FgCAtr%2Fj4oNuL3WYTxOET0jEKL76AusNe7wk8E%3D&ctl00%24ContentPlaceHolder1%24StartYearDropDownList=2013&ctl00%24ContentPlaceHolder1%24EndYearDropDownList=2013&ctl00%24ContentPlaceHolder1%24StartMonthDropDownList=1&ctl00%24ContentPlaceHolder1%24EndMonthDropDownList=10&ctl00%24ContentPlaceHolder1%24FrequencyDropDownList=M&ctl00%24ContentPlaceHolder1%24Button2=Download&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%240=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%241=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%242=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%243=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%244=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%245=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%246=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%247=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%248=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%249=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%2410=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%2411=on&ctl00%24ContentPlaceHolder1%24ColumnsCheckBoxList%2412=on' --compressed

Update
rm8lky.png


For HTTP sessions that may change from time to time. The idea is to create it by visiting the page that generate it first. Capture all the cookies and send it back to the originating page. It's not difficult. Feel free to enquire if you hit any roadblock.
 
Last edited:
You must log in or register to reply here.
Important Forum Advisory Note
This forum is moderated by volunteer moderators who will react only to members' feedback on posts. Moderators are not employees or representatives of HWZ Forums. Forum members and moderators are responsible for their own posts. Please refer to our Community Guidelines and Standards and Terms and Conditions for more information.
Top