This probably won't help much, but when I copy your curl command line
into a file, execute it, then display the html in a browser, I get the
exact same page as the browser shows when going direct to the page. In
other words, your curl command line isn't broken...

The *only* change I made was to put double-quotes around the browser
signature because my shell would do strange things with the [ ] and ( )
parts.

So, it works just fine on a Gentoo Linux box using:

    curl 7.10.1 (i686-pc-linux-gnu) libcurl/7.10.1 OpenSSL/0.9.6g
zlib/1.1.4

No cookies, no redirects, nothing fancy... There is, however, an
enormous amount of html fluff in the returned page. About 3/4's through
the file is the "Zoekresultaten - leverbare titels" text from the top of
the white area on the page.

Ralph Mitchell

ValueCD WebMaster wrote:

> I am having a heck of a hard time with the following
> problem:
>
> I'm trying to use CURL to fetch me product pages from a
> public e-commerce site based on UPC search (=bar code). The
> appropriate HTML form exists on the site, and in a regular
> browser (Internet Explorer) it works just fine:
>
> http://www.nl.bol.com/is-bin/INTERSHOP.enfinity/eCS/Store/nl
> /-/EUR/BOL_ParametricSearch-Start;sid=GftHZ1xgP-BHQGG7IN2EIC
> 5Ww0-7atG-Yew=?Section=MUSIC&orenge_quicksearch=true&orenge_
> quicksearch_subset=ean%7CEAN&search_query_keyword=5099750800
> 027
>
> This long URL displays "Bruce Springsteen: The Rising" page
> IN A BROWSER.
>
> However, when I try to get the very same page by CURL from
> the command line:
>
> curl -A
> Mozilla/4.0+[en]+(compatible;+MSIE+6.01;+Windows+XP) -o
> local_page.htm
> "http://www.nl.bol.com/is-bin/INTERSHOP.enfinity/eCS/Store/n
> l/-/EUR/BOL_ParametricSearch-Start;sid=GftHZ1xgP-BHQGG7IN2EI
> C5Ww0-7atG-Yew=?Section=MUSIC&orenge_quicksearch=true&orenge
> _quicksearch_subset=ean%7CEAN&search_query_keyword=509975080
> 0027"
>
> the server returns a page saying in so many words 'NO MATCH
> FOUND".
>
> Meaning CURL fetches a totally different page for the very
> same URL than a regular browser. Quite clearly, something is
> either missing or incorrect in my syntax.
>
> I've spend hours trying to figure out where the problem is,
> with no success. The site is public so there are no
> passwords or anything.
>
> I made CURL to send the same browser signature my browser
> has (with the -A option); no difference. I think it is
> possibly cookie-related, but I've tried to capture the
> header into a file first (with the -D option) and then send
> it with the subsequent request (with the -b option); no
> difference. I've tried the -L option if perhaps the page is
> redirected, no go either.
>
> I'm increasingly desperate. Can please someone "show me the
> light" before I lose my mind?
> Thank you very much!!
>
> (Note: The URLs are very long so your e-mail client may
> display them broken down to several lines; in that case you
> need to assemble each in a single URL before using them.)
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
Received on 2002-12-07