On 13 June 2013 18:43, Jimmy Niner <data9er_at_gmail.com> wrote:

> Is this possible? curl -L "domain.com/search.php?[urlserial.txt]"
> File.#1.htm
>
> I get how to do stuff like this:
>
> http://domain.com/SearchString?[1-100]&RemainingString -o SaveAs.#1.htm
> http://domain.com/SearchString?[001-100]&RemainingString -o SaveAs.#1.htm
> http://domain.com/SearchString?[a-z]&RemainingString -o SaveAs.#1.htm
> http://domain.com/SearchString?[1-100:10]&RemainingString -o SaveAs.#1.htm
> http://domain.com/SearchString?[a-z:2]&RemainingString -o SaveAs.#1.htm
>
> What I am trying to do is create a file that curl will read thousands of
> random numbers from it and insert those numbers 1 by 1 into the string.
>
> So this
> curl -L "http://domain.com/search.php?[urlserial.txt]" File.#1.htm
>
> opens urlserial.txt and reads 1 serial per call call and inserts the
> serial into the string and continues on its merry way.
>
> Obviously I can do this ...
> http://domain.com/SearchString?00000001&RemainingString -o
> SaveAs.00000001.htm
> http://domain.com/SearchString?00009001&RemainingString -o
> SaveAs.00009001.htm
> http://domain.com/SearchString?0ABC0301&RemainingString -o
> SaveAs.0ABC0301.htm
>
> But that starts making for some very large bat or sh files especially when
> you start reaching the hundreds of thousands or in some cases millions of
> urls for a site.
>
> For my purposes the best method would be to list the filenames/search in a
> single file then all one would need to do is call that file form within the
> string.
>
> Is there a way to do this with curl, or do I need to list them the long
> way inside a file?
>

Not sure what the above has to do with pycurl.

What about:

$ while read serial; do curl "
http://domain.com/SearchString?${serial}&RemainingString<http://domain.com/SearchString?00009001&RemainingString>"
-o "SaveAs.${serial}.htm"; done <urlserial.txt

-- 
Michael Wood <esiotrot_at_gmail.com>

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2013-06-14