1. You could use pycurl instead of subprocess + curl
2. You aren't escaped shell arguments, possibly leading to security issues
3. shell=True is bad, possibly leading to security issues

Here is a better version of your script:

user_agent ="Mozilla/5.0 (Windows; U; Windows NT 6.1; )
AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4"
url1="http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
cmd = ['curl', '-sS', '-A', user_agent, '--compressed', '-L', url1]
stdout = subprocess.check_output(cmd, stdout=subprocess.PIPE)

Am 22.08.2017 um 17:06 schrieb bruce via curl-users:
> Hi.
>
> Testing out a curl cmd in a test py script (centos)
>
> The basic url
> http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493
>
> works with no issue in the browser.
>
> As far as I can tell, the server is compressing/encoding the returned data.
>
> The test
>
> curl -vvv -sS -A "Mozilla/5.0 (Windows; U; Windows NT 6.1; )
> AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3
> Safari/533.19.4" --compressed -L
> "http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
>
> works from the cmdline.
>
> However, running the test curl from a test py generates a malformed curl error!!
>
>
> test py script::
> user_agent ="Mozilla/5.0 (Windows; U; Windows NT 6.1; )
> AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3
> Safari/533.19.4"
>
> url1="http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
>
> cmd='curl -sS '
> cmd=cmd+'-A "'+user_agent+'"'
> cmd=cmd+' --compressed '
> cmd=cmd+' -L "'+url1+'" '
> proc=subprocess.Popen(cmd, shell=True,stdout=subprocess.PIPE)
> s=proc.communicate()[0].strip()
>
> Is there something else in terms of headers I can try instead of the
> "compressed"??
>
> Is there something that someone can see that I can try to deal with
> the malformed issue!
>
> Without the --compressed, the returned content is "garbled". I've also
> tried to use '-H Accept-Encoding: deflate,gzip;' which results in
> garbled content as well.
>
> Comments/Thoughts?...
>
> Thanks
> -----------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
> Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-08-22