The "malformed" error is probably because you don't escape values.
For me it works with the following code:

user_agent = """Mozilla/5.0 (Windows; U; Windows NT 6.1; ) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4"""
url1 = "http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
stdout = subprocess.check_output(['curl', '-sS', '-A', user_agent, '--compressed', '-L', url1])

Am 22.08.2017 um 17:34 schrieb bruce:
> Thanks for the reply.
>
> However I'm really interested in what might trigger the "malformed"
> issue.. given that the cmdline isn't throwing an error..
>
>
>
> On Tue, Aug 22, 2017 at 11:27 AM, Space One <space_at_wechall.net> wrote:
>> 1. You could use pycurl instead of subprocess + curl
>> 2. You aren't escaped shell arguments, possibly leading to security issues
>> 3. shell=True is bad, possibly leading to security issues
>>
>> Here is a better version of your script:
>>
>> user_agent ="Mozilla/5.0 (Windows; U; Windows NT 6.1; )
>> AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4"
>> url1="http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
>> cmd = ['curl', '-sS', '-A', user_agent, '--compressed', '-L', url1]
>> stdout = subprocess.check_output(cmd, stdout=subprocess.PIPE)
>>
>> Am 22.08.2017 um 17:06 schrieb bruce via curl-users:
>>> Hi.
>>>
>>> Testing out a curl cmd in a test py script (centos)
>>>
>>> The basic url
>>> http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493
>>>
>>> works with no issue in the browser.
>>>
>>> As far as I can tell, the server is compressing/encoding the returned data.
>>>
>>> The test
>>>
>>> curl -vvv -sS -A "Mozilla/5.0 (Windows; U; Windows NT 6.1; )
>>> AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3
>>> Safari/533.19.4" --compressed -L
>>> "http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
>>>
>>> works from the cmdline.
>>>
>>> However, running the test curl from a test py generates a malformed curl error!!
>>>
>>>
>>> test py script::
>>> user_agent ="Mozilla/5.0 (Windows; U; Windows NT 6.1; )
>>> AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3
>>> Safari/533.19.4"
>>>
>>> url1="http://www.bkstr.com/webapp/wcs/stores/servlet/booklookServlet?bookstore_id-1=047&term_id-1=Fall+2017&crn-1=12493"
>>>
>>> cmd='curl -sS '
>>> cmd=cmd+'-A "'+user_agent+'"'
>>> cmd=cmd+' --compressed '
>>> cmd=cmd+' -L "'+url1+'" '
>>> proc=subprocess.Popen(cmd, shell=True,stdout=subprocess.PIPE)
>>> s=proc.communicate()[0].strip()
>>>
>>> Is there something else in terms of headers I can try instead of the
>>> "compressed"??
>>>
>>> Is there something that someone can see that I can try to deal with
>>> the malformed issue!
>>>
>>> Without the --compressed, the returned content is "garbled". I've also
>>> tried to use '-H Accept-Encoding: deflate,gzip;' which results in
>>> garbled content as well.
>>>
>>> Comments/Thoughts?...
>>>
>>> Thanks
>>> -----------------------------------------------------------
>>> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
>>> Etiquette: https://curl.haxx.se/mail/etiquette.html

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-08-23