On Thu, Mar 30, 2017 at 2:03 PM, bruce <badouglas_at_gmail.com> wrote:

> Hi.
>
> I know this is a bit off topic. It's python + curl...!
>
>
> Trying to understand the "correct" way to run a sys command ("curl")
> and to get the potential stderr. Checking Stackoverflow (SO), implies
> that I should be able to use a raw/text cmd, with "shell=true".
>

Never use "shell=true". You want to specify the command to Popen as a list.
cmd=['curl', '-sS', '-A', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;
Trident/5.0; yie8)' , ......]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
s_out, s_err = p.communicate()

>
> If I leave the stderr out, and just use
> s=proc.communicate()
> the test works...
>
> Any pointers on what I might inspect to figure out why this hangs on
> the proc.communicate process/line??
>
> I'm showing a very small chunk of the test, but its the relevant piece.
>
> Thanks
>
>
> .
> .
> .
>
> cmd='[r" curl -sS '
> #cmd=cmd+'-A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0)
> Gecko/20100101 Firefox/38.0"'
> cmd=cmd+"-A '"+user_agent+"'"
> ##cmd=cmd+' --cookie-jar '+cname+' --cookie '+cname+' '
> cmd=cmd+' --cookie-jar '+ff+' --cookie '+ff+' '
> #cmd=cmd+'-e "'+referer+'" -d "'+tt+'" '
> #cmd=cmd+'-e "'+referer+'" '
> cmd=cmd+"-L '"+url1+"'"+'"]'
> #cmd=cmd+'-L "'+xx+'" '
>
> try_=1
> while(try_):
> proc=subprocess.Popen(cmd,
> shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
> s,err=proc.communicate()
> s=s.strip()
> err=err.strip()
>
> if(err==0):
> try_=''
>
> .
> .
> .
>
> the cmd is generated to be:
> cmd=[r" curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
> 6.1; Trident/5.0; yie8)' --cookie-jar
> /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
> --cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c
> 88e.lwp
> -L 'http://www6.austincc.edu/schedule/index.php?op=browse&opcla
> ss=ViewSched&term=216F000&disciplineid=PCACC&yr=2016&ct=CC'"]
>
>
>
>
> test code hangs, ctrl-C generates the following:
> ^CTraceback (most recent call last):
> File "/crawl_tmp/austinccFetch_cloud_test.py", line 3363, in <module>
> ret=fetchClassSectionFacultyPage(a)
> File "/crawl_tmp/austinccFetch_cloud_test.py", line 978, in
> fetchClassSectionFacultyPage
> (s,err)=proc.communicate()
> File "/usr/lib64/python2.6/subprocess.py", line 732, in communicate
> stdout, stderr = self._communicate(input, endtime)
> File "/usr/lib64/python2.6/subprocess.py", line 1328, in _communicate
> stdout, stderr = self._communicate_with_poll(input, endtime)
> File "/usr/lib64/python2.6/subprocess.py", line 1400, in
> _communicate_with_poll
> ready = poller.poll(self._remaining_time(endtime))
> KeyboardInterrupt
>
>
>
> This works from the cmdline:
> curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;
> Trident/5.0; yie8)' --cookie-jar
> /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
> --cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c
> 88e.lwp
> -L 'http://www6.austincc.edu/schedule/index.php?op=browse&opcla
> ss=ViewSched&term=216F000&disciplineid=PCACC&yr=2016&ct=CC'
>

-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-03-31