Hi.

I know this is a bit off topic. It's python + curl...!

Trying to understand the "correct" way to run a sys command ("curl")
and to get the potential stderr. Checking Stackoverflow (SO), implies
that I should be able to use a raw/text cmd, with "shell=true".

If I leave the stderr out, and just use
     s=proc.communicate()
the test works...

Any pointers on what I might inspect to figure out why this hangs on
the proc.communicate process/line??

I'm showing a very small chunk of the test, but its the relevant piece.

Thanks

.
.
.

  cmd='[r" curl -sS '
  #cmd=cmd+'-A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0)
Gecko/20100101 Firefox/38.0"'
  cmd=cmd+"-A '"+user_agent+"'"
  ##cmd=cmd+' --cookie-jar '+cname+' --cookie '+cname+' '
  cmd=cmd+' --cookie-jar '+ff+' --cookie '+ff+' '
  #cmd=cmd+'-e "'+referer+'" -d "'+tt+'" '
  #cmd=cmd+'-e "'+referer+'" '
  cmd=cmd+"-L '"+url1+"'"+'"]'
  #cmd=cmd+'-L "'+xx+'" '

  try_=1
  while(try_):
    proc=subprocess.Popen(cmd,
shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    s,err=proc.communicate()
    s=s.strip()
    err=err.strip()

    if(err==0):
      try_=''

.
.
.

the cmd is generated to be:
cmd=[r" curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
6.1; Trident/5.0; yie8)' --cookie-jar
/crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
--cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
   -L 'http://www6.austincc.edu/schedule/index.php?op=browse&opclass=ViewSched&term=216F000&disciplineid=PCACC&yr=2016&ct=CC'"]

test code hangs, ctrl-C generates the following:
^CTraceback (most recent call last):
  File "/crawl_tmp/austinccFetch_cloud_test.py", line 3363, in <module>
    ret=fetchClassSectionFacultyPage(a)
  File "/crawl_tmp/austinccFetch_cloud_test.py", line 978, in
fetchClassSectionFacultyPage
    (s,err)=proc.communicate()
  File "/usr/lib64/python2.6/subprocess.py", line 732, in communicate
    stdout, stderr = self._communicate(input, endtime)
  File "/usr/lib64/python2.6/subprocess.py", line 1328, in _communicate
    stdout, stderr = self._communicate_with_poll(input, endtime)
  File "/usr/lib64/python2.6/subprocess.py", line 1400, in
_communicate_with_poll
    ready = poller.poll(self._remaining_time(endtime))
KeyboardInterrupt

This works from the cmdline:
curl -sS -A 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1;
Trident/5.0; yie8)' --cookie-jar
/crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
--cookie /crawl_tmp/fetchContentDir/12f5e67c_156e_11e7_9c09_3a9e85f3c88e.lwp
   -L 'http://www6.austincc.edu/schedule/index.php?op=browse&opclass=ViewSched&term=216F000&disciplineid=PCACC&yr=2016&ct=CC'
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-03-30