curl / Mailing Lists / curl-users / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: Curl Help?

From: bruce via curl-users <curl-users_at_cool.haxx.se>
Date: Mon, 18 Jan 2021 17:08:12 -0500

Hi Ray.

Thanks for the reply!!

I had thought of doing a selenium/py piece of code to drive the
returned content/html.

The weird thing. Other sites that have this similar layout with the
"Term" icon, allow a simple "curl" process to generate the data.
Welcome to the world of different versions/behind the scenes
functions...
If you created a test chunk of code could you fire it to me?

I hadn't seen the 2nd level html call based on the term #.

thanks..

On Sun, Jan 17, 2021 at 6:08 PM Ray Satiro via curl-users
<curl-users_at_cool.haxx.se> wrote:
>
> On 1/17/2021 11:13 AM, bruce via curl-users wrote:
> > The test site is a Psoft site:
> > https://cmsweb.pscs.calpoly.edu/psc/CSLOPRD/EMPLOYEE/SA/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
> > https://cmsweb.pscs.calpoly.edu/psp/CSLOPRD/EMPLOYEE/SA/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
> >
> > I've been successful in using curl to "simulate" and drive testing a
> > few other similar sites. In this situation, the site has a "term"
> > element that's being driven by a:
> > javascript:pAction_win0(document.win0,'SLO_SS_DERIVED_STRM$prompt')
> >
> > In order to try to craft the "curl" for this. I used the following steps:
> > -Fire up Firefox v78
> > set the developer tools/Network tools
> > initiate the url;
> > https://cmsweb.pscs.calpoly.edu/psc/CSLOPRD/EMPLOYEE/SA/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
> >
> > select the "calpoly" as the institution
> > clear the network traffic section
> > select the "term" icon
> > (this generates the 'popup/frame' for the term list)
>
>
> It is a web application that is implemented heavily in javascript. You
> could use a web debugger like fiddler in an attempt to view and rebuild
> the commands but I doubt that's going to be tenable long term. It looks
> like in the post that SLO_SS_DERIVED_STRM is set to the terminal number
> and then after that a get request is made to the same URL and replies
> with more html and buried in there is the actual response "No matching
> values were found".
>
> IMO curl is not the right tool for this, look into headless browser
> webscraping.
>
>
> -----------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
> Etiquette: https://curl.haxx.se/mail/etiquette.html
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2021-01-18