On Tue, Mar 16, 2010 at 3:27 PM, Demetrius Michael <dem_z_at_mac.com> wrote:

> Sorry for top posting, it's just the entire thread's been top posted. I
> also would like to know the answer for this.
>
> Because I'm very, *very* stuck.
>
> I have a phpbb forum, and every day I post the same material (the actual
> content changes, but the routine to get the content is the same).
> I've computerized the routine; now I'm trying computerize the* login >
> start thread > post > log ou*t bit.
>
> To login is easy, the session id is jammed into the cookie. Anything w/
> posting has a lot of hidden variables though.
>
> So what I've done is this:
>
> 1. $ curl - b cookies.txt - c cookies.txt URLTOPOST > savedfile1
> 2. parse the save file for hidden forms (in this case the fields turned out
> to be: "*topic_cur_post_id=24718", "lastclick=1268762819",
> "creation_time=1268762819",
> "form_token=11d7810297885073332758c13ca11de86af52cc1"*)
> 3. Reformat for next curl command.
> 4. $ curl -b cookies.txt -c cookies.txt -F 'post=Submit' -F 'message=
> this is my test message ' -F 'disable_bbcode=0' -F 'disable_smilies=0' -F
> 'attach_sig=1' -F 'topictype=0'-F 'topic_cur_post_id=24718'-F
> 'lastclick=1268762819'-F 'creation_time=1268762819'-F
> 'form_token=11d7810297885073332758c13ca11de86af52cc1' URLTOPOST > savedfile2
>
> Ran #4, and the forum barked *The submitted form was invalid. Try
> submitting again. *
>
> When looking at the source code, the hidden variables changed!
> ("topic_cur_post_id=24718", "lastclick=1268762820",
> "creation_time=1268762821", "form_token=2d4945d0c0e392d41240935848d6e9780726108d")
>
> Is there any way for curl to get the content, without killing the session,
> then POST after I've parsed for the hidden variables? Because in this case,
> it seems I get the content twice (giving me two different hidden fields)....
> Or let me rephrase, is it possible to parse for those known hidden fields
> w/i the first command line?
>

Do you really need to do that kind of form post?? I've screen-scraped a lot
of web pages, including logging into 3 major travel sites to check flights
and car rental availability, and I've never used the "-F" option.

I think the server isn't recognising what you're posting, or there's
something missing, so the server generates a new page so you can have
another go. For example, creation_time is the time in
seconds-since-the-epoch. Presumably that's when the page was generated to
send to you. It'll have a new value every time. And last_click is one
second earlier, which I guess records the time the server thinks you clicked
the link.

I would do the post like this:

    curl -b cookies.txt -c cookies.txt \
       -d 'post=Submit' \
       --data-urlencode 'message= this is my test message ' \
       -d 'disable_bbcode=0' \
       -d 'disable_smilies=0' \
       -d 'attach_sig=1' \
       -d 'topictype=0' \
       -d 'topic_cur_post_id=24718' \
       -d 'lastclick=1268762819' \
       -d 'creation_time=1268762819' \
       -d 'form_token=11d7810297885073332758c13ca11de86af52cc1' \
       URLTOPOST > savedfile2

Do you have access to the server's log to see what error messages it might
be recording??

Ralph Mitchell

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-03-17