On Thu, Jul 01, 2010 at 12:26:04AM -0600, Donald Boissonneault wrote:
> I wrote the following function, however due to slow dns resolving I need
> to make it so I can pull like 10 urls at a time or more. Would like to
> be able to change that depending on connection speed. Below is the code.
> I have been trying to do this with multi-urls for over 3 months. I am
> very new to c programming, but am making very good progress. However,
> this multi-tasking is really confusing me. How could I turn the below
> code into something so it would load the urls IP, HTTP Response code,
> URL re-direct address and the html body into a an array. I know the
> purpose of programming is to write your own code, but I am totally lost
> here. I do not start school till fall and would like to continue working
> on my program. If you can help I would be very happy.
> Thank you,
> Don
>
>
>
> #include <curl/curl.h>
> #include <curl/types.h>
> #include <curl/easy.h>
>
> using namespace std;
>
> #include "MakeFile.h"
>
> char* memory;
> size_t UrlConnectionHtmlBody_size;
>
> static size_t write_data(char *ptr, size_t size, size_t nmemb, void
> *stream);
>
> static size_t write_data(char *ptr, size_t size, size_t nmemb, void
> *stream)
> {
> size_t mem;
> //increase the memory buffer size being held
> mem = size * nmemb;
> // set the sizt_t to know how long the char* is
> UrlConnectionHtmlBody_size += mem;
> if (mem>0)
> {
> memory = (char*)realloc(memory, UrlConnectionHtmlBody_size);
> }
> else
> {
> memory = (char*) malloc(UrlConnectionHtmlBody_size);
> }
> // store the data
> if (mem)
> {
> memcpy(&(memory[UrlConnectionHtmlBody_size-mem]), ptr, mem);
> };
> return mem;
> };
>
> void UrlGetInfo(char* VarUrlToGet)
> {
> const char *p = VarUrlToGet; // get const char * representation
> printf("Get Url %s\n",VarUrlToGet);
> //Reset string varable for getting data
> memory = NULL;
> UrlConnectionHtmlBody_size = 0;
> CURL *curl_handle;
> CURLcode res;
> curl_global_init(CURL_GLOBAL_ALL);
>
> /* init the curl session */
> curl_handle = curl_easy_init();
>
> /* set URL to get */
> curl_easy_setopt(curl_handle, CURLOPT_URL, p);
>
> /* no progress meter please */
> curl_easy_setopt(curl_handle, CURLOPT_NOPROGRESS, 1L);
>
> /* send all data to this function */
> curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, write_data);
>
> /*
> * Notice here that if you want the actual data sent anywhere else
> but
> * stdout, you should consider using the CURLOPT_WRITEDATA option.
> */
>
> /* get it! */
> res = curl_easy_perform(curl_handle);
> if(CURLE_OK == res)
> {
> //set the information for the body to the UrlInfo
>
> // pointer Redirect Site
> char *ra;
> char *ip;
> long HttpResponse;
> /* get the CURLINFO_HTTP_CONNECTCODE*/
> res = curl_easy_getinfo(curl_handle, CURLINFO_RESPONSE_CODE,
> &HttpResponse);
> /* ask for the ReDirectAddress*/
> res = curl_easy_getinfo(curl_handle, CURLINFO_REDIRECT_URL,
> &ra);
> if((CURLE_OK == res) && ra)
> {
> };
> // Get the IP address for the web site
> res = curl_easy_getinfo(curl_handle, CURLINFO_PRIMARY_IP, &ip);
> if((CURLE_OK == res) && ip)
> {
> };
> }
> free (memory);
> /* cleanup curl stuff */
> curl_easy_cleanup(curl_handle);
> };

I'm not certain what exactly you're trying to accomplish with this code,
but there's a good example of how to download 10 requests at a time
on curl's website:

http://curl.haxx.se/libcurl/c/10-at-a-time.html

You'll need to use the multi interface if you'd like to perform multiple
downloads using a single thread of control.

-j

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-07-01