curl-library
retrieving files by http won't give me the whole file !
Date: Mon, 15 Nov 2004 18:34:45 +0100
Hello everyone :)
I'm running into a problem...
I need to retrieve a bunch of files through http.
so i make as much threads as i need to and from them, i call libcurl as this:
extern "C"
{
void *pulls_one_url(void *urlstruct)
{
CURL *http_get_handle;
arguments *arg;
arg = (arguments*)urlstruct;
http_get_handle = curl_easy_init();
// Setting now the options for my handle
curl_easy_setopt(http_get_handle, CURLOPT_URL, (char*)arg->url);
curl_easy_setopt(http_get_handle, CURLOPT_PORT, 80);
curl_easy_setopt(http_get_handle, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(http_get_handle, CURLOPT_MAXREDIRS, 1);
curl_easy_setopt(http_get_handle, CURLOPT_HTTP_VERSION,
CURL_HTTP_VERSION_1_1);
curl_easy_setopt(http_get_handle, CURLOPT_WRITEFUNCTION, http_output);
curl_easy_setopt(http_get_handle, CURLOPT_FILE, arg->id);
curl_easy_perform(http_get_handle);
curl_easy_cleanup(http_get_handle);
return NULL;
}
}
extern "C"
{
int http_output(void *buf, size_t size, size_t nmemb, void *stream)
{
char *buffer = (char*)emalloc(size * nmemb + 1);
strncpy(buffer, (char*)buf, size * nmemb);
buffer[size * nmemb] = '\0';
zend_printf("size = %d, nmemb = %d strlen(buf) = %d\n<br>", size, nmemb,
strlen((char*)buf));
add_assoc_string(http_requests_return, (char*)stream, buffer, 0);
return 0;
}
}
So it works pretty well and the output generated is the following :
Querying 14 sites...
size = 1, nmemb = 2656 strlen(buf) = 2656
<br>size = 1, nmemb = 2394 strlen(buf) = 2394
<br>size = 1, nmemb = 2920 strlen(buf) = 2920
<br>size = 1, nmemb = 1169 strlen(buf) = 1169
<br>size = 1, nmemb = 1266 strlen(buf) = 1266
<br>size = 1, nmemb = 1156 strlen(buf) = 1156
<br>size = 1, nmemb = 1030 strlen(buf) = 1030
<br>size = 1, nmemb = 2498 strlen(buf) = 2498
<br>size = 1, nmemb = 2482 strlen(buf) = 2482
<br>size = 1, nmemb = 2478 strlen(buf) = 2478
<br>size = 1, nmemb = 2638 strlen(buf) = 2638
<br>size = 1, nmemb = 2498 strlen(buf) = 2498
<br>size = 1, nmemb = 2503 strlen(buf) = 2503
So i have a retrieve. but it's incomplete for example, all i get from
linux.com is the following:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
<TITLE>The Linux Home Page at Linux Online</TITLE>
<META name="description" content="Comprehensive information and resources
about the Linux Operating System.">
<META name="keywords" content="Linux, Linux Online, Torvalds, Linus, operating
system, UNIX, web, www, software, hardware, support, help, information,
resources, drivers, manual, documentation project, FAQ, miniHOWTO, howto,
documentation, Linux Homepage, Linux Home Page, Linus Torvalds, Redhat,
Slackware, Yggdrasil, Debian, LinuxPro, OpenLinux, FTP, download, projects,
application, usergroup, user group, mailing list, kernel">
<META NAME="ROBOTS" CONTENT="NOCACHE,NOARCHIVE">
</HEAD>
<!-- header -->
<BODY BGCOLOR="#F4F1BC">
<A HREF="/robot/index.html"><IMG SRC="/images/colour/yellow.gif" BORDER=0
WIDTH=1 HEIGHT=1 ALT=""></A><A HREF="mailto:trapper_at_linux.org?subject=DO NOT
SEND EMAIL HERE"><IMG SRC="/images/colour/clear.gif" BORDER=0 HEIGHT=1
WIDTH=1 ALT=""></A>
<TABLE BORDER=0 CELLPADDING=0 CELLSPACING=0 BGCOLOR="#A41F1F" WIDTH=780>
<TR>
<TD VALIGN=TOP COLSPAN=2>
<TABLE BORDER=0 CELLPADDING=0 CELLSP
And nothing else... I tried many things but nothing seems to help. Anyone has
an idea ? thank you
Received on 2004-11-15