Re: curl_multi - Reading response headers before reading the whole response body

From: Josh Handley via curl-library <curl-library_at_cool.haxx.se>
Date: Sat, 6 Feb 2021 07:42:34 -0500

Here is what I do:

In the header function (set with CURLOPT_HEADERFUNCTION) I read the headers until I get a blank header since a blank header indicates the end of the headers. There is an exception to this however since you can get an HTTP continue header followed by a blank header which is then followed by more headers. Once I get a blank header that is NOT directly following the continue I set a flag to indicatd I’m done reading headers. I also set that flag if the write function (set with CURLOPT_WRITEFUNCTION) gets called as this also indicates that the headers are complete. Once that flag is set, I call curl_easy_getinfo to the response code.

No guarantees that this works with every server but it works with the servers that my app uses.

Here is the relevant code:

    bool IsContinueHeader(const CString& header)
    {
        return header.GetLength() > 12 && header.Left(5) == "HTTP/" && header.Right(12) == "100 Continue";
    }

    size_t headerCallback(char* data, size_t size, size_t nmemb, CurlOperationState* state)
    {
        CString header = UTF8Convert::UTF8ToWide(std::string_view(data, size * nmemb));
        header.TrimRight(); // remove the trailing crlf
        if (header.IsEmpty()) {
            // An empty header signals end of headers, but not if it is after a "HTTP/1.1 100 Continue" which comes before reading input data
            if (!state->m_response_headers.empty()) {
                const CString& last_header = state->m_response_headers.at(state->m_response_headers.size() - 1);
                if (!IsContinueHeader(last_header))
                    state->m_finished_reading_headers = true;
            }
        }
        else {
            state->m_response_headers.push_back(header);
        }
        return size * nmemb;
    }


bool CurlHttpConnection::RunLoop()
{
    int numfds;
    CURLMcode mc = curl_multi_wait(m_multi_handle, NULL, 0, 1000, &numfds);
    if (mc != CURLM_OK)
        throw SyncException(L"Network error");

    if (!numfds)
        Sleep(100);

    int still_running = 0;
    curl_multi_perform(m_multi_handle, &still_running);

    CheckForErrors();

    return still_running > 0;
}

HttpResponse CurlHttpConnection::Request(const HttpRequest& request)
{
    Cleanup();
    m_state = std::make_unique<CurlOperationState>(request, m_listener);
    SetupEasyHandle();
    curl_multi_add_handle(m_multi_handle, m_state->m_easy_handle);

    int still_running;
    curl_multi_perform(m_multi_handle, &still_running);

    while (!m_state->m_finished_reading_headers) {
        RunLoop();
    }

    int http_status;
    curl_easy_getinfo(m_state->m_easy_handle, CURLINFO_RESPONSE_CODE, &http_status);
    HttpResponse response(http_status, m_state->m_response_headers);

> On Feb 6, 2021, at 1:22 AM, Idan Freiberg via curl-library <curl-library_at_cool.haxx.se> wrote:
>
> Hi curl developers,
>
> I'm using the multi-stack interface in order to read a large response body in chunk, with the help of the CURLOPT_WRITEFUNCTION and some code similar to the one in fcurl project.
>
> I wonder if its possible to fetch info like response code, cookies, response headers before the running easy handle is done (which means all the response body was read).
>
> Technically reading the response headers should be possible as they are the first thing received. The thing is i'm not sure how to check they are fully received after a curl_multi_perform() call?
> Some times when using HTTP auth, you will receive the subsequent 401 response code instead of the actual response code returns after authentication.
>
> Any suggestions?
>
> Thanks
>
> --
> Idan Freiberg
> Mobile: +972-52-2925213
> -------------------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html



-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-02-06