Maybe build with debug symbols present so that the stack trace offers more details? If not, we really can't do much unless you detail how to reproduce this. problem.

Maybe build with debug symbols present so that the stack trace offers more details? If not, we really can't do much unless you detail how to reproduce this. problem.

Fully understand @bagder, thank for the tip.

I will hunt a way to reproduce the problem, could of course be me doing something wrong as well.

No news. Let's reopen if we can reproduce or otherwise act on it.

Hello,

I have a dozen of reproductions of that stacktrace. Below stacktrace based on this commit.
v1.41.0 of nghttp2 is in use.

Looking onto code, it corresponds. Where crash happens on second line:

  stream->closed = TRUE;
  stream->error = error_code;

Stacktrace:

0x000000555edb5768 ( -http2.c:1061)		on_stream_close
0x000000555ede0e88 ( -nghttp2_session.c:1196)		nghttp2_session_close_stream
0x000000555ede1a0c	( -nghttp2_session.c:2975)		nghttp2_session_mem_send_internal
0x000000555ede1dd8 ( + 0x00c22dd8)		nghttp2_session_send
0x000000555edb4a7c	( -http2.c:1713)		h2_session_send
0x000000555edb496c	( + 0x00bf596c)		h2_process_pending_input
0x000000555edb4cd8 ( -http2.c:1588)		http2_handle_stream_close
0x000000555edb3c40	( -http2.c)		cf_h2_recv
0x000000555edc7350	( + 0x00c08350)		Curl_read
0x000000555edcd5e0	( -transfer.c:461)		Curl_readwrite
0x000000555edc3744	( -multi.c:2443)		multi_runsingle
0x000000555edc3060	( + 0x00c04060)		curl_multi_perform

Hope it would help with resolving the issue, and it would be nice to reopen :)

Which version exactly are you building with? On tag curl-8_0_1 the code is at other line numbers.

I spot no less than 37 individual commits to lib/http2.c since 8.0.1 in git. It would make a lot of sense to first test this problem with curl 8.2.0 or current git since it is likely that we already fixed or at least changed this issue.

Thank you for fast responses!

Which version exactly are you building with? On tag curl-8_0_1 the code is at other line numbers.

Head was at f7a3fae for the stacktrace. I the left link to make it easy to looks at the tree snapshot at the moment and check sources.

I spot no less than 37 individual commits to lib/http2.c since 8.0.1 in git. It would make a lot of sense to first test this problem with curl 8.2.0 or current git since it is likely that we already fixed or at least changed this issue.

It definitelly might be a case. But what was curious for me, is that in the code below, if consider everything executed in order, assigning something to 'closed' does not cause violation. I was thinking, that may be a clue for someone deep into the code.

stream->closed = TRUE;
stream->error = error_code;

Checked other crashes, and found that there are some for very old libcurl.so version 7.70.0.

0x00000000f70b8844	(libcurl.so -http2.c:809)		on_stream_close
0x00000000f6eebd63	(libnghttp2.so -nghttp2_session.c:1175)		nghttp2_session_close_stream
0x00000000f6eecff9	(libnghttp2.so -nghttp2_session.c:2941)		nghttp2_session_mem_send_internal
0x00000000f6eed489	(libnghttp2.so -nghttp2_session.c:3209)		nghttp2_session_send
0x00000000f70b7f7b	(libcurl.so -http2.c:1506)		h2_session_send
0x00000000f70b7b63	(libcurl.so -http2.c:1856)		http2_send
0x00000000f70c29e5	(libcurl.so -sendf.c:349)		Curl_write
0x00000000f70c6ce9	(libcurl.so -transfer.c:1139)		Curl_readwrite
0x00000000f70bec7d	(libcurl.so -multi.c:2095)		multi_runsingle
0x00000000f70bf91b	(libcurl.so -multi.c:2833)		multi_socket
0x00000000f70bf99b	(libcurl.so -multi.c:2956)		curl_multi_socket_action

Downloaded zipped sources, and lookc like it's quite the same line:

stream->closed = TRUE;

So, it might exist for years already.

So, it might exist for years already.

Sure, and we might already have fixed it. Why don't you start with checking if this is already fixed?

I do not think you are debugging what you think you are debugging. That code originally mentioned does not exist in 7.70.0.

So, it might exist for years already.

Sure, and we might already have fixed it. Why don't you start with checking if this is already fixed?

I plan to test latest nghttp2 and curl release, but currently far away from a proper computer so it will be a couple weeks before I can get a proper look at things.

So, it might exist for years already.

Sure, and we might already have fixed it. Why don't you start with checking if this is already fixed?

Sorry, It's not something I may reproduce easily, rather collect bugrepors from devices in a field. After some time I'd see if it's fixed.
My hope was that if you see exact place the issue happens, you may definitivelly say it's fixed or not, but if it's not a case, also fine.

Ah, ok. Yeah that makes it more complicated-

Since we believe this is fixed, I will close this now. If you think otherwise after having had a chance to try this out (for a while), just let us know and we come back and continue.

Since we believe this is fixed, I will close this now. If you think otherwise after having had a chance to try this out (for a while), just let us know and we come back and continue.

FYI I am running cURL release 8.2.1 along with nghttp2 release 1.55.1 now, will update bug in some days about current status.

Still see this with cURL 8.2.1 and nghttp2 1.55.1, reverting cURL to 7.86.0 works around the issue.

Backtrace is slightly different, but I would state still similar. Sadly I have not been able to trigger this behavior through testing, but see it from embedded devices running in the field.

#0  0x0000007f782ac19c in (/lib/libcurl.so.4.8.0 @ 0x0003c19c) on_stream_close + 0x3c
#1  0x0000007f7726bde4 in (/lib/libnghttp2.so.14.24.3 @ 0x0000bde4) nghttp2_session_close_stream + 0x84
#2  0x0000007f7726ffe8 in (/lib/libnghttp2.so.14.24.3 @ 0x0000ffe8) nghttp2_session_mem_send_internal + 0x218
#3  0x0000007f77271108 in (/lib/libnghttp2.so.14.24.3 @ 0x00011108) nghttp2_session_send + 0x78
#4  0x0000007f782aae84 in (/lib/libcurl.so.4.8.0 @ 0x0003ae84) h2_progress_egress + 0xc4
#5  0x0000007f782ac304 in (/lib/libcurl.so.4.8.0 @ 0x0003c304) cf_h2_recv + 0xa4
#6  0x0000007f782c329c in (/lib/libcurl.so.4.8.0 @ 0x0005329c) Curl_read + 0x5c
#7  0x0000007f782d26f8 in (/lib/libcurl.so.4.8.0 @ 0x000626f8) Curl_readwrite + 0x778
#8  0x0000007f782bb274 in (/lib/libcurl.so.4.8.0 @ 0x0004b274) multi_runsingle + 0x974
#9  0x0000007f782bc4cc in (/lib/libcurl.so.4.8.0 @ 0x0004c4cc) curl_multi_perform + 0xfc

Does your code pause or resume connections? Does it call libcurl functions from more than one thread? Can you please give us some idea of how you use libcurl even if you don't have a self contained repro? Are you saving any debug logs from these devices?

Here is a simple program that sets environment variable CURL_DEBUG=http/2 and uses CURLOPT_DEBUGFUNCTION to record the verbose information to debug.log, including extra info for http/2 in curl debug builds.

#include <stdio.h>
#include <stdlib.h>
#include <curl/curl.h>

int debug_callback(CURL *handle, curl_infotype type, char *data, size_t size,
                   void *clientp)
{
  if(type == CURLINFO_TEXT && size) {
    fprintf((FILE *)clientp, "== Info [%p]: %.*s", handle, (int)size, data);
  }
  return 0;
}

static char linebuf[4096 + 1];

int main()
{
  CURL *curl = NULL;
  CURLcode res = CURLE_FAILED_INIT;
  FILE *fp = NULL;
  const char *logname = "debug.log";

  if(putenv("CURL_DEBUG=http/2")) {
    fprintf(stderr, "Error: putenv failed to set CURL_DEBUG\n");
    goto error;
  }

  fp = fopen(logname, "a");
  if(!fp) {
    fprintf(stderr, "Error: fopen failed to open %s\n", logname);
    goto error;
  }

#ifdef WIN32
  /* Set file buffer mode to unbuffered.
     Windows doesn't support line buffering. */
  setvbuf(fp, NULL, _IONBF, 0);
#else
  /* Set file buffer mode to line buffering. */
  setvbuf(fp, linebuf, _IOLBF, sizeof(linebuf));
#endif

  if(curl_global_init(CURL_GLOBAL_ALL)) {
    fprintf(stderr, "Error: curl_global_init failed\n");
    goto error;
  }

  curl = curl_easy_init();
  if(!curl) {
    fprintf(stderr, "Error: curl_easy_init failed\n");
    goto error;
  }
  
  fprintf(fp, "\n!!!!!!!!!!!!!!!!!\n!!!!! START !!!!!\n!!!!!!!!!!!!!!!!!\n");

  curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, debug_callback);
  curl_easy_setopt(curl, CURLOPT_DEBUGDATA, fp);
  curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  curl_easy_setopt(curl, CURLOPT_CAINFO, "curl-ca-bundle.crt");

  curl_easy_setopt(curl, CURLOPT_URL, "https://curl.se");

  res = curl_easy_perform(curl);

  if(res != CURLE_OK) {
    fprintf(fp, "libcurl: (%d) %s\n", res, curl_easy_strerror(res));
    if(stderr != fp)
      fprintf(stderr, "libcurl: (%d) %s\n", res, curl_easy_strerror(res));
  }

  goto cleanup;
error:
  res = CURLE_FAILED_INIT;
cleanup:
  if(curl)
    curl_easy_cleanup(curl);
  curl_global_cleanup();
  if(fp)
    fclose(fp);
  return (res == CURLE_OK ? 0 : 1);
}

Would it be possible to get the line number of the crash? I fail to see where this is happening in 8.1.2.

We're seeing this in the wild with a beta that includes 8.0.1. Unfortunately we weren't able to update to 8.2.1 so the code referenced will be a bit out of date. Hopefully there's enough that might give you all some hints on what's happening.

https://github.com/curl/curl/blob/b16d1fa8ee567b52c09a0f89940b07d8491b881d/lib/http2.c#L1049C22-L1060

An engineer looking at the crash dumps see that the address indicated for Curl_easy *data_s had been discarded and the address retrieved from data_s->p.http is invalid. Writing TRUE to stream->closed triggered the crash.

They went further into the crash dumps and saw that when it happened the value of error_code passed to on_stream_close corresponded to NGHTTP2_STREAM_CLOSED for 317 out of 318 reports. The single outlier had the register holding that value set to NGHTTP2_REFUSED_STREAM.

We've yet to create a reduced case that triggers the issue. If we get that we'll follow up back here. If there's any more information needed I'll help however I can. Thanks in advance!

@bagder @icing is ☝️ of any use to you all?

Unfortunately, I was not able to make any progress on this. If we do not find a way to reproduce with current curl, there is nothing I can think of.

I don't see how we can act on this without a reproducible, without it happening on a recent version and without anyone digging in to analyze and figure out what happens. Closing.

If something of this changes substantially, we can always reopen again.

https://gist.github.com/yunanchangCorp/c4a7939fd39ca4a5f0d46a95cb693f87

here are some logs from a failed request (remove company related information), I'm using libcurl, tried setting trace using curl_global_trace, not sure if it's the correct way.

Also note that I also changed some pare of code to make it log more (like making mstate log completed too).

Also found that the filter got a CF_CTRL_DATA_DONE before CF_CTRL_DATA_DETACH, so it actually had a stream the first time, sorry for being misleading.

@yunanchangCorp you are talking about a failed request or also about the crash in on_stream_close(). I want to make sure that we are not mixing things up here...thanks.

yes I'm not mixing things up, a failed request get executed later and trying to close the stream results in a crash (since that request already closed), that's what I have observed.

I'm suspecting nghttp2_submit_rst_stream is not cleaning up ob_syn in nghttp2, adding more log and wait for crash.

Made #12562 with some small changes in clearing stream user data and checks on the seen user data in on_stream_close. Maybe that helps you to track down the fault. I am still unable to reproduce locally a crash.

I may have got the problem, please hear my story and see if it makes sense...

I have almost zero HTTP2 knowledge before solving this bug, so may make some mistake in detail, please don't hesitate to correct me.

So the story:

For a request that is timed out. It is closed through CF_CTRL_DATA_DONE. Which will go through cf_h2_cntrl and then http2_data_done.

Inside http2_data_done, we call nghttp2_submit_rst_stream to say we are not doing this frame anymore.

Inside nghttp2, nghttp2_submit_rst_stream in turns called nghttp2_session_add_rst_stream,
which, marks the related outbound item to be cancelled, by finding it though stream id.

Then it returned, curl returned, too, and mark the handle as complete, the user closes the easy handle.

--------------------------------------------- 15 minutes later --------------------------------------------------
(15 min in my case, may be longer or shorter)

In nghttp2, the same session goes on, and sometime in the future it will get the previously cancelled outbound item.
here, and try to process that.

But it's not skipping the item, instead it goes into session_prep_frame and starts to open a stream, and until after the stream is opened, it start checking if the item is cancelled, returning a error.

But the stream is opened now, it must close the stream. And to close the stream, it must call the callback which is curl's on_stream_close.

And finally inside on_stream_close, we tried to access the easy handle that is closed 15 minutes ago, so it crashed.

I currently feel it's nghttp2's problem for trying to open and close stream for a cancelled item.
If you agree with this, I will open an issue for them and see how to fix that (not really sure why they don't check cancel before opening the stream).

Also I will try to create a minimal example based on this story, though I don't know how to control HTTP2 frames precisely so might or might not succeed...

I has a app use libcurl, recent get error from windows.

I can not get traceback. (with some error like this, after I upgrade llvm toolchain)

 attribute, but range extraction failed (missing or invalid range list table), please file a bug and attach the file at the start of this error message

runManually indexing DWARF for test.exe...  
error: unknown errorindexing DWARF for

the last logs is from on_curl_data callback.

test with #12562

  * frame #0: 0x00007ff7bad3768c aplugin.exe`Curl_llist_insert_next
    frame #1: 0x00007ff7bab1b2aa aplugin.exe`multi_runsingle
    frame #2: 0x00007ff7bab1d5b2 aplugin.exe`multi_socket
    frame #3: 0x00007ff7bab1db0d aplugin.exe`curl_multi_socket_action
    frame #4: 0x00007ff7ba955f3e aplugin.exe`curl_perform
    frame #5: 0x00007ff7baae9387 aplugin.exe`uv__process_poll_req
    frame #6: 0x00007ff7baabd4d3 aplugin.exe`uv__process_reqs
    frame #7: 0x00007ff7baabc78b aplugin.exe`uv_run

I think you are onto something here. I opened nghttp2/nghttp2#2018 at nghttp2 to get their opinion on my theory on what is happening. Let's see how they respond.

In the meantime, you could check in your local setup if calling nghttp2_session_set_stream_user_data(ctx->h2, stream->id, NULL); at the beginning of http2_data_done() before nghttp2_submit_rst_stream() does solve the issue for you.

ok, adding right before nghttp2_submit_rst_stream and waiting for crash (hope not).

    if(MSTATE_COMPLETED == data->mstate) {
      if(data->set.fmultidone) {
        /* signal via callback instead */
        data->set.fmultidone(data, result);
      }
      else {
        /* now fill in the Curl_message with this info */
        msg = &data->msg;

        msg->extmsg.msg = CURLMSG_DONE;
        msg->extmsg.easy_handle = data;
        msg->extmsg.data.result = result;

        multi_addmsg(multi, msg);
        DEBUGASSERT(!data->conn);
      }
      multistate(data, MSTATE_MSGSENT);

      /* add this handle to the list of msgsent handles */
	  printf("[%s:%d] data=%p\n", __FILE__, __LINE__, data);
      Curl_llist_insert_next(&multi->msgsent, multi->msgsent.tail, data,
                             &data->connect_queue);
      /* unlink from the main list */
      unlink_easy(multi, data);
      return CURLM_OK;
    }
  } while((rc == CURLM_CALL_MULTI_PERFORM) || multi_ischanged(multi, FALSE));

after upgrade to 8.50 version (with 12562 patch ), I get crash at this call from lldb:

 thread #1, stop reason = Exception 0xc0000005 encountered at address 0x7ff6ab987b4c: Access violation reading location 0xffffffffffffffff
  * frame #0: 0x00007ff7bad3768c aplugin.exe`Curl_llist_insert_next
    frame #1: 0x00007ff7bab1b2aa aplugin.exe`multi_runsingle
    frame #2: 0x00007ff7bab1d5b2 aplugin.exe`multi_socket
    frame #3: 0x00007ff7bab1db0d aplugin.exe`curl_multi_socket_action
    frame #4: 0x00007ff7ba955f3e aplugin.exe`curl_perform
    frame #5: 0x00007ff7baae9387 aplugin.exe`uv__process_poll_req
    frame #6: 0x00007ff7baabd4d3 aplugin.exe`uv__process_reqs
    frame #7: 0x00007ff7baabc78b aplugin.exe`uv_run

this is come from windows. (other platfrom not upgraded)

any suggestion how can I trace this problem?

@calvin2021y please do not highjack other issues. This crash is happening elsewhere. Open a new issue instead. Thank you.

will do.

before I trace to this line (multi.c 2660 call Curl_llist_insert_next), I was think this is the same issue.

Curl_llist_insert_next

make sure you are following the thread safety guidelines

@icing after 16 hours there's no single crash (usually it will have 6~7). I think this solves the problem.

So the real story is the nghttp2 requires the user to take care of the user data before cancelling, rather than nghttp2 skip the cancelled item, right? And curl made some mistake so the user data is not correctly cleared.

@icing after 16 hours there's no single crash (usually it will have 6~7). I think this solves the problem.

Very good news! I'm going to merge #12562 soonish.

Pasing on some details from my testing as well. I use curl 8.5.0 release + change in #12562, and I have not seen any crashes yet. It has been running for 2 days, a bit limited amount of testing due to slow startup after the holidays.

Just to make sure @bagder, @egtvedt , is this merged in version 8.5.0? I am encountering the same issue, and I don't see the fix listed on the patch notes. Thanks!

Just to make sure @bagder, @egtvedt , is this merged in version 8.5.0? I am encountering the same issue, and I don't see the fix listed on the patch notes. Thanks!

No, this is not merged and released with version 8.5.0, you need to cherry pick commit 3538027 on top of the 8.5.0 release. IIRC it was a trivial cherry pick.

I would assume this will be part of 8.6.0 release, but it is completely up to @bagder to steer the release train (-:

All commits present in master will be part of the next release. This includes 3538027. Scheduled to ship on January 31, 2024. The tentative release notes for the next release is always present here.