Re: Feature request: new callback to report http chunks boundaries

From: Henrik Holst via curl-library <curl-library_at_cool.haxx.se>
Date: Mon, 21 Jun 2021 01:33:49 +0200

It depends on which JSON parser that you use, e.g YAJL have a setting
yajl_allow_multiple_values which allows you to constantly feed data into
the parser and it will still handle each JSON object as separate objects.
And I bet that other parsers have similar functionality.

To get out of the error cycle, I would upon the first error start to look
for whatever your server has as start of object, e.g your server might
start each an every object with something like:

{
  "kind": "

So after experiencing a parser error I would simply append the received
data in a buffer until say strstr on the buffer yielded a non NULL reply on
that string and then simply feed the returned pointer from strstr to the
JSON parser and start all over again. This is how I have to do it with both
JSON and XML objects being pushed over a regular tcp/ip stream where there
are no boundary information at all to work with.

/HH

Den sön 20 juni 2021 kl 23:20 skrev Daniel F via curl-library <
curl-library_at_cool.haxx.se>:

> W dniu 2021-06-20 22:22, Dan Fandrich via curl-library napisał(a):
> > On Sun, Jun 20, 2021 at 05:34:00PM +0200, Daniel F via curl-library
> > wrote:
> >> I am writing app which will receive updates (in JSON format) from
> >> server.
> >> Server uses HTTP chunked transfer encoding. I also use TLS to encrypt
> >> transmission. If I read curl code in lib/http_chunks.c correctly,
> >> library
> >> guarantees that contents of multiple chunks is not mixed when callback
> >> registered with CURLOPT_WRITEFUNCTION option is called. However
> >> opposite is
> >> not true, write callback may be called multiple times for one chunk.
> >> This
> >> means that I need some extra callback in order to detect chunk
> >> boundaries,
> >> but I do not see such callback in the code. I think of two possible
> >> callbacks: one called before first part of a chunk is delivered to the
> >> app
> >> (it may have chunk size as a param), and another called after last
> >> part of
> >> chunk is delivered to the app. Any of them would be fine for me. Could
> >> you
> >> add it?
> >
> > HTTP chunking is part of HTTP and should be transparent to the
> > application. It
> > sounds like what you're doing is a layer violation. What is the purpose
> > of
> > knowing chunking boundaries?
> > -------------------------------------------------------------------
> > Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> > Etiquette: https://curl.se/mail/etiquette.html
>
> My app needs to watch updates done to some Kubernetes objects. Every
> update is sent as a new HTTP chunk with JSON object inside. I need to
> know boundaries to know when I can start parsing received JSON.
> Currently I do not know this, so JSON parsing would report error if only
> part of chunk is received. I could ignore this error, assume that chunk
> is not complete, then later append next part of received data to
> previous one and try to parse again. This usually would work, except for
> case when server would send invalid JSON - in such case code would
> generate parse error again and again. So I look for better way to handle
> this.
> -------------------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html


-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-06-21