Hi Daniel,

2012/6/19 Daniel Stenberg <daniel_at_haxx.se>:
> On Tue, 19 Jun 2012, Marc Hoersken wrote:
>
>> I gave this another thought and figured out that we also still need to be
>> able to handle more data internally than BUFSIZE. For the same reason we
>> actually need the internal buffers, which I explained earlier in this
>> thread, we also need to be able to handle more data being received and
>> decrypted internally. Especially since length(encrypted) !=
>> length(decrypted) as SSL/TLS supports compression and of course the
>> encrypted data stream contrains SSL/TLS packet information.
>
>
> Ah, good thinking. Your patch was merged and pushed just now!

thanks! Actually there is one more thing we can discuss. In the
current implementation the increased buffer sizes are tried to be
reduced to the initial buffer size after each and every receive
operation. On the one hand this means that libcurl would free up
memory after some high memory usage scenario which could be very
important for long-term SSL/TLS connections. On the other hand this
means again that we have some realloc calls and in this case they
might result in very odd buffer sizes. They won't produce
fragmentation in the first place, since they are decreasing the buffer
size, but in future receive operations the same step size is added to
this now odd number.

The current rule for buffer reduction is the following (in pseudocode):

length(buffer) > initial_size
  new_length = max(initial_size, used_length)
  buffer = realloc(buffer, new_length)

This could result in the following scenario:

1. initial size: 16384
2. receive operation requiring more data to be able to decrypt SSL/TLS
packet, new size: 16384 + 8192 = 24576
3. only 8000 bytes were requested by the user, left-over reduced size,
greater than initial: 24576 - 8000 = 16576
4. next receive operation requires even more data to be stored, new
size: 16576 + 8192 = 24768

This might not be such a bad example, but I just wanted to show how
the reduction code actually makes curl use buffer sizes which may not
a power of 2 or be equal to the optimal memory allocation size of the
system.

Therefore we need to decide whether we want curl to free up memory as
early as possible or not. We could even try to optimize the reduction
code to only realloc to multiples of the step size.

What do you think?

Just for experimentation, I have attached a patch that removes the
current buffer reduction code and shows you were it actually is in the
code.

Best regards,
Marc

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html