Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: HSTS cache cap allows eviction of security entries
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Timothe Litt <litt_at_acm.org>
Date: Wed, 1 Apr 2026 18:00:51 -0400
You would always store (new, updated) records on disk - which is
essentially unlimited. [If an adversary can manage to fill your disk
with HSTS data, perhaps he deserves to win. You have bigger problems.]
Initially, you read from disk to get the HSTS record(s) each time you
encounter a site.
You do not load the entire disk file into memory.
Now sites without HSTS incur a disk read, but there's no record.
Cache the result (in memory) either way. The next reference to such a
site won't do a pointless read.
Most sites don't have HSTS records, so a negative result is the common
case..
Make the *memory* cache LRU & limit its size. Frequently used sites
will hit in memory as before. First access will incur the disk penalty,
which you can optionally reduce by loading/saving the *memory* cache in
another file - if access patterns make it worthwhile.
This makes the HSTS data store unlimited, and bounds the memory
footprint no matter how many sites an instance of libcurl visits. There
is a performance tradeoff - depending on how effective your cache is.
If you think you'll have "zillions" of sites, you can index, shard, or
sort the file, or move frequent records to the top at rundown.
But a simple linear search of the disk file probably handles all but
extreme use cases. A database scales to 100s of millions of sites for
such cases - if you think that a a long-lived crawler is going to access
that many HSTS sites. I'm not inclined to worry about that - but you
always go your own way.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
On 01-Apr-26 17:14, Daniel Stenberg wrote:
> On Wed, 1 Apr 2026, Timothe Litt via curl-library wrote:
>
>> Store the HSTS list on disk (persisting it is good);
>
> That's why libcurl offers that in its API. It still needs to be able
> to function with the cache in memory.
>
>> you can use a memory cache of both positive [site -> HSTS records]
>> and negative [site -> 'has no HSTS'] entries - and limit its size.
>
> A typical user scenario has perhaps a handful of hostnames in the list
> that should be bumped to HTTPS. I don't see how adding negative info
> to this makes the data smaller.
>
>> Or let a database (e.g. SQLite) manage the list. You don't have to
>> invent your own.
>
> This is over-engineering terrority. SQLite is itself larger than the
> whole of libcurl. HSTS is a rather tiny edge feature. No one wants
> libcurl to explode in size and complexity just to support this.
>
Received on 2026-04-02
Date: Wed, 1 Apr 2026 18:00:51 -0400
You would always store (new, updated) records on disk - which is
essentially unlimited. [If an adversary can manage to fill your disk
with HSTS data, perhaps he deserves to win. You have bigger problems.]
Initially, you read from disk to get the HSTS record(s) each time you
encounter a site.
You do not load the entire disk file into memory.
Now sites without HSTS incur a disk read, but there's no record.
Cache the result (in memory) either way. The next reference to such a
site won't do a pointless read.
Most sites don't have HSTS records, so a negative result is the common
case..
Make the *memory* cache LRU & limit its size. Frequently used sites
will hit in memory as before. First access will incur the disk penalty,
which you can optionally reduce by loading/saving the *memory* cache in
another file - if access patterns make it worthwhile.
This makes the HSTS data store unlimited, and bounds the memory
footprint no matter how many sites an instance of libcurl visits. There
is a performance tradeoff - depending on how effective your cache is.
If you think you'll have "zillions" of sites, you can index, shard, or
sort the file, or move frequent records to the top at rundown.
But a simple linear search of the disk file probably handles all but
extreme use cases. A database scales to 100s of millions of sites for
such cases - if you think that a a long-lived crawler is going to access
that many HSTS sites. I'm not inclined to worry about that - but you
always go your own way.
Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.
On 01-Apr-26 17:14, Daniel Stenberg wrote:
> On Wed, 1 Apr 2026, Timothe Litt via curl-library wrote:
>
>> Store the HSTS list on disk (persisting it is good);
>
> That's why libcurl offers that in its API. It still needs to be able
> to function with the cache in memory.
>
>> you can use a memory cache of both positive [site -> HSTS records]
>> and negative [site -> 'has no HSTS'] entries - and limit its size.
>
> A typical user scenario has perhaps a handful of hostnames in the list
> that should be bumped to HTTPS. I don't see how adding negative info
> to this makes the data smaller.
>
>> Or let a database (e.g. SQLite) manage the list. You don't have to
>> invent your own.
>
> This is over-engineering terrority. SQLite is itself larger than the
> whole of libcurl. HSTS is a rather tiny edge feature. No one wants
> libcurl to explode in size and complexity just to support this.
>
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
- application/pgp-signature attachment: OpenPGP digital signature