introducing "curldown" for libcurl man page format #12730

bagder · 2024-01-17T13:04:01Z

overview

curldown is this new file format for libcurl man pages. It is markdown inspired with differences:

Each file has a set of leading headers with meta-data
Supports a small subset of markdown
Uses .md file extensions for editors/IDE/GitHub to treat them nicely
Generates man pages very similar to the previous ones
Generates man pages that still convert nicely to HTML on the website
Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)

tools:

cd2nroff: converts from curldown to nroff man page
nroff2cd: convert an (old) nroff man page to curldown
cdall: convert many nroff pages to curldown versions
cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

tasks

The tests that rely on the man pages to be present will be disabled for the CI jobs that use cmake on Appveyor for now.

scripts/nroff2cd

bagder · 2024-01-17T14:04:06Z

The CURLDOWN.md file within the PR is the embryo of documentation for the format and how to use it.

docs/CURLDOWN.md

bagder · 2024-01-18T23:58:55Z

An interesting side-effect of switching to .md is that all of a sudden, the proselint CI job got a lot of more text to complain about... 😆

bagder · 2024-01-22T21:38:43Z

I believe this is PR working cmake wise with @levitte's fixes but it still fails four tests on appveyor due to windows-related reasons, such as no shell expansion on wildcards and something else: 1140 1173 1177 1477

It will disable them on appveyor and move forward if that is the only remaining nit.

vszakats · 2024-01-22T22:04:37Z

scripts/cd2nroff

+    my $blankline = 0;
+    my $header = 0;
+
+    push @desc, ".\\\" generated by cd2nroff $cd2nroff from $f\n";


In a local test run (I used CMake) $f remains empty for all output files:

.\" generated by cd2nroff 0.1 from .TH CURLINFO_APPCONNECT_TIME 3 "December 14 2023" libcurl

what is the full command line used when cd2nroff is invoked?

Oh, I see why

diff --git a/docs/libcurl/CMakeLists.txt b/docs/libcurl/CMakeLists.txt index fc1283a53..0d52a7e56 100644 --- a/docs/libcurl/CMakeLists.txt +++ b/docs/libcurl/CMakeLists.txt @@ -34,11 +34,11 @@ function(add_manual_pages _listname) else() string(REPLACE ".3" ".md" _mdfile "${CMAKE_CURRENT_SOURCE_DIR}/${_file}") endif() add_custom_command(OUTPUT "${_rofffile}" - COMMAND ${PROJECT_SOURCE_DIR}/scripts/cd2nroff < ${_mdfile} > ${_rofffile} + COMMAND ${PROJECT_SOURCE_DIR}/scripts/cd2nroff ${_mdfile} > ${_rofffile} DEPENDS "${_mdfile}" VERBATIM ) endforeach() endfunction()

They look like this:

cd ./curl/_x64-win-ucrt-cmake-llvm-bld/docs/libcurl/opts && \ ./curl/scripts/cd2nroff \ < ./curl/docs/libcurl/opts/CURLINFO_APPCONNECT_TIME.md \ > ./curl/_x64-win-ucrt-cmake-llvm-bld/docs/libcurl/opts/CURLINFO_APPCONNECT_TIME.3

Right, my diff above changes that command line, to pass the file name instead < [filename] which is what made it blank there.

Thanks! That fixed it ...but now there is another issue: The filename embedded contains the whole (absolute) path of the source filename, which breaks reproducibility and may also leak local info. (This may be CMake-specific, as CMake loves absolute paths, causing this class of issues everywhere.)

.\" generated by cd2nroff 0.1 from /<home-dir>/<path-to>/curl/docs/libcurl/opts/CURLINFO_CONNECT_TIME_T.md .TH CURLINFO_CONNECT_TIME_T 3 "December 14 2023" libcurl

Perhaps the simplest fix is to omit this info from the generated output? Or strip all path from it?

It does not break reproducibility, you just have to rebuild with the same paths. But yes, I'll make the script cut off the dir name. I want the file name mentioned there to help users find the source.

Patch to drop the source filename:

diff --git a/scripts/cd2nroff b/scripts/cd2nroff index 08fbd768e..734cbf8f3 100755 --- a/scripts/cd2nroff +++ b/scripts/cd2nroff @@ -186,7 +186,7 @@ sub single { my $blankline = 0; my $header = 0; - push @desc, ".\\\" generated by cd2nroff $cd2nroff from $f\n"; + push @desc, ".\\\" generated by cd2nroff $cd2nroff\n"; push @desc, ".TH $title $section \"$date\" $source\n"; while(<$fh>) { $line++;

To strip path:

diff --git a/scripts/cd2nroff b/scripts/cd2nroff index 08fbd768e..d9938b6e9 100755 --- a/scripts/cd2nroff +++ b/scripts/cd2nroff @@ -30,6 +30,8 @@ Converts a curldown file to nroff (man page). =end comment =cut +use File::Basename; + my $cd2nroff = "0.1"; # to keep check my $dir; my $extension; @@ -186,7 +188,7 @@ sub single { my $blankline = 0; my $header = 0; - push @desc, ".\\\" generated by cd2nroff $cd2nroff from $f\n"; + push @desc, ".\\\" generated by cd2nroff $cd2nroff from " . basename($f) . "\n"; push @desc, ".TH $title $section \"$date\" $source\n"; while(<$fh>) { $line++;

vszakats · 2024-01-22T22:14:48Z

Adding a CMake option to disable building docs:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 70ca457a9..18d8d085b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -306,6 +306,7 @@ endif()
 
 find_package(Perl)
 
+option(BUILD_DOCS "to build manual pages" ON)
 option(ENABLE_MANUAL "to provide the built-in manual" OFF)
 
 if(ENABLE_MANUAL AND PERL_FOUND)
diff --git a/docs/libcurl/CMakeLists.txt b/docs/libcurl/CMakeLists.txt
index fc1283a53..c719c9764 100644
--- a/docs/libcurl/CMakeLists.txt
+++ b/docs/libcurl/CMakeLists.txt
@@ -54,11 +54,13 @@ add_custom_command(OUTPUT libcurl-symbols.md
   VERBATIM
 )
 
-add_manual_pages(man_MANS)
-add_custom_target(man ALL DEPENDS ${man_MANS})
-if(NOT CURL_DISABLE_INSTALL)
-  install(FILES "$<LIST:TRANSFORM,${man_MANS},PREPEND,${CMAKE_CURRENT_BINARY_DIR}/>"
-          DESTINATION ${CMAKE_INSTALL_MANDIR}/man3)
-endif()
+if(BUILD_DOCS)
+  add_manual_pages(man_MANS)
+  add_custom_target(man ALL DEPENDS ${man_MANS})
+  if(NOT CURL_DISABLE_INSTALL)
+    install(FILES "$<LIST:TRANSFORM,${man_MANS},PREPEND,${CMAKE_CURRENT_BINARY_DIR}/>"
+            DESTINATION ${CMAKE_INSTALL_MANDIR}/man3)
+  endif()
 
-add_subdirectory(opts)
+  add_subdirectory(opts)
+endif()

(Edited to be enabled by default.)

bagder · 2024-01-22T22:19:47Z

They are built with autotools by default and there is no option to switch that off...

bagder · 2024-01-22T22:21:45Z

On my machine, building all libcurl ~500 man pages takes 800 milliseconds. It is a pretty quick operation, even if my machine might be above average CPU and disk speed wise.

vszakats · 2024-01-22T22:28:44Z

I takes almost as long as the build itself here, but I'm on fairly old gear. I'd appreciate an option to turn this off, as this will increase greatly the time for each build. When doing them by the hundreds, it adds up. (Even building curl.1/hugehelp takes a fair amount, but I can tweak that locally.) For some reason autotools never built man pages with the curl-for-win script, I don't know why, with --enable-manual set. [edit: since there was no build-phase involved, it was instantaneous and I never noticed it, but it did in fact copy the .3 files to the install target. This would make an option to disable it useful here as well after this PR.]

Edit: with this PR applied, autotools quits with:

make[3]: *** No rule to make target `curl-config.1', needed by `all-am'.  Stop.

bagder · 2024-01-22T22:36:16Z

Fair enough!

vszakats · 2024-01-22T22:51:45Z

Re: autotools, it did have this working before this patch, but was unnoticably fast (being just a handful of install commands). It may be noticable after this patch. Making a --disable-docs option useful there as well.

But, for now autotools builds quit with this error (with this PR applied), so I couldn't actually test it:

make[3]: *** No rule to make target `curl-config.1', needed by `all-am'.  Stop.

bagder · 2024-01-22T23:00:07Z

Edit: with this PR applied, autotools quits with:

We use autotools in numerous CI builds, including windows (on appveyor) and none of them fail like that... I'm not saying there is no problem left, but I might not consider that a blocker for merging this.

bagder · 2024-01-22T23:11:49Z

This is the biggest PR I/we have done in a long time. I completely expect that there will be some issues popping up once we merge this, but I don't think they are worse than we can just fix them. I believe this is a good PR.

vszakats · 2024-01-22T23:23:58Z

No worries, re-run it in a clean sandbox, and that resolved it.

gvanem · 2024-01-23T14:06:32Z

This scripts/cd2nroff file yet is another Windows unfriendly Perl script IMHO.
My Cygwin Perl bitches on this command:

f:/Cygwin64/bin/perl.exe ../scripts/cd2nroff ../docs/libcurl/ABI.md > ../docs/libcurl/ABI.3
../docs/libcurl/ABI.md:68:1:ERROR: no header present

But this works fine:

f:/Cygwin64/bin/perl.exe ../scripts/cd2nroff < ../docs/libcurl/ABI.md > ../docs/libcurl/ABI.3

And so does my good old StrawBerry Perl (v5.18.4 from 2014).
A newline-issue on if(/^---/) { once again. Arg!

Edit: Cygwin's Perl doesn't work at all. Hence using StrawBerry Perl.

bagder · 2024-01-23T14:32:51Z

It built in several Windows CI jobs, but I'm sure we can improve it.

gvanem · 2024-01-23T16:57:22Z

several Windows CI jobs

Since all those CI jobs checks out with git and config.autocrlf=false presumably?
I do not know muchPerl, but this open($fh, "<:crlf", "$f") opens in text-mode?

bagder · 2024-01-23T17:04:40Z

Since all those CI jobs checks out with git and config.autocrlf=false presumably?

No, I can't see anything that sets config.autocrlf=false or sets it specifically for .md files. Maybe we should rather set them to eol=lf ?

I do not know muchPerl, but this open($fh, "<:crlf", "$f") opens in text-mode?

Whenever I try to guess how something works with text files on Windows, I seem to guess wrong. So I don't know.

gvanem · 2024-01-23T20:11:11Z

Another thing I feel is strange. From the bottom of e.g. man CURLOPT_RESUME_FROM:

English text, but a Norwegian date (!)

bagder · 2024-01-23T22:22:00Z

I bet you can fix that by changing/setting some locale environment variable(s)...

- cmake: enable `BUILD_DOCS` by default (this controls converting and installing `.3` files from `.md` sources) - cmake: speed up generating `.3` files by using a single command per directory, instead of a single command per file. This reduces external commands by about a thousand. (There remains some CMake logic kicking in resulting in 500 -one per file- external `-E touch_nocreate` calls.) - cd2nroff: add ability to process multiple input files. - cd2nroff: add `-k` option to use the source filename to form the output filename. (instead of the default in-file `Title:` line.) Follow-up to 3f08d80 Follow-up to ea0b575 #12753 Follow-up to eefcc1b #12730 Closes #12762

gvanem · 2024-01-24T13:08:06Z

... some locale environment variable(s)...

Sure. LC_ALL=C or something? But man curl looks what I'd expect:

bagder · 2024-01-24T14:00:16Z

That I cannot explain because to me it looks like the generate the date strings identically!

It was accidentally added in curl#12730 Follow-up to eefcc1b

It was accidentally added in curl#12730 Co-authored-by: Lukáš Zaoral <lzaoral@redhat.com> Signed-off-by: Jan Macku <jamacku@redhat.com> Follow-up to eefcc1b

It was accidentally added in #12730 Co-authored-by: Lukáš Zaoral <lzaoral@redhat.com> Signed-off-by: Jan Macku <jamacku@redhat.com> Follow-up to eefcc1b Closes #12843

bagder added documentation script labels Jan 17, 2024

bagder changed the title ~~aintroducing "curldown" for libcurl man page format~~ introducing "curldown" for libcurl man page format Jan 17, 2024

bagder force-pushed the bagder/curldown branch from e67a9d8 to fe96ca5 Compare January 17, 2024 13:06

martinpeck reviewed Jan 17, 2024

View reviewed changes

scripts/nroff2cd Outdated Show resolved Hide resolved

bagder mentioned this pull request Jan 17, 2024

docs: cleanup nroff format use #12731

Closed

bagder force-pushed the bagder/curldown branch from fe96ca5 to 0139d7f Compare January 17, 2024 14:03

github-actions bot added the CI Continuous Integration label Jan 17, 2024

bagder force-pushed the bagder/curldown branch from 0139d7f to 272b028 Compare January 17, 2024 22:22

This comment was marked as resolved.

Sign in to view

bagder force-pushed the bagder/curldown branch 2 times, most recently from 57f7358 to ddbf3f7 Compare January 18, 2024 09:57

bagder marked this pull request as ready for review January 18, 2024 09:57

github-actions bot added the libcurl API label Jan 18, 2024

bagder force-pushed the bagder/curldown branch from ddbf3f7 to 2380072 Compare January 18, 2024 10:27

This comment was marked as resolved.

Sign in to view

cmeister2 reviewed Jan 18, 2024

View reviewed changes

docs/CURLDOWN.md Show resolved Hide resolved

bagder force-pushed the bagder/curldown branch from 2380072 to dd0a3d5 Compare January 18, 2024 12:01

github-actions bot added the tests label Jan 18, 2024

This comment was marked as resolved.

Sign in to view

bagder force-pushed the bagder/curldown branch 5 times, most recently from b8c1be7 to bd659c5 Compare January 18, 2024 22:31

bagder mentioned this pull request Jan 18, 2024

Update CURLOPT_QUOTE.3 #12736

Closed

fixup appveyor to disable documentation tests

b47ab3a

levitte mentioned this pull request Jan 22, 2024

Install curl.1 with cmake as well #12759

Closed

vszakats reviewed Jan 22, 2024

View reviewed changes

bagder closed this in eefcc1b Jan 22, 2024

vszakats mentioned this pull request Jan 23, 2024

cmake: speed up curldown processing, enable by default #12762

Closed

bagder deleted the bagder/curldown branch January 24, 2024 13:59

jamacku added a commit to jamacku/curl that referenced this pull request Feb 1, 2024

fix(docs): remove mk-ca-bundle.1 from man_MANS

6dd6b04

It was accidentally added in curl#12730 Follow-up to eefcc1b

jamacku mentioned this pull request Feb 1, 2024

fix(docs): remove mk-ca-bundle.1 from man_MANS #12843

Closed

bagder pushed a commit that referenced this pull request Feb 1, 2024

docs: remove mk-ca-bundle.1 from man_MANS

a911f4f

It was accidentally added in #12730 Co-authored-by: Lukáš Zaoral <lzaoral@redhat.com> Signed-off-by: Jan Macku <jamacku@redhat.com> Follow-up to eefcc1b Closes #12843

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introducing "curldown" for libcurl man page format #12730

introducing "curldown" for libcurl man page format #12730

bagder commented Jan 17, 2024 •

edited

bagder commented Jan 17, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

bagder commented Jan 18, 2024

bagder commented Jan 22, 2024

vszakats Jan 22, 2024 •

edited

bagder Jan 22, 2024

bagder Jan 22, 2024

bagder Jan 22, 2024

vszakats Jan 22, 2024 •

edited

bagder Jan 22, 2024

vszakats Jan 22, 2024 •

edited

bagder Jan 22, 2024

vszakats Jan 22, 2024

vszakats commented Jan 22, 2024 •

edited

bagder commented Jan 22, 2024

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024 •

edited

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024

bagder commented Jan 22, 2024

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024

gvanem commented Jan 23, 2024 •

edited

bagder commented Jan 23, 2024

gvanem commented Jan 23, 2024

bagder commented Jan 23, 2024

gvanem commented Jan 23, 2024

bagder commented Jan 23, 2024

gvanem commented Jan 24, 2024

bagder commented Jan 24, 2024

introducing "curldown" for libcurl man page format #12730

introducing "curldown" for libcurl man page format #12730

Conversation

bagder commented Jan 17, 2024 • edited

overview

tools:

CI

tasks

bagder commented Jan 17, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

bagder commented Jan 18, 2024

bagder commented Jan 22, 2024

vszakats Jan 22, 2024 • edited

Choose a reason for hiding this comment

bagder Jan 22, 2024

Choose a reason for hiding this comment

bagder Jan 22, 2024

Choose a reason for hiding this comment

bagder Jan 22, 2024

Choose a reason for hiding this comment

vszakats Jan 22, 2024 • edited

Choose a reason for hiding this comment

bagder Jan 22, 2024

Choose a reason for hiding this comment

vszakats Jan 22, 2024 • edited

Choose a reason for hiding this comment

bagder Jan 22, 2024

Choose a reason for hiding this comment

vszakats Jan 22, 2024

Choose a reason for hiding this comment

vszakats commented Jan 22, 2024 • edited

bagder commented Jan 22, 2024

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024 • edited

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024

bagder commented Jan 22, 2024

bagder commented Jan 22, 2024

vszakats commented Jan 22, 2024

gvanem commented Jan 23, 2024 • edited

bagder commented Jan 23, 2024

gvanem commented Jan 23, 2024

bagder commented Jan 23, 2024

gvanem commented Jan 23, 2024

bagder commented Jan 23, 2024

gvanem commented Jan 24, 2024

bagder commented Jan 24, 2024

bagder commented Jan 17, 2024 •

edited

vszakats Jan 22, 2024 •

edited

vszakats Jan 22, 2024 •

edited

vszakats Jan 22, 2024 •

edited

vszakats commented Jan 22, 2024 •

edited

vszakats commented Jan 22, 2024 •

edited

gvanem commented Jan 23, 2024 •

edited