tool: add command line variables #11346

bagder · 2023-06-19T14:21:39Z

Support command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)).

Variable contents can be expanded in option parameters using {{name}}" if the option name is prefixed with --expand-. This gets the contents of the variable name inserted, or a blank if the name does not exist as a variable. Insert {{ verbatim in the string by prefixing it with a backslash, like \{{.

You an access and expand environment variables by first importing them. You can select to either require the environment variable to be set or you can provide a default value in case it is not already set. Plain --variable %name imports the variable called name but exits with an error if that environment variable is not already set. To provide a default value if it is not set, use --variable %name=content or --variable %name@content.

Example: get the USER environment variable into the URL. Fail if USER is not set:

--variable '%USER'
--expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with trim, it can output the contents as a JSON quoted string with json and it can URL encode the string with urlencode. You apply the function(s) to a variable expansion, add them colon separated to the right side of the variable. They are then executed in a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data. if HOME is not set, use "dummy" as default:

--variable %HOME=dummy
--expand-variable fix@{{HOME}}/.secret
--expand-data "{{fix:trim:urlencode}}"
https://example.com/

Documented. Eight new test cases.

jay · 2023-06-20T03:57:06Z

I think 'encode' or 'urlencode' is easier to understand than 'percent' or %. --expand-data "{{fix:urlencode}}", also wouldn't it be easier if it could all be done in one step

edit:

Percent-encodes the entire contents.

ah, I didn't see that. Why would you percent encode the entire thing? Is there really a need for this type of functionality?

emanuele6 · 2023-06-20T04:08:11Z

@jay percent urlencodes only the value of the variable when expanding it, you use it when you want to a variable as a path component of a URL for example

--variable "foo=$foo" --expand-url 'https://example.org/{{foo:percent}}'

If you don't use :percent you will expand the value of {{foo}} directly potentially adding queries, fragments, slashes, etc

You mostly use bare {{foo}} for passing full values like --expand-user-agent {{foo}} or --expand-url {{foo}}, or for --expand-data '@{{foo}}/file.txt' where you don't need to prepare the value for an injection. Then :percent when you need to inject into a URL, :json when you need inject a compent into a JSON payload, etc

Support command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable contents can be expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Expand environment variables in addition to curl variables by prefixing the name with "env:" Example: get the USER environment variable into the URL: --expand-url = "https://example.com/api/{{env:USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", it can output the contents as a JSON quoted string with "json" and it can URL encode the string with "percent". You apply function to a variable expansion, add them colon separated to the right side of the variable. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --expand-variable fix@{{env:HOME}}/.secret --expand-data "{{fix:trim:percent}}" https://example.com/ Documented. Six new test cases. Co-brainstormed-by: Emanuele Torre Closes #11346

docs/cmdline-opts/variable.d

bagder · 2023-06-20T09:29:31Z

To consider: expanding a variable that contains a null byte. What to do with that null byte if neither json nor percent are applied? Should we cut the content at the null? Seems illogical. Should we replace/encode it with something that can be displayed? If so, with what? A period?

emanuele6 · 2023-06-20T12:30:26Z

Most UNIX shells (with few exceptions, e.g. ksh93 deletes everything after the first NUL) will remove all NUL bytes when reading the output of a command for $(), so "$(printf 'a\0b\0\0\0cdef')" => abcdef; that could be another option, I think it would be less surprising than adding dots to the value.
Bash also prints a warning when a NUL byte is removed this way (at most one warning per expansion) to let the user know it is happening.

Another option is to just fail with a fatal error if a user tries to expand a variable that contains NUL without using a function that escapes, which I honestly don't think is that horrible. Afterall, this can only happen if the user uses something like --variable foo@file (it cannot happen for environment variables or variables set with =), and they uses {{foo}} without :json/:percent which I think would be pretty niche; then if {{foo}} contains NUL in that case, the input is most definitely malformed.

bagder · 2023-06-21T06:14:25Z

After having slept on it, I believe I now favor the "fail with a fatal error" option. This will be the most helpful thing to users as it makes it immediately apparent what happens, as removing or replacing the null bytes will lead to more hard-to-debug errors to puzzle users.

If we want a replace option, we could instead implement a dedicated function for that purpose.

eloydegen · 2023-06-21T09:36:12Z

I think the documentation should include some note about the security risks. If an attacker is able to modify an environment variable it might be possible to exfiltrate data which might otherwise not be possible.

bagder · 2023-06-21T09:44:21Z

I think the documentation should include some note about the security risks. If an attacker is able to modify an environment variable it might be possible to exfiltrate data which might otherwise not be possible.

Providing clear documentation is of course valuable and a priority. Do you have any proposed wording?

An attacker that can change environment variables for curl is a serious attack already before this, since curl will acknowledge several variables outside of this new feature.

Also, this new feature is only expanding environment variables that is explicitly asked for, which is a reason we add this --expand- prefix thing.

Support command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable contents can be expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Expand environment variables in addition to curl variables by prefixing the name with "env:" Example: get the USER environment variable into the URL: --expand-url = "https://example.com/api/{{env:USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", it can output the contents as a JSON quoted string with "json" and it can URL encode the string with "percent". You apply function to a variable expansion, add them colon separated to the right side of the variable. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --expand-variable fix@{{env:HOME}}/.secret --expand-data "{{fix:trim:percent}}" https://example.com/ Documented. Eight new test cases. Co-brainstormed-by: Emanuele Torre Closes #11346

bagder · 2023-06-21T11:46:31Z

Me and @emanuele6 brainstormed a bit more on IRC.

What if you want to force an environment variable to be set or
What if you want a default value in case the environment variable is not set?

We came up with this tweak to support both those things, that I think fits rather nicely:

remove {{%name}} support, get back to a single name space where we need to "import" environment variables to be able to expand them

Import environment variables like this:

--variable %name imports an environment variable, fails with an error if not set. It can then be expanded as normal with {{name}}.
--variable %name=default imports an environment variable, or sets it to the default value. The default can then also be read from a file with --variable %name@default
~~- name can not be the same name as an existing variable~~

emanuele6 · 2023-06-21T12:00:15Z

name can not be the same name as an existing variable

Personally I think you should be allowed to redefine a variable.

While --variable foo=bar --variable foo=baz is likely a mistake, it is possible that the curlrc is defining with common names like proxy, host or ua, and if a script tries to use --variable proxy= unknowingly, it will fail unexpectedly.

Since variables are only used by the user for --expand- options, and not by curl internally, I think it should be fine to allow redefining them.

bagder · 2023-06-21T12:01:30Z

I think it should be fine to allow redefining them.

It works for me. I'll amend.

src/var.c

Support command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable contents can be expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Expand environment variables in addition to curl variables by prefixing the name with "env:" Example: get the USER environment variable into the URL: --expand-url = "https://example.com/api/{{env:USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", it can output the contents as a JSON quoted string with "json" and it can URL encode the string with "percent". You apply function to a variable expansion, add them colon separated to the right side of the variable. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --expand-variable fix@{{env:HOME}}/.secret --expand-data "{{fix:trim:percent}}" https://example.com/ Documented. Eight new test cases. Co-brainstormed-by: Emanuele Torre Closes #11346

Add support for command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable content is expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Import an environment variable with --variable %name. It makes curl exit with an error if the environment variable is not set. It can also rather get a default value if the variable does not exist, using =content or @file like shown above. Example: get the USER environment variable into the URL: --variable %USER --expand-url = "https://example.com/api/{{USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", it can output the contents as a JSON quoted string with "json" and it can URL encode the string with "urlencode". You apply function to a variable expansion, add them colon separated to the right side of the variable. They are then performed in a left to right order. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --variable %HOME=/home/default --expand-variable fix@{{HOME}}/.secret --expand-data "{{fix:trim:urlencode}}" https://example.com/ Documented. Many new test cases. Co-brainstormed-by: Emanuele Torre Closes #11346

Add support for command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable content is expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Import an environment variable with --variable %name. It makes curl exit with an error if the environment variable is not set. It can also rather get a default value if the variable does not exist, using =content or @file like shown above. Example: get the USER environment variable into the URL: --variable %USER --expand-url = "https://example.com/api/{{USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", output the contents as a JSON quoted string with "json", URL encode it with "url" and base 64 encode it with "b64". To apply functions to a variable expansion, add them colon separated to the right side of the variable. They are then performed in a left to right order. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --variable %HOME=/home/default --expand-variable fix@{{HOME}}/.secret --expand-data "{{fix:trim:urlencode}}" https://example.com/ Documented. Many new test cases. Co-brainstormed-by: Emanuele Torre Closes #11346

src/var.c

bagder · 2023-07-31T06:22:19Z

thanks @jay!

Add support for command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)). Variable content is expanded in option parameters using "{{name}}" (without the quotes) if the option name is prefixed with "--expand-". This gets the contents of the variable "name" inserted, or a blank if the name does not exist as a variable. Insert "{{" verbatim in the string by prefixing it with a backslash, like "\\{{". Import an environment variable with --variable %name. It makes curl exit with an error if the environment variable is not set. It can also rather get a default value if the variable does not exist, using =content or @file like shown above. Example: get the USER environment variable into the URL: --variable %USER --expand-url = "https://example.com/api/{{USER}}/method" When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with "trim", output the contents as a JSON quoted string with "json", URL encode it with "url" and base 64 encode it with "b64". To apply functions to a variable expansion, add them colon separated to the right side of the variable. They are then performed in a left to right order. Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data: --variable %HOME=/home/default --expand-variable fix@{{HOME}}/.secret --expand-data "{{fix:trim:url}}" https://example.com/ Documented. Many new test cases. Co-brainstormed-by: Emanuele Torre Assisted-by: Jat Satiro Closes curl#11346

bagder added cmdline tool feature-window A merge of this requires an open feature window labels Jun 19, 2023

github-actions bot added the tests label Jun 19, 2023

bagder mentioned this pull request Jun 19, 2023

tool_parsecfg: support environment variables in config files #11311

Closed

2 tasks

bagder force-pushed the bagder/variables branch from b12e1cc to 2c3fe9c Compare June 20, 2023 06:42

ukaea-chah reviewed Jun 20, 2023

View reviewed changes

docs/cmdline-opts/variable.d Outdated Show resolved Hide resolved

bagder force-pushed the bagder/variables branch from 9d19b96 to 79e2010 Compare June 21, 2023 10:06

github-advanced-security bot found potential problems Jun 21, 2023

View reviewed changes

src/var.c Fixed Show fixed Hide fixed

bagder force-pushed the bagder/variables branch from e293b32 to 26e18d8 Compare June 21, 2023 22:19

bagder force-pushed the bagder/variables branch from 26e18d8 to 67274d1 Compare June 22, 2023 08:37

bagder changed the title ~~tool: add "variable" support~~ tool: add command line variables Jun 22, 2023

bagder self-assigned this Jun 22, 2023

bagder force-pushed the bagder/variables branch from be86528 to c73bae6 Compare July 9, 2023 09:22

emanuele6 mentioned this pull request Jul 9, 2023

tool_writeout_json: fix encoding of control characters #11414

Closed

bagder force-pushed the bagder/variables branch from c73bae6 to 444c5cb Compare July 9, 2023 21:06

bagder mentioned this pull request Jul 13, 2023

Regression on parsing large config files #11431

Closed

bagder force-pushed the bagder/variables branch from 444c5cb to 7eca623 Compare July 13, 2023 22:05

bagder force-pushed the bagder/variables branch from 7eca623 to c9d1284 Compare July 21, 2023 12:35

bagder force-pushed the bagder/variables branch from aaa8272 to a1daccf Compare July 22, 2023 22:38

bagder force-pushed the bagder/variables branch from a1daccf to 173f76b Compare July 30, 2023 22:03

github-advanced-security bot found potential problems Jul 30, 2023

View reviewed changes

src/var.c Dismissed Show dismissed Hide dismissed

bagder and others added 2 commits July 31, 2023 08:00

fixup cmake fix for windows?

1a1f82f

add base64 curlx unit to pre-generated projects and winbuild

2087173

bagder closed this in 2e160c9 Jul 31, 2023

bagder deleted the bagder/variables branch July 31, 2023 09:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tool: add command line variables #11346

tool: add command line variables #11346

bagder commented Jun 19, 2023 •

edited

jay commented Jun 20, 2023 •

edited

emanuele6 commented Jun 20, 2023 •

edited

bagder commented Jun 20, 2023

emanuele6 commented Jun 20, 2023 •

edited

bagder commented Jun 21, 2023

eloydegen commented Jun 21, 2023

bagder commented Jun 21, 2023

bagder commented Jun 21, 2023 •

edited

emanuele6 commented Jun 21, 2023

bagder commented Jun 21, 2023

bagder commented Jul 31, 2023

tool: add command line variables #11346

tool: add command line variables #11346

Conversation

bagder commented Jun 19, 2023 • edited

jay commented Jun 20, 2023 • edited

emanuele6 commented Jun 20, 2023 • edited

bagder commented Jun 20, 2023

emanuele6 commented Jun 20, 2023 • edited

bagder commented Jun 21, 2023

eloydegen commented Jun 21, 2023

bagder commented Jun 21, 2023

bagder commented Jun 21, 2023 • edited

emanuele6 commented Jun 21, 2023

bagder commented Jun 21, 2023

bagder commented Jul 31, 2023

bagder commented Jun 19, 2023 •

edited

jay commented Jun 20, 2023 •

edited

emanuele6 commented Jun 20, 2023 •

edited

emanuele6 commented Jun 20, 2023 •

edited

bagder commented Jun 21, 2023 •

edited