Too be honest, I have not done anything to rule out git-ftp or the server other than post an issue with git-ftp.

Everything was working fine yesterday, then this morning when I updated curl via homebrew, the issue started. That is the only recent change as no other updates have occurred.

Git-ftp was lasted updated a year ago. I use cPanel [106.0.11] on my VPS which was updated on Dec 1, 2022

I just did a quick test by creating 140 files which successfully uploaded. I then reverted the commit and git-ftp successfully deleted the files just uploaded.

As best I can determine git-ftp creates a file with the files to upload and passes that to curl.

Without a much more detailed description of what is happening at a protocol level and info about what libcurl is told to do for these transfers and specific info how/when it fails, it is difficult for us to do anything more than blindly guess.

So this morning I did a bit of digging into git-ftp:

This is the command that is executed:

curl -s --globoff -# --ftp-create-dirs -K /var/folders/p1/2q6_xz196qsg9g3j67d8qf7c0000gn/T/git-ftp-XXXXXX.C12jHSJ5/curl_upload_list

sample record from the curl_upload_list - 1380 entries
xxxxx is the domain
yyyyyyy was the password

-T "./www/templates/site/xxxxx/right-col.twig"
url = "ftp://xxxxx:yyyyyyy@ftp.xxxxx.com/www/templates/site/xxxxx/right-col.twig"

The first timeout occurs at record 528

* Remembering we are in dir "www/cmsb/lib//menus/_codeGenerator/"
< 226-File successfully transferred
< 226 0.083 seconds (measured here), 52.88 Kbytes per second
* Connection #0 to host ftp.xxxxx.com left intact

* Found bundle for host: 0x600003faf540 [serially]
* Re-using existing connection #0 with host ftp.xxxxx.com
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Extended Passive mode OK (|||57903|)
* Connecting to 111.111.111.111 (111.111.111.111) port 57903
*   Trying 111.111.111.111:57903...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#0)
> STOR api_categoryMenu.php
< 150 Accepted data connection
} [15151 bytes data]
* We are completely uploaded and fine
################################################################################################################# 100.0%
* Remembering we are in dir "www/cmsb/lib//menus/_codeGenerator/"
* FTP response timeout
* control connection looks dead
* Closing connection 0
curl: (28) FTP response timeout

*   Trying 111.111.111.111:21...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#1)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 1 of 50 allowed.
< 220-Local time is now 16:44. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.

< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD www
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /public_html
> CWD cmsb
< 250 OK. Current directory is /public_html/cmsb
> CWD lib
< 250 OK. Current directory is /public_html/cmsb/lib
> CWD menus
< 250 OK. Current directory is /public_html/cmsb/lib/menus
> CWD _codeGenerator
< 250 OK. Current directory is /public_html/cmsb/lib/menus/_codeGenerator
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||49664|)
* Connecting to 111.111.111.111 (111.111.111.111) port 49664
*   Trying 111.111.111.111:49664...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#1)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR api_comboPage.php
< 150 Accepted data connection
} [9745 bytes data]
* We are completely uploaded and fine

second timeout occurred at record 565
died on record 636

* Remembering we are in dir "www/cmsb/plugins//csvExport/"
< 226-File successfully transferred
< 226 0.084 seconds (measured here), 23.43 Kbytes per second
* Connection #2 to host ftp.xxxxx.com left intact

* Found bundle for host: 0x600003fa0060 [serially]
* Re-using existing connection #2 with host ftp.xxxxx.com
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Extended Passive mode OK (|||63906|)
* Connecting to 111.111.111.111 (111.111.111.111) port 63906
*   Trying 111.111.111.111:63906...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#2)
> STOR csvExport.php
< 150 Accepted data connection
} [9432 bytes data]
* We are completely uploaded and fine
################################################################################################################# 100.0%
* Remembering we are in dir "www/cmsb/plugins//csvExport/"
< 226-File successfully transferred
< 226 0.083 seconds (measured here), 111.52 Kbytes per second
* Connection #2 to host ftp.xxxxx.com left intact

curl: (27) Out of memory

If you can provide some guidance on what else I can do to trouble shoot this issue, I am happy to do so.
I very much appreciate your help.

remove now TY, and password reset

I agree the FTP timeout does not appear to be an issue as it does reconnect.

I opened the activity monitor and when curl first starts, it is using 2.0MB of memory
When the (27) out of memory error occurs it is at 2.3MB

Currently:
ulimit -u 2048
ulimit -n 524288
cgroups is not install on my machine.

I don't know what you mean by 'instrument curl' or how to do that.

I have been using iterm2 so tried using terminal and got exactly the same results.

ulimit -u is the number of processes available and not memory. I am well below that in running processes.
Memory available is unlimited (iterm is using 160MiB)

git-ftp (bash script) only calls curl once and passes the list of files to upload.

I am not a c programmer so don't know how to compile a debug version of curl. If someone could compile a macos version with debug for me, I am happy to run it and upload any logs.

I was trying to download a 7.85 macos binary but can't find one to see if that one worked.

I really appreciate any help I get.

I tried to compile curl with

./configure --with-openssl --enable-debug && make -j4

and got the following error"

configure: OPT_OPENSSL: yes
configure: OPENSSL_ENABLED:
configure: error: --with-openssl was given but OpenSSL could not be detected

I am running macos Monterey v12.6.2. This way outside my skill set but willing to try again.

I couldn't figure out how to compile a debug version of curl so plan B.

I was able to restore curl 7.85 from my timemachine backup then reran my original command git-ftp push and it was successful.

So I conclude that it is with curl 7.86 & 7.87 and likely the changes in memory handling.

I think it is unlikely to be because of "changes in memory handling" but is more likely a misleading error code.

I created a config file with 120 separate upload files in a similar fashion to this report (one line of -T and another one with the target URL), and I have rerun it many times using 7.87.0 stock from Debian and my dev version of the master branch but I have not managed to reproduce this problem.

Possible it is mac related, so unless someone can provide me with a compiled debug version, I will use 7.85.

I can't afford to spend much more time on this.

I just reran my test in my mac with curl from git master. I increased the number of files to see if perhaps the number of uploads could be the key. This time I uploaded 3600 files serially. Each uploaded file was 1.3MB big. Worked fine. I still cannot reproduce this problem.

Thanks @Badger,

Did you have all the files in a txt file with one call to curl or did you call curl 3600 times? When you say serially, I assume you mean 3800 calls to curl.

Also I have curl installed via homebrew. Are these potential differences?

One command line with -K specifying a file with 7200 lines of -T and url = like you showed.

Serially, as in not parallel (-Z).

Also I have curl installed via homebrew. Are these potential differences?

Not to my knowledge, no.

Any chance you could compile a debug version for mac for me and I will try it here.

Sorry, I've not had time for that and I don't think I will for the foreseeable future.

I have had the same issue since late last year as well. I use git-ftp for a side project I don't touch often so it hasn't been worth my time to debug, but getting git-ftp working again would save me a lot of time and hassle, so if I can be of any help as well let me know. I am using macOS 13 Ventura.

When I try to use git-ftp with -v I also see the curl: (27) Out of memory error, except in my case I'm not trying to change many files. In this example there are 14 files to upload, but I've seen the same error no matter how many or few files I try.

I don't know if the problem started right after moving to Ventura since I upgraded macOS a few months before touching this project again, but I have curl 7.86.0 installed so I am going to try going back to 7.85.0 as well and see if that fixes things for me.

❯ git ftp push -v
Sat Feb  4 14:43:05 PST 2023: git-ftp version 1.6.0 running on Darwin Harstons-iMac.home 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64
Sat Feb  4 14:43:05 PST 2023: Syncroot is './'.
Sat Feb  4 14:43:05 PST 2023: Host is '216.198.213.4'.
Sat Feb  4 14:43:05 PST 2023: User is 'kalbik320'.
Sat Feb  4 14:43:05 PST 2023: Read password from keychain.
Sat Feb  4 14:43:05 PST 2023: Password is set.
Sat Feb  4 14:43:05 PST 2023: CACert is ''.
Sat Feb  4 14:43:05 PST 2023: Insecure is '0'.
Sat Feb  4 14:43:05 PST 2023: Proxy is ''.
Sat Feb  4 14:43:05 PST 2023: Protocol not set, using default protocol ftp://.
Sat Feb  4 14:43:05 PST 2023: Path is ''.
Sat Feb  4 14:43:05 PST 2023: The remote sha1 is saved in file '.git-ftp.log'.
Sat Feb  4 14:43:05 PST 2023: Check if curl is functional.
Sat Feb  4 14:43:05 PST 2023: Retrieving last commit from ftp://kalbik320:***@216.198.213.4/.
###################################################################### 100.0%
Sat Feb  4 14:43:06 PST 2023: Last deployed SHA1 for 216.198.213.4/ is ed90df31faa781e43f2ba1055710e31f6c471c21.
Sat Feb  4 14:43:06 PST 2023: Having files to sync.
Sat Feb  4 14:43:06 PST 2023: 14 files to sync:
Sat Feb  4 14:43:06 PST 2023: [1 of 14] Buffered for upload 'change logs/Organization.txt'.
Sat Feb  4 14:43:06 PST 2023: Uploading ...
curl: (27) Out of memory
Sat Feb  4 14:43:06 PST 2023: fatal: Could not upload files., exiting...

Okay, removing my MacPorts installation of curl 7.86.0 puts me back to the system provided curl 7.85.0, and now I get a different error:

curl: (3) URL using bad/illegal format or missing URL

So maybe I was seeing the issue where @dfandrich mentioned with the erroneous error. I'll try 7.87.0 and see what I get.

There was a fix for an issue where curl would erroneously return "Out of memory" for a bad URL in yesterday's release...

Curl 7.87.0 gives me the same output as 7.86.0 did, with curl: (27) Out of memory.

I'm not ftp or curl expert, I rarely use them for anything, so any advice on a command to try to manually do what git-ftp is doing for me would be great, so I can see if using curl directly solves the issue.

Argh... I feel like an idiot now, but at least I found my issue. Notice that the file it failed to push had a space in the path. 🤦‍♂️ Replacing the space with an underscore fixed my issue.

That explains why it doesn't matter how many files I tried, that file was always among them, and I changed it late last year, so that explains why I started seeing the issue then.

So, the real issue I am seeing is that curl is reporting error (27) Out of memory instead of (3) URL using bad/illegal format or missing URL, which sent me down a wild goose chase.

That makes it easy to reproduce:

$ curl -T nothing -v 'ftp://example.com/with space'
curl: (27) Out of memory

That's at HEAD (7.87.1-DEV).

And on the macOS provided 7.85.0 I get:

❯ curl -T nothing -v 'ftp://example.com/with space'
curl: Can't open 'nothing'!
curl: try 'curl --help' or 'curl --manual' for more information
curl: (26) Failed to open/read local data from file/application

Now that I fixed that one path issue I installed curl 7.87.0 and tried pushing a larger change (982 files) and I hit the same Out of memory error on one of the files and then curl exited.

Going back to 7.85.0 I got the proper curl: (3) URL using bad/illegal format or missing URL error and then curl continued to the next file as expected, giving me that error for each of the files with the bad path.

Does #10414 fix it for you?

I'll build and try it out.

I am struggling with build settings. I can build curl, but when I try to use it with git-ftp I get errors.

Looking at the code changes though I'm pretty confident you got it. Classic C issue where the errors don't get propagated properly.

Actually, all this will fix is to give the ability to git-ftp to display a proper error message. git-ftp will still need to be fixed to escape URLs properly so they are valid when given to curl.

I can confirm that spaces in the path are my issue as well with curl 7.87

Yes, this doesn't fix the issue with git-ftp, but at least it lets the user know what the actual issue is.

And there is a PR open with the fix for git-ftp: git-ftp/git-ftp#638

I applied that change to my local copy and it fixed my issues.

I can confirm that the fix git-ftp/git-ftp#638 fixes the issue for me as well.