From: Toon Claes <toon@iotcl.com>
To: git@vger.kernel.org
Cc: Toon Claes <toon@iotcl.com>, Jeff King <peff@peff.net>
Subject: [PATCH v2 0/7] Show progress when downloading from bundle URIs
Date: Wed, 19 Feb 2025 15:30:18 +0100 [thread overview]
Message-ID: <20250219-toon-bundleuri-progress-v2-0-a84e7ffa921a@iotcl.com> (raw)
In-Reply-To: <20240508124453.600871-1-toon@iotcl.com>
When a user clones a repository, they see what's happening in the
messages like "Enumerating objects" and "Receiving objects". But when
a user clones a repository that uses bundle URIs they see:
Cloning into 'repo.git'
And then they have to wait until all bundles are downloaded before they
see any other message. When the bundles are large, this can take a lot
of time and the user might consider the process hangs and they kill it.
This patch series introduces progress displayed to the user while
bundles are downloaded. The full output of a clone using bundle URIs
will look something like:
Cloning into 'repo.git'...
Downloading via HTTP: 21% (351812809/1620086598), 315.34 MiB | 49.84 MiB/s
Downloading via HTTP: 77% (1247493865/1620086598), 1.15 GiB | 34.31 MiB/s
Downloading via HTTP: 100% (1620086598/1620086598), 1.50 GiB | 37.97 MiB/s, done.
remote: Enumerating objects: 1322255, done.
remote: Counting objects: 100% (611708/611708), done.
remote: Total 1322255 (delta 611708), reused 611708 (delta 611708), pack-reused 710547
Receiving objects: 100% (1322255/1322255), 539.66 MiB | 31.57 MiB/s, done.
etc...
In this version I'm adding two commits from Peff. And I'm adding one
commit on top to fix the failure in t5558. I'd like to get some feedback
on that commit in particular.
---
Changes in v2:
- Added commit "http: silence stderr when progress is enabled"
- Added commit "progress: allow pure-throughput progress meters" from
https://lore.kernel.org/git/20111110075300.GK27950@sigill.intra.peff.net/
- Added commit "http: turn off curl signals" from
https://lore.kernel.org/git/20240509165212.GC1708095@coredump.intra.peff.net/
- Cleanup both CURLOPT_NOPROGRESS and CURLOPT_NOPROGRESS.
- Remove unneeded fflush().
- Link to v1: https://lore.kernel.org/git/20240508124453.600871-1-toon@iotcl.com/
Signed-off-by: Toon Claes <toon@iotcl.com>
---
Jeff King (2):
progress: allow pure-throughput progress meters
http: turn off curl signals
Toon Claes (5):
progress: add function to set total
http: add the ability to log progress
remote-curl: optionally show progress for HTTP get
bundle-uri: enable git-remote-https progress
http: silence stderr when progress is enabled
bundle-uri.c | 2 +-
http.c | 39 ++++++++++++++++++++++++++++++++
http.h | 5 +++++
progress.c | 40 ++++++++++++++++++++++++---------
progress.h | 2 ++
remote-curl.c | 8 ++++++-
t/helper/test-progress.c | 5 +++++
t/t0500-progress-display.sh | 55 +++++++++++++++++++++++++++++++++++++++++++++
t/t5557-http-get.sh | 15 +++++++++++++
9 files changed, 159 insertions(+), 12 deletions(-)
---
Range-diff versus v1:
1: 3eb1958b8b ! 1: 7acc83d92b progress: add function to set total
@@ Commit message
Signed-off-by: Toon Claes <toon@iotcl.com>
## progress.c ##
-@@ progress.c: static struct progress *start_progress_delay(const char *title, uint64_t total,
+@@ progress.c: static struct progress *start_progress_delay(struct repository *r,
return progress;
}
@@ progress.h: void progress_test_force_update(void);
void display_throughput(struct progress *progress, uint64_t total);
void display_progress(struct progress *progress, uint64_t n);
+void progress_set_total(struct progress *progress, uint64_t total);
- struct progress *start_progress(const char *title, uint64_t total);
- struct progress *start_sparse_progress(const char *title, uint64_t total);
- struct progress *start_delayed_progress(const char *title, uint64_t total);
+ struct progress *start_progress(struct repository *r,
+ const char *title, uint64_t total);
+ struct progress *start_sparse_progress(struct repository *r,
## t/helper/test-progress.c ##
@@ t/helper/test-progress.c: int cmd__progress(int argc, const char **argv)
if (*end != '\0')
- die("invalid input: '%s'\n", line.buf);
+ die("invalid input: '%s'", line.buf);
display_progress(progress, item_count);
+ } else if (skip_prefix(line.buf, "total ", (const char **) &end)) {
+ uint64_t total = strtoull(end, &end, 10);
-: ---------- > 2: 7a81cda492 progress: allow pure-throughput progress meters
-: ---------- > 3: 86a9120539 http: turn off curl signals
2: b74ce999a1 ! 4: 3a73d68448 http: add the ability to log progress
@@ http.c: static void http_opt_request_remainder(CURL *curl, off_t pos)
#define HTTP_REQUEST_FILE 1
+static int http_progress_callback(void *clientp, curl_off_t dltotal,
-+ curl_off_t dlnow, curl_off_t ultotal,
-+ curl_off_t ulnow)
++ curl_off_t dlnow, curl_off_t ultotal UNUSED,
++ curl_off_t ulnow UNUSED)
+{
+ struct progress *progress = clientp;
+
@@ http.c: static int http_request(const char *url,
http_follow_config == HTTP_FOLLOW_INITIAL)
curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);
+ if (options && options->progress) {
-+ progress = start_progress(_("Downloading via HTTP"), 0);
++ progress = start_progress(the_repository, _("Downloading via HTTP"), 0);
+
+ curl_easy_setopt(slot->curl, CURLOPT_NOPROGRESS, 0L);
+ curl_easy_setopt(slot->curl, CURLOPT_XFERINFODATA, progress);
@@ http.c: static int http_request(const char *url,
ret = run_one_slot(slot, &results);
+ if (progress) {
++ curl_easy_setopt(slot->curl, CURLOPT_NOPROGRESS, 1L);
+ curl_easy_setopt(slot->curl, CURLOPT_XFERINFODATA, NULL);
++ curl_easy_setopt(slot->curl, CURLOPT_XFERINFOFUNCTION, NULL);
+ stop_progress(&progress);
+ }
+
3: 5b5e7e86c5 ! 5: 01b2a0042f remote-curl: optionally show progress for HTTP get
@@ t/t5557-http-get.sh: test_expect_success 'get by URL: 200' '
+ EOF
+
+ git remote-http $url <input 2>err &&
-+ test_grep "^Downloading via HTTP: 100%" err
++ test_grep "^Downloading via HTTP: 100%" err
+'
+
test_done
4: 98ef4dfade ! 6: 3df90aa17b bundle-uri: enable git-remote-https progress
@@ bundle-uri.c: static int download_https_uri_to_file(const char *file, const char
}
+ fprintf(child_in, "option progress true\n");
-+ fflush(child_in);
-+
fprintf(child_in, "get %s %s\n\n", uri, file);
cleanup:
-: ---------- > 7: c30c3bf4fe http: silence stderr when progress is enabled
---
base-commit: a554262210b4a2ee6fa2d594e1f09f5830888c56
change-id: 20250219-toon-bundleuri-progress-3d902efeba06
Thanks
--
Toon
next prev parent reply other threads:[~2025-02-19 14:30 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-08 12:44 [PATCH 0/4] bundle-uri: show progress when downloading from bundle URIs Toon Claes
2024-05-08 12:44 ` [PATCH 1/4] progress: add function to set total Toon Claes
2024-05-08 12:44 ` [PATCH 2/4] http: add the ability to log progress Toon Claes
2024-05-08 16:52 ` Eric Sunshine
2024-05-09 16:34 ` Jeff King
2024-05-09 16:51 ` Junio C Hamano
2024-05-09 17:04 ` Jeff King
2024-05-09 16:52 ` Jeff King
2024-05-08 12:44 ` [PATCH 3/4] remote-curl: optionally show progress for HTTP get Toon Claes
2024-05-08 22:29 ` Junio C Hamano
2024-05-08 12:44 ` [PATCH 4/4] bundle-uri: enable git-remote-https progress Toon Claes
2024-05-09 16:46 ` Jeff King
2025-02-14 11:26 ` Toon Claes
2025-02-21 7:36 ` Jeff King
2024-05-08 23:49 ` [PATCH 0/4] bundle-uri: show progress when downloading from bundle URIs Junio C Hamano
2025-02-19 14:30 ` Toon Claes [this message]
2025-02-19 14:30 ` [PATCH v2 1/7] progress: add function to set total Toon Claes
2025-02-21 7:43 ` Jeff King
2025-02-19 14:30 ` [PATCH v2 2/7] progress: allow pure-throughput progress meters Toon Claes
2025-02-19 14:30 ` [PATCH v2 3/7] http: turn off curl signals Toon Claes
2025-02-19 14:30 ` [PATCH v2 4/7] http: add the ability to log progress Toon Claes
2025-02-19 14:30 ` [PATCH v2 5/7] remote-curl: optionally show progress for HTTP get Toon Claes
2025-02-19 14:30 ` [PATCH v2 6/7] bundle-uri: enable git-remote-https progress Toon Claes
2025-02-19 14:30 ` [PATCH v2 7/7] http: silence stderr when progress is enabled Toon Claes
2025-02-21 7:48 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250219-toon-bundleuri-progress-v2-0-a84e7ffa921a@iotcl.com \
--to=toon@iotcl.com \
--cc=git@vger.kernel.org \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).