diff options
author | Eric Wong <e@80x24.org> | 2023-03-05 22:18:11 +0000 |
---|---|---|
committer | Eric Wong <e@80x24.org> | 2023-03-07 19:12:35 +0000 |
commit | 8eca0987cfbcea8b1276d9852fb26d0ddb11ed3b (patch) | |
tree | e22fa6d6a79047464f005a9b5ec1ebd7ea50f199 | |
parent | b1444df229235fdc26edb157f471aa041ed87dde (diff) | |
download | public-inbox-8eca0987cfbcea8b1276d9852fb26d0ddb11ed3b.tar.gz |
Basically, public-inbox-clone has become grok-pull without config files nor absolute paths.
-rw-r--r-- | Documentation/public-inbox-clone.pod | 62 | ||||
-rwxr-xr-x | script/public-inbox-clone | 12 |
2 files changed, 57 insertions, 17 deletions
diff --git a/Documentation/public-inbox-clone.pod b/Documentation/public-inbox-clone.pod index 7774d4ed..af4e8e95 100644 --- a/Documentation/public-inbox-clone.pod +++ b/Documentation/public-inbox-clone.pod @@ -15,14 +15,16 @@ making the initial clone of a remote HTTP(S) public-inbox. It allows cloning multi-epoch v2 inboxes with a single command and zero configuration. -In public-inbox 2.0+, public-inbox-clone can also mirror multiple -inboxes or code repositories using manifest.js.gz files like -L<grok-pull(1)> from grokmirror. +In public-inbox 2.0+, public-inbox-clone can create and maintain +a mirror of multiple inboxes or code repositories using manifest.js.gz +files like L<grok-pull(1)> from grokmirror. L<public-inbox-fetch(1)> is +NOT required when using this mode. It does not run L<public-inbox-init(1)> nor L<public-inbox-index(1)>. Those commands must be run separately if serving/searching the mirror is required. As-is, -public-inbox-clone is suitable for creating a git-only backup. +public-inbox-clone is suitable for creating a git-only backup +without Xapian and SQLite indices. When cloning a single inbox, public-inbox-clone creates a Makefile with handy targets to update the inbox once indexed. @@ -31,11 +33,12 @@ not be rewritten by L<public-inbox-fetch(1)> unless it is removed completely. public-inbox-clone does not use nor require any extra -configuration files (not even C<~/.public-inbox/config>). +configuration files (not even C<~/.public-inbox/config>), +but it can download snippets suitable for adding to any +L<public-inbox-config(5)> file. L<public-inbox-fetch(1)> may be used to keep a single C<INBOX_DIR> -up-to-date. When using manifest.js.gz, public-inbox-clone is used -to maintain the multi-repository mirror. +up-to-date. For v2 inboxes, it will create a C<$INBOX_DIR/manifest.js.gz> file to speed up subsequent L<public-inbox-fetch(1)>. @@ -63,17 +66,17 @@ Default: C<0..~0> or C<0..> or C<..~0> =item --include=PATTERN -When cloning a top-level with multiple inboxes, only clone inboxes and -repositories matching a given wildcard pattern (using C<*?> and C<[]> is -supported). +When cloning a top-level with multiple inboxes via manifest, +only clone inboxes and repositories matching a given wildcard pattern +(using C<*?> and C<[]> is supported). This is a new option in public-inbox 2.0+ =item --exclude=PATTERN -When cloning a top-level with multiple inboxes, ignore inboxes and -repositories matching the given wildcard pattern. Supports the same -wildcards as L</--include> +When cloning a top-level with multiple inboxes via manifest, +ignore inboxes and repositories matching the given wildcard pattern. +Supports the same wildcards as L</--include> This is a new option in public-inbox 2.0+ @@ -204,6 +207,39 @@ Whether to wrap L<git(1)> and L<curl(1)> commands with L<torsocks(1)>. Default: C<auto> +=item -j JOBS +=item --jobs=JOBS + +The number of parallel processes to spawn at once for various network +operations using L<git(1)> and/or L<curl(1)>. + +=back + +=head1 EXAMPLES + +=for comment +Sticking to smaller projects in examples to minimize load on servers + +=over + +=item To mirror the most recent epochs of dwarves and LTTng inboxes: + + public-inbox-clone --epoch=~0 \ + --include='*lttng*' --include='*dwarves' \ + https://80x24.org/lore/ /path/to/inbox-mirror + +C<https://lore.kernel.org/> may be used instead of C<https://80x24.org/lore/> + +=item To mirror all code repos of the sparse project: + + public-inbox-clone --objstore= --project-list= --prune \ + --include='*sparse*' --inbox-config=never \ + --remote-manifest=https://80x24.org/lore/pub/manifest.js.gz \ + https://80x24.org/lore/ /path/to/code-mirror + +C<https://git.kernel.org/> may be used instead of C<https://80x24.org/lore/> +and the C<--remote-manifest> option can be omitted. + =back =head1 CONTACT diff --git a/script/public-inbox-clone b/script/public-inbox-clone index 10ad3487..5b365df7 100755 --- a/script/public-inbox-clone +++ b/script/public-inbox-clone @@ -6,21 +6,25 @@ use v5.12; use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev); my $opt = {}; my $help = <<EOF; # the following should fit w/o scrolling in 80x24 term: -usage: public-inbox-clone INBOX_URL [DESTINATION] +usage: public-inbox-clone [OPTIONS] INBOX_URL [INBOX_DIR] + public-inbox-clone [OPTIONS] ROOT_URL [DESTINATION] - clone remote public-inboxes + clone remote public-inboxes or grokmirror manifests options: --epoch=RANGE range of v2 epochs to clone (e.g `2..5', `~0', `~1..') - --objstore=DIR share storage for coderepos --torsocks VAL whether or not to wrap git and curl commands with torsocks (default: `auto') Must be one of: `auto', `no' or `yes' --dry-run | -n show what would be cloned without cloning --verbose | -v increase verbosity (may be repeated) - --quiet | -q increase verbosity (may be repeated) + --quiet | -q disable progress reporting -C DIR chdir to specified directory + +See public-inbox-clone(1) man page for --manifest, --remote-manifest, +--objstore, --project-list, --post-update-hook, --include, --exclude, +--prune, --keep-going, --jobs, --inbox-config EOF # cgit calls it `project-list', grokmirror calls it `projectslist', |