From: Anton <anton.khazan@gmail.com>
To: Kerin Millar <kfm@plushkava.net>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>, netfilter@vger.kernel.org
Subject: Re: Is there an efficient way to delete multiple elements from a set?
Date: Fri, 2 Feb 2024 02:36:54 +0200 [thread overview]
Message-ID: <CAJE3=w=zOc_yt5WeJA9XuDfStLe9KArnp5t5N-_Fw4kDFPZ+3w@mail.gmail.com> (raw)
In-Reply-To: <f1787459-57fb-43e9-b04d-bad0e296dc80@app.fastmail.com>
At this point I have moved on from the idea of aggregating multiple ip
blocks in a single set because I want my project to work well on
embedded devices which have limited memory and CPU power, and, as I
discovered by trial and error, currently adding elements is not an
option for this application because of high memory consumption and
other performance issues (which I reported in Bug 1735).
I figured that nftables provides a way to swap sets without offline
time, and it's good to have this confirmed. That said, it would be
really nice to have a swap command for nftables sets. This would be a
quality of life improvement that I'm sure many people would
appreciate.
Thanks everyone for your help.
On Thu, Feb 1, 2024 at 2:21 PM Kerin Millar <kfm@plushkava.net> wrote:
>
> Hi Pablo,
>
> On Thu, 1 Feb 2024, at 9:41 AM, Pablo Neira Ayuso wrote:
> > On Wed, Jan 31, 2024 at 10:14:41AM +0200, Anton wrote:
> >> Hello, I've been experimenting with nftables sets for the purpose of
> >> geoip blocking. Let's say I'd like to add ip blocks for multiple
> >> countries to a blacklist or to a whitelist. Perhaps the most efficient
> >> way to do that would be by combining all required ip blocks in one set
> >> (for each family). However since country ip blocks are a moving
> >> target, I would need to regularly refresh parts of that set. My idea
> >> was to delete all ip addresses corresponding to an ip block from the
> >> set and then add the updated ip block. The problem is, this is very
> >> slow. While adding an ip block takes (in my VM) 0.09s, deleting all
> >> ip's from that same block takes 14.5s.
> >>
> >> This is how I'm doing the deletion and the time measurement:
> >> printf '%s\n' "delete element inet test testset { $(cat test.set) };"
> >> | /usr/bin/time -f %es nft -f -
> >>
> >> (the test.set file stores a comma-separated list of subnets)
> >>
> >> Is there a more efficient way to do this? I could of course flush the
> >> set and rebuild it every time I need to update some part of it, but I
> >> thought I'd ask before deciding to implement that.
> >
> > It is possible to flush the set and fill up with content again:
> >
> > flush set inet test testset
> > add element inet test testset { ... }
>
> I figured that this approach was already on the cards ("I could of course flush the set and rebuild"), though it is possible that Anton wasn't aware that it can be done atomically.
>
> As far I understand the original post, he is aggregating multiple country blocklists to form a single set - a blacklist or a whitelist, as it was put. That would explain his interest in deleting subsets to begin with. The flush/rebuild approach may well be faster but it would also require reading in all of the subsets again - even those that haven't changed. Perhaps not a big deal in the greater scheme of things but it does make me wonder whether there's room for improvement as far as deletions go. I might test this on the next occasion that I'm experimenting with set behaviour.
>
> --
> Kerin Millar
prev parent reply other threads:[~2024-02-02 0:37 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-31 8:14 Is there an efficient way to delete multiple elements from a set? Anton
2024-01-31 8:21 ` Reindl Harald
2024-01-31 8:27 ` Anton
2024-01-31 12:20 ` Kerin Millar
2024-01-31 22:13 ` Anton
2024-01-31 22:39 ` Kerin Millar
2024-01-31 23:11 ` Anton
2024-02-01 9:41 ` Pablo Neira Ayuso
2024-02-01 9:42 ` Pablo Neira Ayuso
2024-02-01 10:24 ` Jozsef Kadlecsik
2024-02-01 12:20 ` Kerin Millar
2024-02-02 0:36 ` Anton [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJE3=w=zOc_yt5WeJA9XuDfStLe9KArnp5t5N-_Fw4kDFPZ+3w@mail.gmail.com' \
--to=anton.khazan@gmail.com \
--cc=kfm@plushkava.net \
--cc=netfilter@vger.kernel.org \
--cc=pablo@netfilter.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).