* fanotify and files being moved or deleted
@ 2024-05-21 1:03 Jonathan Gilbert
2024-05-21 3:58 ` Amir Goldstein
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Gilbert @ 2024-05-21 1:03 UTC (permalink / raw
To: linux-fsdevel
Hello :-)
I want to use fanotify to construct a best-effort log of changes to
the filesystem over time. It would be super useful if events like
FAN_MOVE_SELF and FAN_DELETE_SELF could report the path that the file
_was_ at just prior to the event. Reporting an FID value is of limited
use, because even if it still exists, looking up the name (e.g. by
open_by_handle_at, the way fatrace does) will only reveal the new name
after a FAN_MOVE_SELF -- and after a FAN_DELETE_SELF, the file no
longer has any path!
I understand that in terms of a strictly accurate reconstruction of
changes over time, fanotify events are of limited use, because they
aren't guaranteed to be ordered and from what I have read it seems it
is possible for some changes to "slip through" from time to time. But,
this is not a problem for my use case.
I have no idea what things are available where in the kernel code that
generates these events, but in the course of writing the code that
reads the event data that gets sent to an fanotify fd, I was thinking
that the simplest way to achieve this would be for FAN_MOVE_SELF and
FAN_DELETE_SELF events to have associated info structures with paths
in them. FAN_DELETE_SELF could provide an info structure with the path
that just got unlinked, and FAN_MOVE_SELF could provide two info
structures, one for the old path and one for the new.
Of course, it is possible that this information isn't currently
available at the spot where the events are being generated!
But, this would be immensely useful to my use case. Any possibility?
Thanks,
Jonathan Gilbert
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-21 1:03 fanotify and files being moved or deleted Jonathan Gilbert
@ 2024-05-21 3:58 ` Amir Goldstein
2024-05-21 16:03 ` Jonathan Gilbert
0 siblings, 1 reply; 7+ messages in thread
From: Amir Goldstein @ 2024-05-21 3:58 UTC (permalink / raw
To: Jonathan Gilbert; +Cc: linux-fsdevel, Jan Kara
On Tue, May 21, 2024 at 4:04 AM Jonathan Gilbert <logic@deltaq.org> wrote:
>
> Hello :-)
>
> I want to use fanotify to construct a best-effort log of changes to
> the filesystem over time. It would be super useful if events like
> FAN_MOVE_SELF and FAN_DELETE_SELF could report the path that the file
> _was_ at just prior to the event. Reporting an FID value is of limited
> use, because even if it still exists, looking up the name (e.g. by
> open_by_handle_at, the way fatrace does) will only reveal the new name
> after a FAN_MOVE_SELF -- and after a FAN_DELETE_SELF, the file no
> longer has any path!
>
> I understand that in terms of a strictly accurate reconstruction of
> changes over time, fanotify events are of limited use, because they
> aren't guaranteed to be ordered and from what I have read it seems it
> is possible for some changes to "slip through" from time to time. But,
> this is not a problem for my use case.
>
> I have no idea what things are available where in the kernel code that
> generates these events, but in the course of writing the code that
> reads the event data that gets sent to an fanotify fd, I was thinking
> that the simplest way to achieve this would be for FAN_MOVE_SELF and
> FAN_DELETE_SELF events to have associated info structures with paths
> in them. FAN_DELETE_SELF could provide an info structure with the path
> that just got unlinked, and FAN_MOVE_SELF could provide two info
> structures, one for the old path and one for the new.
>
> Of course, it is possible that this information isn't currently
> available at the spot where the events are being generated!
This statement is correct for FAN_DELETE_SELF.
>
> But, this would be immensely useful to my use case. Any possibility?
>
FAN_DELETE emitted for every unlink() has the unlinked file name -
a file can have many names (i.e. hardlinks) will almost always come
before the final FAN_DELETE_SELF, which is emitted only when st_nlink
drops to zero.and last file reference is closed.
I say almost always, because moving over a file, can also unlink it,
so either FAN_DELETE or FAN_RENAME should be observed
before FAN_DELETE_SELF and those should be enough for your
purpose.
FAN_MOVE_SELF could in theory have info about source and target
file names, same as FAN_RENAME because it is being generated
within the exact same fsnotify_move() hook, but that's the reason
that FAN_RENAME is enough for your purpose.
FAN_MOVE_SELF intentionally does not carry this information
so that watchers of FAN_MOVE_SELF could get all the move events
merged and get a single move event with the FID after a series of
many renames.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-21 3:58 ` Amir Goldstein
@ 2024-05-21 16:03 ` Jonathan Gilbert
2024-05-21 17:13 ` Amir Goldstein
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Gilbert @ 2024-05-21 16:03 UTC (permalink / raw
To: linux-fsdevel
Hmm, okay. In earlier testing, I must have had a bug because I wasn't
seeing filenames for FAN_MOVE or FAN_DELETE. But, my code is more
robust now, and when I switch it to those events I do see filenames --
but not paths. Looks like I can do the open_by_handle_at trick on the
fd in the main FAN_MOVED_FROM, FAN_MOVED_TO and FAN_DELETE event and
that'll give me the directory path and then I can combine it with the
file name in the info structure?
Are FAN_MOVED_FROM and FAN_MOVED_TO guaranteed to be emitted
atomically, or is there a possibility they could be split up by other
events? If so, could there be multiple overlapping
FAN_MOVED_FROM/FAN_MOVED_TO pairs under the right circumstances??
One other thing I'm seeing is that in enumerating the mount table in
order to mark things, I find multiple entries with the same fsid.
These seem to be cases where an item _inside another mount_ has been
used as the device for a mount. One example is /boot/grub, which is
mounted from /boot/efi/grub, where /boot/efi is itself mounted from a
physical device. When enumerating the mounts, both of these return the
same fsid from fstatfs. There is at least one other with such a
collision, though it does not appear in fstab. Both the root
filesystem / and a filesystem mounted at
/var/snap/firefox/common/host-unspell return the same fsid. Does this
mean that there is simply a category of event that cannot be
guaranteed to return the correct path, because the only identifying
information, the fsid, isn't guaranteed to be unique? Or is there a
way to resolve this?
Thanks,
Jonathan Gilbert
On Mon, May 20, 2024 at 10:58 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, May 21, 2024 at 4:04 AM Jonathan Gilbert <logic@deltaq.org> wrote:
> >
> > Hello :-)
> >
> > I want to use fanotify to construct a best-effort log of changes to
> > the filesystem over time. It would be super useful if events like
> > FAN_MOVE_SELF and FAN_DELETE_SELF could report the path that the file
> > _was_ at just prior to the event. Reporting an FID value is of limited
> > use, because even if it still exists, looking up the name (e.g. by
> > open_by_handle_at, the way fatrace does) will only reveal the new name
> > after a FAN_MOVE_SELF -- and after a FAN_DELETE_SELF, the file no
> > longer has any path!
> >
> > I understand that in terms of a strictly accurate reconstruction of
> > changes over time, fanotify events are of limited use, because they
> > aren't guaranteed to be ordered and from what I have read it seems it
> > is possible for some changes to "slip through" from time to time. But,
> > this is not a problem for my use case.
> >
> > I have no idea what things are available where in the kernel code that
> > generates these events, but in the course of writing the code that
> > reads the event data that gets sent to an fanotify fd, I was thinking
> > that the simplest way to achieve this would be for FAN_MOVE_SELF and
> > FAN_DELETE_SELF events to have associated info structures with paths
> > in them. FAN_DELETE_SELF could provide an info structure with the path
> > that just got unlinked, and FAN_MOVE_SELF could provide two info
> > structures, one for the old path and one for the new.
> >
> > Of course, it is possible that this information isn't currently
> > available at the spot where the events are being generated!
>
> This statement is correct for FAN_DELETE_SELF.
>
> >
> > But, this would be immensely useful to my use case. Any possibility?
> >
>
> FAN_DELETE emitted for every unlink() has the unlinked file name -
> a file can have many names (i.e. hardlinks) will almost always come
> before the final FAN_DELETE_SELF, which is emitted only when st_nlink
> drops to zero.and last file reference is closed.
>
> I say almost always, because moving over a file, can also unlink it,
> so either FAN_DELETE or FAN_RENAME should be observed
> before FAN_DELETE_SELF and those should be enough for your
> purpose.
>
> FAN_MOVE_SELF could in theory have info about source and target
> file names, same as FAN_RENAME because it is being generated
> within the exact same fsnotify_move() hook, but that's the reason
> that FAN_RENAME is enough for your purpose.
>
> FAN_MOVE_SELF intentionally does not carry this information
> so that watchers of FAN_MOVE_SELF could get all the move events
> merged and get a single move event with the FID after a series of
> many renames.
>
> Thanks,
> Amir.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-21 16:03 ` Jonathan Gilbert
@ 2024-05-21 17:13 ` Amir Goldstein
2024-05-21 21:09 ` Jonathan Gilbert
0 siblings, 1 reply; 7+ messages in thread
From: Amir Goldstein @ 2024-05-21 17:13 UTC (permalink / raw
To: Jonathan Gilbert; +Cc: linux-fsdevel, Jan Kara
On Tue, May 21, 2024 at 7:04 PM Jonathan Gilbert <logic@deltaq.org> wrote:
>
> Hmm, okay. In earlier testing, I must have had a bug because I wasn't
> seeing filenames for FAN_MOVE or FAN_DELETE. But, my code is more
> robust now, and when I switch it to those events I do see filenames --
> but not paths. Looks like I can do the open_by_handle_at trick on the
> fd in the main FAN_MOVED_FROM, FAN_MOVED_TO and FAN_DELETE event and
> that'll give me the directory path and then I can combine it with the
> file name in the info structure?
>
Yes. That's the idea.
open_by_handle_at() with the parent's file handle is guaranteed to return
a fd with "connected" path (i.e. known path), unless that directory was deleted.
Note that you will be combining the *current* directory path with the *past*
filename, so you may get a path that never existed in reality, but as you wrote
fanotify is not meant for keeping historical records of the filesystem
namespace.
> Are FAN_MOVED_FROM and FAN_MOVED_TO guaranteed to be emitted
> atomically, or is there a possibility they could be split up by other
> events? If so, could there be multiple overlapping
> FAN_MOVED_FROM/FAN_MOVED_TO pairs under the right circumstances??
You are looking for FAN_RENAME, the new event that combines
information from FAN_MOVED_FROM/FAN_MOVED_TO.
Unlike FAN_MOVED_FROM/FAN_MOVED_TO, FAN_RENAME cannot
be merged with other events like FAN_CREATE/FAN_DELETE because
it does not carry the same type of information.
>
> One other thing I'm seeing is that in enumerating the mount table in
> order to mark things, I find multiple entries with the same fsid.
> These seem to be cases where an item _inside another mount_ has been
> used as the device for a mount. One example is /boot/grub, which is
> mounted from /boot/efi/grub, where /boot/efi is itself mounted from a
> physical device.
Yes, this is called a bind mount, which can be generated using
mount --bind /boot/efi/grub /boot/grub
> When enumerating the mounts, both of these return the
> same fsid from fstatfs. There is at least one other with such a
> collision, though it does not appear in fstab. Both the root
> filesystem / and a filesystem mounted at
> /var/snap/firefox/common/host-unspell return the same fsid. Does this
> mean that there is simply a category of event that cannot be
> guaranteed to return the correct path, because the only identifying
> information, the fsid, isn't guaranteed to be unique? Or is there a
> way to resolve this?
That depends on how you are setting up your watches.
Are you setting up FAN_MARK_FILESYSTEM watches on all
mounted filesystem?
Note that not all filesystems support NFS export file handles,
so not all filesystem support being watched with FAN_REPORT_FID and
FAN_MARK_FILESYSTEM.
If, for example you care about reconstructing changing over certain
paths (e.g. /home), you can keep an open mount_fd of that path when you
start watching it and keep it in a hash table with fsid as the key
(that is how fsnotifywatch does it [1]) and then use that mount_fd whenever
you want to decode the path from a parent file handle.
If /home is a bind mount from, say, /data/home/ and you are watching
both /home and /data, you will need to figure out that they are the same
underlying fs and use a mount_fd of /data.
Then, when you get a file handles of, say, /home/docs, it will be resolved
to path /data/home/docs.
If you try to resolve a file handle of /data/archive using mount_fd that
was opened from /home, the path you will observe is "/", so this will
not be useful to constructing history.
So the answer really depends on what exactly are the requirements
of your tool.
Thanks,
Amir.
[1] https://github.com/inotify-tools/inotify-tools/blob/master/libinotifytools/src/inotifytools.cpp#L1343
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-21 17:13 ` Amir Goldstein
@ 2024-05-21 21:09 ` Jonathan Gilbert
2024-05-22 5:21 ` Amir Goldstein
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Gilbert @ 2024-05-21 21:09 UTC (permalink / raw
To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara
On Tue, May 21, 2024 at 12:13 PM Amir Goldstein <amir73il@gmail.com> wrote:
> Note that you will be combining the *current* directory path with the *past*
> filename, so you may get a path that never existed in reality, but as you wrote
> fanotify is not meant for keeping historical records of the filesystem
> namespace.
And, in practice, this is almost certainly an edge case, because the
vast majority of user-driven activity will be on a time scale such
that the directory path hasn't had a chance to change since the file
event was generated, I think? It's more a, "You can't technically
guarantee 100% consistency", than a, "You should expect consistency
errors regularly," sort of thing?
> > Are FAN_MOVED_FROM and FAN_MOVED_TO guaranteed to be emitted
> > atomically, or is there a possibility they could be split up by other
> > events? If so, could there be multiple overlapping
> > FAN_MOVED_FROM/FAN_MOVED_TO pairs under the right circumstances??
>
> You are looking for FAN_RENAME, the new event that combines
> information from FAN_MOVED_FROM/FAN_MOVED_TO.
Ah, excellent, this is in fact exactly what I needed.
> > One other thing I'm seeing is that in enumerating the mount table in
> > order to mark things, I find multiple entries with the same fsid.
> > These seem to be cases where an item _inside another mount_ has been
> > used as the device for a mount. One example is /boot/grub, which is
> > mounted from /boot/efi/grub, where /boot/efi is itself mounted from a
> > physical device.
>
> Yes, this is called a bind mount, which can be generated using
> mount --bind /boot/efi/grub /boot/grub
Huh, okay. I did some reading about this, and it seems that regardless
of the order in which things are done, quite simply it is always
possible for the same underlying filesystem to be mounted in multiple
places. When this happens, there's no way to tell which mount changes
were made through, but also, it isn't relevant because the same change
is visible through all such overlapping mounts simultaneously. So,
depending on the exact semantics I need, I need to either decide which
mount I'm going to pick as being the most meaningful for the event, or
alternately figure out *all* of the mounts to which the event applies.
> > When enumerating the mounts, both of these return the
> > same fsid from fstatfs. There is at least one other with such a
> > collision, though it does not appear in fstab. Both the root
> > filesystem / and a filesystem mounted at
> > /var/snap/firefox/common/host-unspell return the same fsid. Does this
> > mean that there is simply a category of event that cannot be
> > guaranteed to return the correct path, because the only identifying
> > information, the fsid, isn't guaranteed to be unique? Or is there a
> > way to resolve this?
>
> That depends on how you are setting up your watches.
> Are you setting up FAN_MARK_FILESYSTEM watches on all
> mounted filesystem?
That is the intent, yep.
> Note that not all filesystems support NFS export file handles,
> so not all filesystem support being watched with FAN_REPORT_FID and
> FAN_MARK_FILESYSTEM.
Good to know. My initial use case for the code is on my own personal
machine, which is a pretty much stock Ubuntu 24.04 LTS system with the
default ZFS layout. In my testing thus far, it looks like the kinds of
events I'm looking for are in fact captured.
> If, for example you care about reconstructing changing over certain
> paths (e.g. /home), you can keep an open mount_fd of that path when you
> start watching it and keep it in a hash table with fsid as the key
> (that is how fsnotifywatch does it [1]) and then use that mount_fd whenever
> you want to decode the path from a parent file handle.
Yeah, my starting point for development was the fatrace source code
which also does this.
> If /home is a bind mount from, say, /data/home/ and you are watching
> both /home and /data, you will need to figure out that they are the same
> underlying fs and use a mount_fd of /data.
My current plan is to discard any mounts which specify a root that is
a subpath of another mount, and in the case of multiple mounts of the
same root, pick one to move forward with (with hints from
configuration) and only mark that one.
This is starting to feel like all the bits are coming together. Thanks
so much for your insight and input :-)
Jonathan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-21 21:09 ` Jonathan Gilbert
@ 2024-05-22 5:21 ` Amir Goldstein
2024-05-22 5:26 ` Amir Goldstein
0 siblings, 1 reply; 7+ messages in thread
From: Amir Goldstein @ 2024-05-22 5:21 UTC (permalink / raw
To: Jonathan Gilbert; +Cc: linux-fsdevel, Jan Kara
On Wed, May 22, 2024 at 12:09 AM Jonathan Gilbert <logic@deltaq.org> wrote:
>
> On Tue, May 21, 2024 at 12:13 PM Amir Goldstein <amir73il@gmail.com> wrote:
> > Note that you will be combining the *current* directory path with the *past*
> > filename, so you may get a path that never existed in reality, but as you wrote
> > fanotify is not meant for keeping historical records of the filesystem
> > namespace.
>
> And, in practice, this is almost certainly an edge case, because the
> vast majority of user-driven activity will be on a time scale such
> that the directory path hasn't had a chance to change since the file
> event was generated, I think? It's more a, "You can't technically
> guarantee 100% consistency", than a, "You should expect consistency
> errors regularly," sort of thing?
>
I guess so, but I would not call this "consistency errors", because
fanotify does not claim to report a "consistent" state or a snapshot,
although it could be used to construct an "eventual consistent" state
by examining the information in the events.
- You should expect out of order events.
- You should expect that the object information in the event
that is described with dfid+name+fid may not exist with the
same description at the time that the event is reported.
> > > Are FAN_MOVED_FROM and FAN_MOVED_TO guaranteed to be emitted
> > > atomically, or is there a possibility they could be split up by other
> > > events? If so, could there be multiple overlapping
> > > FAN_MOVED_FROM/FAN_MOVED_TO pairs under the right circumstances??
> >
> > You are looking for FAN_RENAME, the new event that combines
> > information from FAN_MOVED_FROM/FAN_MOVED_TO.
>
> Ah, excellent, this is in fact exactly what I needed.
>
> > > One other thing I'm seeing is that in enumerating the mount table in
> > > order to mark things, I find multiple entries with the same fsid.
> > > These seem to be cases where an item _inside another mount_ has been
> > > used as the device for a mount. One example is /boot/grub, which is
> > > mounted from /boot/efi/grub, where /boot/efi is itself mounted from a
> > > physical device.
> >
> > Yes, this is called a bind mount, which can be generated using
> > mount --bind /boot/efi/grub /boot/grub
>
> Huh, okay. I did some reading about this, and it seems that regardless
> of the order in which things are done, quite simply it is always
> possible for the same underlying filesystem to be mounted in multiple
> places. When this happens, there's no way to tell which mount changes
> were made through, but also, it isn't relevant because the same change
> is visible through all such overlapping mounts simultaneously. So,
> depending on the exact semantics I need, I need to either decide which
> mount I'm going to pick as being the most meaningful for the event, or
> alternately figure out *all* of the mounts to which the event applies.
>
> > > When enumerating the mounts, both of these return the
> > > same fsid from fstatfs. There is at least one other with such a
> > > collision, though it does not appear in fstab. Both the root
> > > filesystem / and a filesystem mounted at
> > > /var/snap/firefox/common/host-unspell return the same fsid. Does this
> > > mean that there is simply a category of event that cannot be
> > > guaranteed to return the correct path, because the only identifying
> > > information, the fsid, isn't guaranteed to be unique? Or is there a
> > > way to resolve this?
> >
> > That depends on how you are setting up your watches.
> > Are you setting up FAN_MARK_FILESYSTEM watches on all
> > mounted filesystem?
>
> That is the intent, yep.
>
> > Note that not all filesystems support NFS export file handles,
> > so not all filesystem support being watched with FAN_REPORT_FID and
> > FAN_MARK_FILESYSTEM.
>
> Good to know. My initial use case for the code is on my own personal
> machine, which is a pretty much stock Ubuntu 24.04 LTS system with the
> default ZFS layout. In my testing thus far, it looks like the kinds of
> events I'm looking for are in fact captured.
>
It's good to know that someone is testing fanotify on ZFS ;-)
> > If, for example you care about reconstructing changing over certain
> > paths (e.g. /home), you can keep an open mount_fd of that path when you
> > start watching it and keep it in a hash table with fsid as the key
> > (that is how fsnotifywatch does it [1]) and then use that mount_fd whenever
> > you want to decode the path from a parent file handle.
>
> Yeah, my starting point for development was the fatrace source code
> which also does this.
>
Cool. I wasn't aware that fatrace has adopted FAN_REPORT_FID a long time
before I added support to inotify-tools :)
> > If /home is a bind mount from, say, /data/home/ and you are watching
> > both /home and /data, you will need to figure out that they are the same
> > underlying fs and use a mount_fd of /data.
>
> My current plan is to discard any mounts which specify a root that is
> a subpath of another mount, and in the case of multiple mounts of the
> same root, pick one to move forward with (with hints from
> configuration) and only mark that one.
>
You can also use open_by_handle() to determine if one mount
is a subtree of another.
if you have two fds and two different fhandles from the root of two mounts
of the same fsid, only one of these commands will result in an fd with
non empty path:
fd2inmount1 = open_by_handle_at(mount1_fd, fhandle2, O_PATH);
fd1inmount2 = open_by_handle_at(mount2_fd, fhandle1, O_PATH);
So you can throw away subtree mounts of the same fsid keeping
only one mount_fd per fsid as you traverse the mounts.
> This is starting to feel like all the bits are coming together. Thanks
> so much for your insight and input :-)
>
You're welcome.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fanotify and files being moved or deleted
2024-05-22 5:21 ` Amir Goldstein
@ 2024-05-22 5:26 ` Amir Goldstein
0 siblings, 0 replies; 7+ messages in thread
From: Amir Goldstein @ 2024-05-22 5:26 UTC (permalink / raw
To: Jonathan Gilbert; +Cc: linux-fsdevel, Jan Kara
> > > If /home is a bind mount from, say, /data/home/ and you are watching
> > > both /home and /data, you will need to figure out that they are the same
> > > underlying fs and use a mount_fd of /data.
> >
> > My current plan is to discard any mounts which specify a root that is
> > a subpath of another mount, and in the case of multiple mounts of the
> > same root, pick one to move forward with (with hints from
> > configuration) and only mark that one.
> >
>
> You can also use open_by_handle() to determine if one mount
> is a subtree of another.
>
> if you have two fds and two different fhandles from the root of two mounts
> of the same fsid, only one of these commands will result in an fd with
> non empty path:
> fd2inmount1 = open_by_handle_at(mount1_fd, fhandle2, O_PATH);
> fd1inmount2 = open_by_handle_at(mount2_fd, fhandle1, O_PATH);
>
> So you can throw away subtree mounts of the same fsid keeping
> only one mount_fd per fsid as you traverse the mounts.
>
Well ,that's incorrect.
You may have bind mount of non-overlapping subtrees
and you may not have the root mount at all in your mount namespace.
In that case, you can always keep all mount_fd's of a certain fsid and try to
resolve the file handles in each one. Not optimal, but this is the information
we have in events at the moment.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-05-22 5:26 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-21 1:03 fanotify and files being moved or deleted Jonathan Gilbert
2024-05-21 3:58 ` Amir Goldstein
2024-05-21 16:03 ` Jonathan Gilbert
2024-05-21 17:13 ` Amir Goldstein
2024-05-21 21:09 ` Jonathan Gilbert
2024-05-22 5:21 ` Amir Goldstein
2024-05-22 5:26 ` Amir Goldstein
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).