From: Yan Zhao <yan.y.zhao@intel.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<x86@kernel.org>, <jgg@nvidia.com>, <kevin.tian@intel.com>,
<iommu@lists.linux.dev>, <pbonzini@redhat.com>,
<seanjc@google.com>, <dave.hansen@linux.intel.com>,
<luto@kernel.org>, <peterz@infradead.org>, <tglx@linutronix.de>,
<mingo@redhat.com>, <bp@alien8.de>, <hpa@zytor.com>,
<corbet@lwn.net>, <joro@8bytes.org>, <will@kernel.org>,
<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>,
<yi.l.liu@intel.com>
Subject: Re: [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains
Date: Fri, 17 May 2024 13:00:38 +0800 [thread overview]
Message-ID: <Zkbkdi+hipp3/YF1@yzhao56-desk.sh.intel.com> (raw)
In-Reply-To: <20240516224442.56df5c23.alex.williamson@redhat.com>
On Thu, May 16, 2024 at 10:44:42PM -0600, Alex Williamson wrote:
> On Fri, 17 May 2024 11:11:48 +0800
> Yan Zhao <yan.y.zhao@intel.com> wrote:
>
> > On Thu, May 16, 2024 at 02:50:09PM -0600, Alex Williamson wrote:
> > > On Mon, 13 May 2024 15:11:28 +0800
> > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > >
> > > > On Fri, May 10, 2024 at 10:57:28AM -0600, Alex Williamson wrote:
> > > > > On Fri, 10 May 2024 18:31:13 +0800
> > > > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > > > >
> > > > > > On Thu, May 09, 2024 at 12:10:49PM -0600, Alex Williamson wrote:
> > > > > > > On Tue, 7 May 2024 14:21:38 +0800
> > > > > > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > ...
> > > > > > > > @@ -1683,9 +1715,14 @@ static int vfio_iommu_replay(struct vfio_iommu *iommu,
> > > > > > > > for (; n; n = rb_next(n)) {
> > > > > > > > struct vfio_dma *dma;
> > > > > > > > dma_addr_t iova;
> > > > > > > > + bool cache_flush_required;
> > > > > > > >
> > > > > > > > dma = rb_entry(n, struct vfio_dma, node);
> > > > > > > > iova = dma->iova;
> > > > > > > > + cache_flush_required = !domain->enforce_cache_coherency &&
> > > > > > > > + !dma->cache_flush_required;
> > > > > > > > + if (cache_flush_required)
> > > > > > > > + dma->cache_flush_required = true;
> > > > > > >
> > > > > > > The variable name here isn't accurate and the logic is confusing. If
> > > > > > > the domain does not enforce coherency and the mapping is not tagged as
> > > > > > > requiring a cache flush, then we need to mark the mapping as requiring
> > > > > > > a cache flush. So the variable state is something more akin to
> > > > > > > set_cache_flush_required. But all we're saving with this is a
> > > > > > > redundant set if the mapping is already tagged as requiring a cache
> > > > > > > flush, so it could really be simplified to:
> > > > > > >
> > > > > > > dma->cache_flush_required = !domain->enforce_cache_coherency;
> > > > > > Sorry about the confusion.
> > > > > >
> > > > > > If dma->cache_flush_required is set to true by a domain not enforcing cache
> > > > > > coherency, we hope it will not be reset to false by a later attaching to domain
> > > > > > enforcing cache coherency due to the lazily flushing design.
> > > > >
> > > > > Right, ok, the vfio_dma objects are shared between domains so we never
> > > > > want to set 'dma->cache_flush_required = false' due to the addition of a
> > > > > 'domain->enforce_cache_coherent == true'. So this could be:
> > > > >
> > > > > if (!dma->cache_flush_required)
> > > > > dma->cache_flush_required = !domain->enforce_cache_coherency;
> > > >
> > > > Though this code is easier for understanding, it leads to unnecessary setting of
> > > > dma->cache_flush_required to false, given domain->enforce_cache_coherency is
> > > > true at the most time.
> > >
> > > I don't really see that as an issue, but the variable name originally
> > > chosen above, cache_flush_required, also doesn't convey that it's only
> > > attempting to set the value if it wasn't previously set and is now
> > > required by a noncoherent domain.
> > Agreed, the old name is too vague.
> > What about update_to_noncoherent_required?
>
> set_noncoherent? Thanks,
>
Concise!
>
> > Then in vfio_iommu_replay(), it's like
> >
> > update_to_noncoherent_required = !domain->enforce_cache_coherency && !dma->is_noncoherent;
> > if (update_to_noncoherent_required)
> > dma->is_noncoherent = true;
> >
> > ...
> > if (update_to_noncoherent_required)
> > arch_flush_cache_phys((phys, size);
> > >
> > > > > > > It might add more clarity to just name the mapping flag
> > > > > > > dma->mapped_noncoherent.
> > > > > >
> > > > > > The dma->cache_flush_required is to mark whether pages in a vfio_dma requires
> > > > > > cache flush in the subsequence mapping into the first non-coherent domain
> > > > > > and page unpinning.
> > > > >
> > > > > How do we arrive at a sequence where we have dma->cache_flush_required
> > > > > that isn't the result of being mapped into a domain with
> > > > > !domain->enforce_cache_coherency?
> > > > Hmm, dma->cache_flush_required IS the result of being mapped into a domain with
> > > > !domain->enforce_cache_coherency.
> > > > My concern only arrives from the actual code sequence, i.e.
> > > > dma->cache_flush_required is set to true before the actual mapping.
> > > >
> > > > If we rename it to dma->mapped_noncoherent and only set it to true after the
> > > > actual successful mapping, it would lead to more code to handle flushing for the
> > > > unwind case.
> > > > Currently, flush for unwind is handled centrally in vfio_unpin_pages_remote()
> > > > by checking dma->cache_flush_required, which is true even before a full
> > > > successful mapping, so we won't miss flush on any pages that are mapped into a
> > > > non-coherent domain in a short window.
> > >
> > > I don't think we need to be so literal that "mapped_noncoherent" can
> > > only be set after the vfio_dma is fully mapped to a noncoherent domain,
> > > but also we can come up with other names for the flag. Perhaps
> > > "is_noncoherent". My suggestion was more from the perspective of what
> > > does the flag represent rather than what we intend to do as a result of
> > > the flag being set. Thanks,
> > Makes sense!
> > I like the name "is_noncoherent" :)
> >
>
next prev parent reply other threads:[~2024-05-17 5:01 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-07 6:18 [PATCH 0/5] Enforce CPU cache flush for non-coherent device assignment Yan Zhao
2024-05-07 6:19 ` [PATCH 1/5] x86/pat: Let pat_pfn_immune_to_uc_mtrr() check MTRR for untracked PAT range Yan Zhao
2024-05-07 8:26 ` Tian, Kevin
2024-05-07 9:12 ` Yan Zhao
2024-05-08 22:14 ` Alex Williamson
2024-05-09 3:36 ` Yan Zhao
2024-05-16 7:42 ` Tian, Kevin
2024-05-16 14:07 ` Sean Christopherson
2024-05-20 2:36 ` Tian, Kevin
2024-05-07 6:20 ` [PATCH 2/5] KVM: x86/mmu: Fine-grained check of whether a invalid & RAM PFN is MMIO Yan Zhao
2024-05-07 8:39 ` Tian, Kevin
2024-05-07 9:19 ` Yan Zhao
2024-05-07 6:20 ` [PATCH 3/5] x86/mm: Introduce and export interface arch_clean_nonsnoop_dma() Yan Zhao
2024-05-07 8:51 ` Tian, Kevin
2024-05-07 9:40 ` Yan Zhao
2024-05-20 14:07 ` Christoph Hellwig
2024-05-21 15:49 ` Jason Gunthorpe
2024-05-21 16:00 ` Jason Gunthorpe
2024-05-22 3:41 ` Yan Zhao
2024-05-28 6:37 ` Christoph Hellwig
2024-06-01 19:46 ` Jason Gunthorpe
2024-05-07 6:21 ` [PATCH 4/5] vfio/type1: Flush CPU caches on DMA pages in non-coherent domains Yan Zhao
2024-05-09 18:10 ` Alex Williamson
2024-05-10 10:31 ` Yan Zhao
2024-05-10 16:57 ` Alex Williamson
2024-05-13 7:11 ` Yan Zhao
2024-05-16 7:53 ` Tian, Kevin
2024-05-16 8:34 ` Tian, Kevin
2024-05-16 20:31 ` Alex Williamson
2024-05-17 17:11 ` Jason Gunthorpe
2024-05-20 2:52 ` Tian, Kevin
2024-05-21 16:07 ` Jason Gunthorpe
2024-05-21 16:21 ` Alex Williamson
2024-05-21 16:34 ` Jason Gunthorpe
2024-05-21 18:19 ` Alex Williamson
2024-05-21 18:37 ` Jason Gunthorpe
2024-05-22 6:24 ` Tian, Kevin
2024-05-22 12:29 ` Jason Gunthorpe
2024-05-22 14:43 ` Alex Williamson
2024-05-22 16:52 ` Jason Gunthorpe
2024-05-22 18:22 ` Alex Williamson
2024-05-22 23:26 ` Tian, Kevin
2024-05-22 23:32 ` Jason Gunthorpe
2024-05-22 23:40 ` Tian, Kevin
2024-05-23 14:58 ` Jason Gunthorpe
2024-05-23 22:47 ` Alex Williamson
2024-05-24 0:30 ` Tian, Kevin
2024-05-24 13:50 ` Jason Gunthorpe
2024-05-22 3:33 ` Yan Zhao
2024-05-22 3:24 ` Yan Zhao
2024-05-22 12:26 ` Jason Gunthorpe
2024-05-24 3:07 ` Yan Zhao
2024-05-16 20:50 ` Alex Williamson
2024-05-17 3:11 ` Yan Zhao
2024-05-17 4:44 ` Alex Williamson
2024-05-17 5:00 ` Yan Zhao [this message]
2024-05-07 6:22 ` [PATCH 5/5] iommufd: " Yan Zhao
2024-05-09 14:13 ` Jason Gunthorpe
2024-05-10 8:03 ` Yan Zhao
2024-05-10 13:29 ` Jason Gunthorpe
2024-05-13 7:43 ` Yan Zhao
2024-05-14 15:11 ` Jason Gunthorpe
2024-05-15 7:06 ` Yan Zhao
2024-05-15 20:43 ` Jason Gunthorpe
2024-05-16 2:32 ` Yan Zhao
2024-05-16 8:38 ` Tian, Kevin
2024-05-16 9:48 ` Yan Zhao
2024-05-17 17:04 ` Jason Gunthorpe
2024-05-20 2:45 ` Yan Zhao
2024-05-21 16:04 ` Jason Gunthorpe
2024-05-22 3:17 ` Yan Zhao
2024-05-22 6:29 ` Yan Zhao
2024-05-22 17:01 ` Jason Gunthorpe
2024-05-27 7:15 ` Yan Zhao
2024-06-01 16:48 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Zkbkdi+hipp3/YF1@yzhao56-desk.sh.intel.com \
--to=yan.y.zhao@intel.com \
--cc=alex.williamson@redhat.com \
--cc=baolu.lu@linux.intel.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=iommu@lists.linux.dev \
--cc=jgg@nvidia.com \
--cc=joro@8bytes.org \
--cc=kevin.tian@intel.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=robin.murphy@arm.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=yi.l.liu@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).