From: Jason Gunthorpe <jgg@nvidia.com>
To: Will Deacon <will@kernel.org>
Cc: iommu@lists.linux.dev, Joerg Roedel <joro@8bytes.org>,
linux-arm-kernel@lists.infradead.org,
Robin Murphy <robin.murphy@arm.com>,
Lu Baolu <baolu.lu@linux.intel.com>,
Jean-Philippe Brucker <jean-philippe@linaro.org>,
Joerg Roedel <jroedel@suse.de>, Moritz Fischer <mdf@kernel.org>,
Moritz Fischer <moritzf@google.com>,
Michael Shavit <mshavit@google.com>,
Nicolin Chen <nicolinc@nvidia.com>,
patches@lists.linux.dev,
Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
Mostafa Saleh <smostafa@google.com>,
Zhangfei Gao <zhangfei.gao@linaro.org>
Subject: Re: [PATCH v5 01/17] iommu/arm-smmu-v3: Make STE programming independent of the callers
Date: Thu, 29 Feb 2024 09:57:33 -0400 [thread overview]
Message-ID: <20240229135733.GD9179@nvidia.com> (raw)
In-Reply-To: <20240227124318.GA14089@willie-the-truck>
On Tue, Feb 27, 2024 at 12:43:18PM +0000, Will Deacon wrote:
> On Fri, Feb 23, 2024 at 11:18:41AM -0400, Jason Gunthorpe wrote:
> > On Thu, Feb 22, 2024 at 05:43:46PM +0000, Will Deacon wrote:
> > > On Wed, Feb 21, 2024 at 10:08:18AM -0400, Jason Gunthorpe wrote:
> > > > On Wed, Feb 21, 2024 at 01:49:23PM +0000, Will Deacon wrote:
> > > >
> > > > > Very roughly, yes, although I'd go further and just return a bitmap of
> > > > > used qwords instead of tracking these bits. Basically, we could have some
> > > > > #defines saying which qwords are used by which configs,
> > > >
> > > > I don't think this will work well for CD's EPD0 case..
> > > >
> > > > static void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits)
> > > > {
> > > > used_bits[0] = cpu_to_le64(CTXDESC_CD_0_V);
> > > > if (!(ent[0] & cpu_to_le64(CTXDESC_CD_0_V)))
> > > > return;
> > > > memset(used_bits, 0xFF, sizeof(struct arm_smmu_cd));
> > > >
> > > > /* EPD0 means T0SZ/TG0/IR0/OR0/SH0/TTB0 are IGNORED */
> > > > if (ent[0] & cpu_to_le64(CTXDESC_CD_0_TCR_EPD0)) {
> > > > used_bits[0] &= ~cpu_to_le64(
> > > > CTXDESC_CD_0_TCR_T0SZ | CTXDESC_CD_0_TCR_TG0 |
> > > > CTXDESC_CD_0_TCR_IRGN0 | CTXDESC_CD_0_TCR_ORGN0 |
> > > > CTXDESC_CD_0_TCR_SH0);
> > > > used_bits[1] &= ~cpu_to_le64(CTXDESC_CD_1_TTB0_MASK);
> > > > }
> > > > }
> > >
> > > Please can you explain more about the issue here? I know what EPDx are,
> > > but I'm not understanding why they're problematic. This presumably
> > > involves a hitless transition to/from an aborting CD?
> >
> > When a process using SVA exits uncleanly the MM is released so the
> > SMMU HW must stop chasing the page table pointers since all that
> > memory will be freed.
> >
> > However, in an unclean exit we can't control the order of shutdown so
> > something like uacce or RDMA may not have quieted the DMA device yet.
> >
> > So there is a period during shutdown where the mm has been released
> > and the device is doing DMA, the desire is that the DMA continue to be
> > handled as a PRI and the SW will return failure for all PRI requests.
> >
> > Specifically we do not want to trigger any dmesg log events during
> > this condition.
>
> Curious, but why is it problematic to log events? As you say, it's an
> "unclean" exit, so it doesn't seem that unreasonable to me.
Well, I would defer to Jean-Philippe, but I can understand the
logic. A user ctrl-c's their application it is not nice to get some
dmesg logs from that.
I recall he felt strongly about this, we had some discussion about it
related to the mmu notifiers back when the iommu drivers were all
updated to the new notifier API I built...
> > Jean-Philippe came up with this solution where we hitlessly use EPD0
> > in release to allow the mm to release the page table while continuing
> > to use the PRI flow.
> >
> > So it is going from a "SVA domain with a page table" to a "SVA domain
> > without a page table but EPD0 set", hitlessly.
>
> Ok, and so the reason this adds complexity is because the set of used
> bits/qwords changes based on something other than the cfg?
There is no cfg for CD entries? I think it is the same issue as SHCFG,
qw1 of CD is not neatly split and qw0/1 are both changing for the EPD0
case - we also zero the unused TCR/TTB.
> I think it's a pretty weak argument for field vs qwords, but it's a
> good counter-example to my naive approach of per-config masks, so
> thanks.
We could do EPD0 just by editting in place, it would be easy to code,
but the point of this design was to never edit a descriptor in place.
Jason
next prev parent reply other threads:[~2024-02-29 13:57 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-06 15:12 [PATCH v5 00/17] Update SMMUv3 to the modern iommu API (part 1/3) Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 01/17] iommu/arm-smmu-v3: Make STE programming independent of the callers Jason Gunthorpe
2024-02-15 13:49 ` Will Deacon
2024-02-15 16:01 ` Jason Gunthorpe
2024-02-15 18:42 ` Robin Murphy
2024-02-15 20:11 ` Robin Murphy
2024-02-16 16:28 ` Will Deacon
2024-02-15 21:17 ` Jason Gunthorpe
2024-02-21 13:49 ` Will Deacon
2024-02-21 14:08 ` Jason Gunthorpe
2024-02-21 16:19 ` Michael Shavit
2024-02-21 16:52 ` Michael Shavit
2024-02-21 17:06 ` Jason Gunthorpe
2024-02-22 17:43 ` Will Deacon
2024-02-23 15:18 ` Jason Gunthorpe
2024-02-27 12:43 ` Will Deacon
2024-02-29 13:57 ` Jason Gunthorpe [this message]
2024-02-06 15:12 ` [PATCH v5 02/17] iommu/arm-smmu-v3: Consolidate the STE generation for abort/bypass Jason Gunthorpe
2024-02-15 17:27 ` Robin Murphy
2024-02-22 17:40 ` Will Deacon
2024-02-23 18:53 ` Jason Gunthorpe
2024-02-27 10:50 ` Will Deacon
2024-02-06 15:12 ` [PATCH v5 03/17] iommu/arm-smmu-v3: Move arm_smmu_rmr_install_bypass_ste() Jason Gunthorpe
2024-02-13 15:37 ` Mostafa Saleh
2024-02-13 16:16 ` Jason Gunthorpe
2024-02-13 16:46 ` Mostafa Saleh
2024-02-15 19:01 ` Robin Murphy
2024-02-15 21:18 ` Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 04/17] iommu/arm-smmu-v3: Move the STE generation for S1 and S2 domains into functions Jason Gunthorpe
2024-02-16 17:12 ` Jason Gunthorpe
2024-02-16 17:39 ` Will Deacon
2024-02-16 17:58 ` Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 05/17] iommu/arm-smmu-v3: Build the whole STE in arm_smmu_make_s2_domain_ste() Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 06/17] iommu/arm-smmu-v3: Hold arm_smmu_asid_lock during all of attach_dev Jason Gunthorpe
2024-02-13 15:38 ` Mostafa Saleh
2024-02-13 16:18 ` Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 07/17] iommu/arm-smmu-v3: Compute the STE only once for each master Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 08/17] iommu/arm-smmu-v3: Do not change the STE twice during arm_smmu_attach_dev() Jason Gunthorpe
2024-02-13 15:40 ` Mostafa Saleh
2024-02-13 16:26 ` Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 09/17] iommu/arm-smmu-v3: Put writing the context descriptor in the right order Jason Gunthorpe
2024-02-13 15:42 ` Mostafa Saleh
2024-02-13 17:50 ` Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 10/17] iommu/arm-smmu-v3: Pass smmu_domain to arm_enable/disable_ats() Jason Gunthorpe
2024-02-13 15:43 ` Mostafa Saleh
2024-02-06 15:12 ` [PATCH v5 11/17] iommu/arm-smmu-v3: Remove arm_smmu_master->domain Jason Gunthorpe
2024-02-13 15:45 ` Mostafa Saleh
2024-02-13 16:37 ` Jason Gunthorpe
2024-02-13 17:00 ` Mostafa Saleh
2024-02-06 15:12 ` [PATCH v5 12/17] iommu/arm-smmu-v3: Check that the RID domain is S1 in SVA Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 13/17] iommu/arm-smmu-v3: Add a global static IDENTITY domain Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 14/17] iommu/arm-smmu-v3: Add a global static BLOCKED domain Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 15/17] iommu/arm-smmu-v3: Use the identity/blocked domain during release Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 16/17] iommu/arm-smmu-v3: Pass arm_smmu_domain and arm_smmu_device to finalize Jason Gunthorpe
2024-02-06 15:12 ` [PATCH v5 17/17] iommu/arm-smmu-v3: Convert to domain_alloc_paging() Jason Gunthorpe
2024-02-07 5:27 ` [PATCH v5 00/17] Update SMMUv3 to the modern iommu API (part 1/3) Nicolin Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240229135733.GD9179@nvidia.com \
--to=jgg@nvidia.com \
--cc=baolu.lu@linux.intel.com \
--cc=iommu@lists.linux.dev \
--cc=jean-philippe@linaro.org \
--cc=joro@8bytes.org \
--cc=jroedel@suse.de \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=mdf@kernel.org \
--cc=moritzf@google.com \
--cc=mshavit@google.com \
--cc=nicolinc@nvidia.com \
--cc=patches@lists.linux.dev \
--cc=robin.murphy@arm.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=smostafa@google.com \
--cc=will@kernel.org \
--cc=zhangfei.gao@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).