From: Yu Zhao <yuzhao@google.com>
To: James Houghton <jthoughton@google.com>
Cc: David Matlack <dmatlack@google.com>,
Andrew Morton <akpm@linux-foundation.org>,
Paolo Bonzini <pbonzini@redhat.com>,
Marc Zyngier <maz@kernel.org>,
Oliver Upton <oliver.upton@linux.dev>,
Sean Christopherson <seanjc@google.com>,
Jonathan Corbet <corbet@lwn.net>,
James Morse <james.morse@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Zenghui Yu <yuzenghui@huawei.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
"H. Peter Anvin" <hpa@zytor.com>,
Steven Rostedt <rostedt@goodmis.org>,
Masami Hiramatsu <mhiramat@kernel.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Shaoqin Huang <shahuang@redhat.com>,
Gavin Shan <gshan@redhat.com>,
Ricardo Koller <ricarkol@google.com>,
Raghavendra Rao Ananta <rananta@google.com>,
Ryan Roberts <ryan.roberts@arm.com>,
David Rientjes <rientjes@google.com>,
Axel Rasmussen <axelrasmussen@google.com>,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev,
kvm@vger.kernel.org, linux-mm@kvack.org,
linux-trace-kernel@vger.kernel.org
Subject: Re: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging
Date: Sat, 20 Apr 2024 18:19:28 -0600 [thread overview]
Message-ID: <CAOUHufb5di7t4aHx7jNJvMd_L85b=xoJvO4V4RGi+nY78rpF_Q@mail.gmail.com> (raw)
In-Reply-To: <CADrL8HUmGFP=w5COnJzyJ+2a2gjCugqQg9WDXQ2ZAK7B9gJThA@mail.gmail.com>
On Fri, Apr 19, 2024 at 3:48 PM James Houghton <jthoughton@google.com> wrote:
>
> On Fri, Apr 19, 2024 at 2:07 PM David Matlack <dmatlack@google.com> wrote:
> >
> > On 2024-04-19 01:47 PM, James Houghton wrote:
> > > On Thu, Apr 11, 2024 at 10:28 AM David Matlack <dmatlack@google.com> wrote:
> > > > On 2024-04-11 10:08 AM, David Matlack wrote:
> > > > bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
> > > > {
> > > > bool young = false;
> > > >
> > > > if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm))
> > > > young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
> > > >
> > > > if (tdp_mmu_enabled)
> > > > young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
> > > >
> > > > return young;
> > > > }
> > > >
> > > > bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
> > > > {
> > > > bool young = false;
> > > >
> > > > if (!range->arg.metadata->bitmap && kvm_memslots_have_rmaps(kvm))
> > > > young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap);
> > > >
> > > > if (tdp_mmu_enabled)
> > > > young |= kvm_tdp_mmu_test_age_gfn(kvm, range);
> > > >
> > > > return young;
> > >
> > >
> > > Yeah I think this is the right thing to do. Given your other
> > > suggestions (on patch 3), I think this will look something like this
> > > -- let me know if I've misunderstood something:
> > >
> > > bool check_rmap = !bitmap && kvm_memslot_have_rmaps(kvm);
> > >
> > > if (check_rmap)
> > > KVM_MMU_LOCK(kvm);
> > >
> > > rcu_read_lock(); // perhaps only do this when we don't take the MMU lock?
> > >
> > > if (check_rmap)
> > > kvm_handle_gfn_range(/* ... */ kvm_test_age_rmap)
> > >
> > > if (tdp_mmu_enabled)
> > > kvm_tdp_mmu_test_age_gfn() // modified to be RCU-safe
> > >
> > > rcu_read_unlock();
> > > if (check_rmap)
> > > KVM_MMU_UNLOCK(kvm);
> >
> > I was thinking a little different. If you follow my suggestion to first
> > make the TDP MMU aging lockless, you'll end up with something like this
> > prior to adding bitmap support (note: the comments are just for
> > demonstrative purposes):
> >
> > bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
> > {
> > bool young = false;
> >
> > /* Shadow MMU aging holds write-lock. */
> > if (kvm_memslots_have_rmaps(kvm)) {
> > write_lock(&kvm->mmu_lock);
> > young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
> > write_unlock(&kvm->mmu_lock);
> > }
> >
> > /* TDM MMU aging is lockless. */
> > if (tdp_mmu_enabled)
> > young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
> >
> > return young;
> > }
> >
> > Then when you add bitmap support it would look something like this:
> >
> > bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
> > {
> > unsigned long *bitmap = range->arg.metadata->bitmap;
> > bool young = false;
> >
> > /* SHadow MMU aging holds write-lock and does not support bitmap. */
> > if (kvm_memslots_have_rmaps(kvm) && !bitmap) {
> > write_lock(&kvm->mmu_lock);
> > young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap);
> > write_unlock(&kvm->mmu_lock);
> > }
> >
> > /* TDM MMU aging is lockless and supports bitmap. */
> > if (tdp_mmu_enabled)
> > young |= kvm_tdp_mmu_age_gfn_range(kvm, range);
> >
> > return young;
> > }
> >
> > rcu_read_lock/unlock() would be called in kvm_tdp_mmu_age_gfn_range().
>
> Oh yes this is a lot better. I hope I would have seen this when it
> came time to actually update this patch. Thanks.
>
> >
> > That brings up a question I've been wondering about. If KVM only
> > advertises support for the bitmap lookaround when shadow roots are not
> > allocated, does that mean MGLRU will be blind to accesses made by L2
> > when nested virtualization is enabled? And does that mean the Linux MM
> > will think all L2 memory is cold (i.e. good candidate for swapping)
> > because it isn't seeing accesses made by L2?
>
> Yes, I think so (for both questions). That's better than KVM not
> participating in MGLRU aging at all, which is the case today (IIUC --
> also ignoring the case where KVM accesses guest memory directly). We
> could have MGLRU always invoke the mmu notifiers, but frequently
> taking the MMU lock for writing might be worse than evicting when we
> shouldn't. Maybe Yu tried this at some point, but I can't find any
> results for this.
No, in this case only the fast path (page table scanning) is disabled.
MGLRU still sees the A-bit from L2 using the rmap, i.e., the slow path
calling folio_check_references().
next prev parent reply other threads:[~2024-04-21 0:20 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-01 23:29 [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting James Houghton
2024-04-01 23:29 ` [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young James Houghton
2024-04-04 18:52 ` David Hildenbrand
2024-04-09 18:31 ` James Houghton
2024-04-09 19:35 ` David Hildenbrand
2024-04-11 0:35 ` James Houghton
2024-04-12 18:45 ` David Matlack
2024-04-19 20:34 ` James Houghton
2024-04-01 23:29 ` [PATCH v3 2/7] KVM: Move MMU notifier function declarations James Houghton
2024-04-01 23:29 ` [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young James Houghton
2024-04-12 20:28 ` David Matlack
2024-04-19 20:41 ` James Houghton
2024-04-01 23:29 ` [PATCH v3 4/7] KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask James Houghton
2024-04-01 23:29 ` [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging James Houghton
2024-04-11 17:08 ` David Matlack
2024-04-11 17:28 ` David Matlack
2024-04-11 18:00 ` David Matlack
2024-04-11 18:07 ` David Matlack
2024-04-19 20:47 ` James Houghton
2024-04-19 21:06 ` David Matlack
2024-04-19 21:48 ` James Houghton
2024-04-21 0:19 ` Yu Zhao [this message]
2024-04-12 20:44 ` David Matlack
2024-04-19 20:54 ` James Houghton
2024-04-01 23:29 ` [PATCH v3 6/7] KVM: arm64: " James Houghton
2024-04-02 4:06 ` Yu Zhao
2024-04-02 7:00 ` Oliver Upton
2024-04-02 7:33 ` Marc Zyngier
2024-04-01 23:29 ` [PATCH v3 7/7] mm: multi-gen LRU: use mmu_notifier_test_clear_young() James Houghton
2024-04-12 18:41 ` [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting David Matlack
2024-04-19 20:57 ` James Houghton
2024-04-19 22:23 ` Oliver Upton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAOUHufb5di7t4aHx7jNJvMd_L85b=xoJvO4V4RGi+nY78rpF_Q@mail.gmail.com' \
--to=yuzhao@google.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=bp@alien8.de \
--cc=catalin.marinas@arm.com \
--cc=corbet@lwn.net \
--cc=dave.hansen@linux.intel.com \
--cc=dmatlack@google.com \
--cc=gshan@redhat.com \
--cc=hpa@zytor.com \
--cc=james.morse@arm.com \
--cc=jthoughton@google.com \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.linux.dev \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=maz@kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@redhat.com \
--cc=oliver.upton@linux.dev \
--cc=pbonzini@redhat.com \
--cc=rananta@google.com \
--cc=ricarkol@google.com \
--cc=rientjes@google.com \
--cc=rostedt@goodmis.org \
--cc=ryan.roberts@arm.com \
--cc=seanjc@google.com \
--cc=shahuang@redhat.com \
--cc=suzuki.poulose@arm.com \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
--cc=yuzenghui@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).