From: Reinette Chatre <reinette.chatre@intel.com>
To: Jarkko Sakkinen <jarkko@kernel.org>,
Kristen Carlson Accardi <kristen@linux.intel.com>
Cc: <dave.hansen@linux.kernel.org>, <tj@kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-sgx@vger.kernel.org>,
<cgroups@vger.kernel.org>,
Dave Hansen <dave.hansen@linux.intel.com>,
"Thomas Gleixner" <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, "Borislav Petkov" <bp@alien8.de>,
<x86@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>,
<zhiquan1.li@intel.com>, Sean Christopherson <seanjc@google.com>
Subject: Re: [PATCH 01/26] x86/sgx: Call cond_resched() at the end of sgx_reclaim_pages()
Date: Tue, 15 Nov 2022 17:00:08 -0800 [thread overview]
Message-ID: <a4eb5ab0-bf83-17a4-8bc0-a90aaf438a8e@intel.com> (raw)
In-Reply-To: <Y3QgZ+ZKAfJ92U5L@kernel.org>
Hi Jarkko,
On 11/15/2022 3:27 PM, Jarkko Sakkinen wrote:
> On Fri, Nov 11, 2022 at 10:35:06AM -0800, Kristen Carlson Accardi wrote:
>> From: Sean Christopherson <sean.j.christopherson@intel.com>
>>
>> In order to avoid repetition of cond_resched() in ksgxd() and
>> sgx_alloc_epc_page(), move the invocation of post-reclaim cond_resched()
>> inside sgx_reclaim_pages(). Except in the case of sgx_reclaim_direct(),
>> sgx_reclaim_pages() is always called in a loop and is always followed
>> by a call to cond_resched(). This will hold true for the EPC cgroup
>> as well, which adds even more calls to sgx_reclaim_pages() and thus
>> cond_resched(). Calls to sgx_reclaim_direct() may be performance
>> sensitive. Allow sgx_reclaim_direct() to avoid the cond_resched()
>> call by moving the original sgx_reclaim_pages() call to
>> __sgx_reclaim_pages() and then have sgx_reclaim_pages() become a
>> wrapper around that call with a cond_resched().
>>
>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
>> Cc: Sean Christopherson <seanjc@google.com>
>> ---
>> arch/x86/kernel/cpu/sgx/main.c | 17 +++++++++++------
>> 1 file changed, 11 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
>> index 160c8dbee0ab..ffce6fc70a1f 100644
>> --- a/arch/x86/kernel/cpu/sgx/main.c
>> +++ b/arch/x86/kernel/cpu/sgx/main.c
>> @@ -287,7 +287,7 @@ static void sgx_reclaimer_write(struct sgx_epc_page *epc_page,
>> * problematic as it would increase the lock contention too much, which would
>> * halt forward progress.
>> */
>> -static void sgx_reclaim_pages(void)
>> +static void __sgx_reclaim_pages(void)
>> {
>> struct sgx_epc_page *chunk[SGX_NR_TO_SCAN];
>> struct sgx_backing backing[SGX_NR_TO_SCAN];
>> @@ -369,6 +369,12 @@ static void sgx_reclaim_pages(void)
>> }
>> }
>>
>> +static void sgx_reclaim_pages(void)
>> +{
>> + __sgx_reclaim_pages();
>> + cond_resched();
>> +}
>> +
>> static bool sgx_should_reclaim(unsigned long watermark)
>> {
>> return atomic_long_read(&sgx_nr_free_pages) < watermark &&
>> @@ -378,12 +384,14 @@ static bool sgx_should_reclaim(unsigned long watermark)
>> /*
>> * sgx_reclaim_direct() should be called (without enclave's mutex held)
>> * in locations where SGX memory resources might be low and might be
>> - * needed in order to make forward progress.
>> + * needed in order to make forward progress. This call to
>> + * __sgx_reclaim_pages() avoids the cond_resched() in sgx_reclaim_pages()
>> + * to improve performance.
>> */
>> void sgx_reclaim_direct(void)
>> {
>> if (sgx_should_reclaim(SGX_NR_LOW_PAGES))
>> - sgx_reclaim_pages();
>> + __sgx_reclaim_pages();
>
> Is it a big deal to have "extra" cond_resched?
>
sgx_reclaim_direct() is used to ensure that there is enough
SGX memory available to make forward progress within a loop that
may span a large range of pages. sgx_reclaim_direct()
ensures that there is enough memory right before it depends on
that available memory. I think that giving other tasks an opportunity
to run in the middle is risky since these other tasks may end
up consuming the SGX memory that was just freed up and thus increase
likelihood of the operation to fail with user getting an EAGAIN error.
Additionally, in a constrained environment where sgx_reclaim_direct()
is needed to reclaim pages an additional cond_resched() could cause
user visible slow down when operating on a large memory range.
Reinette
next prev parent reply other threads:[~2022-11-16 1:00 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-11 18:35 [PATCH 00/26] Add Cgroup support for SGX EPC memory Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 01/26] x86/sgx: Call cond_resched() at the end of sgx_reclaim_pages() Kristen Carlson Accardi
2022-11-15 23:27 ` Jarkko Sakkinen
2022-11-16 1:00 ` Reinette Chatre [this message]
2022-11-11 18:35 ` [PATCH 02/26] x86/sgx: Store struct sgx_encl when allocating new va pages Kristen Carlson Accardi
2022-11-15 23:31 ` Jarkko Sakkinen
2022-11-11 18:35 ` [PATCH 03/26] x86/sgx: Add 'struct sgx_epc_lru' to encapsulate lru list(s) Kristen Carlson Accardi
2022-11-15 23:35 ` Jarkko Sakkinen
2022-11-11 18:35 ` [PATCH 04/26] x86/sgx: Use sgx_epc_lru for existing active page list Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 05/26] x86/sgx: Track epc pages on reclaimable or unreclaimable lists Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 06/26] x86/sgx: Introduce RECLAIM_IN_PROGRESS flag for EPC pages Kristen Carlson Accardi
2022-11-15 23:42 ` Jarkko Sakkinen
2022-11-11 18:35 ` [PATCH 07/26] x86/sgx: Use a list to track to-be-reclaimed pages during reclaim Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 08/26] x86/sgx: Add EPC page flags to identify type of page Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 09/26] x86/sgx: Allow reclaiming up to 32 pages, but scan 16 by default Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 10/26] x86/sgx: Return the number of EPC pages that were successfully reclaimed Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 11/26] x86/sgx: Add option to ignore age of page during EPC reclaim Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 12/26] x86/sgx: Add helper to retrieve SGX EPC LRU given an EPC page Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 13/26] x86/sgx: Prepare for multiple LRUs Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 14/26] x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgroup Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 15/26] x86/sgx: Add helper to grab pages from an arbitrary EPC LRU Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 16/26] x86/sgx: Add EPC OOM path to forcefully reclaim EPC Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 17/26] cgroup/misc: Add notifier block list support for css events Kristen Carlson Accardi
2022-11-14 22:42 ` Tejun Heo
2022-11-14 23:10 ` Kristen Carlson Accardi
2022-11-14 23:11 ` Tejun Heo
2022-11-14 23:17 ` Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 18/26] cgroup/misc: Expose root_misc Kristen Carlson Accardi
2022-11-14 22:19 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 19/26] cgroup/misc: Expose parent_misc() Kristen Carlson Accardi
2022-11-14 22:30 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 20/26] cgroup/misc: allow users of misc cgroup to read specific cgroup usage Kristen Carlson Accardi
2022-11-14 22:31 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 21/26] cgroup/misc: allow misc cgroup consumers to read the max value Kristen Carlson Accardi
2022-11-14 22:33 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 22/26] cgroup/misc: Add private per cgroup data to struct misc_cg Kristen Carlson Accardi
2022-11-14 22:34 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 23/26] cgroup/misc: Add tryget functionality for misc controller Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 24/26] cgroup/misc: Add SGX EPC resource type Kristen Carlson Accardi
2022-11-11 18:35 ` [PATCH 25/26] x86/sgx: Add support for misc cgroup controller Kristen Carlson Accardi
2022-11-14 22:38 ` Tejun Heo
2022-11-11 18:35 ` [PATCH 26/26] Docs/x86/sgx: Add description for cgroup support Kristen Carlson Accardi
2022-11-12 9:28 ` Bagas Sanjaya
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a4eb5ab0-bf83-17a4-8bc0-a90aaf438a8e@intel.com \
--to=reinette.chatre@intel.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=dave.hansen@linux.intel.com \
--cc=dave.hansen@linux.kernel.org \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=kristen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).