From: Mikko Ylinen <mikko.ylinen@linux.intel.com>
To: Haitao Huang <haitao.huang@linux.intel.com>
Cc: jarkko@kernel.org, dave.hansen@linux.intel.com, tj@kernel.org,
linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org,
cgroups@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Jonathan Corbet <corbet@lwn.net>,
kai.huang@intel.com, reinette.chatre@intel.com,
Kristen Carlson Accardi <kristen@linux.intel.com>,
zhiquan1.li@intel.com, seanjc@google.com, bagasdotme@gmail.com,
linux-doc@vger.kernel.org, zhanb@microsoft.com,
anakrish@microsoft.com
Subject: Re: [PATCH v3 22/28] Docs/x86/sgx: Add description for cgroup support
Date: Thu, 17 Aug 2023 18:18:06 +0300 [thread overview]
Message-ID: <ZN46Lj8Ctde+QyPD@himmelriiki> (raw)
In-Reply-To: <20230712230202.47929-23-haitao.huang@linux.intel.com>
On Wed, Jul 12, 2023 at 04:01:56PM -0700, Haitao Huang wrote:
> From: Kristen Carlson Accardi <kristen@linux.intel.com>
>
> Add initial documentation of how to regulate the distribution of
> SGX Enclave Page Cache (EPC) memory via the Miscellaneous cgroup
> controller.
>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Signed-off-by: Kristen Carlson Accardi <kristen@linux.intel.com>
> Cc: Sean Christopherson <seanjc@google.com>
> Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
> ---
> Documentation/arch/x86/sgx.rst | 77 ++++++++++++++++++++++++++++++++++
> 1 file changed, 77 insertions(+)
>
> diff --git a/Documentation/arch/x86/sgx.rst b/Documentation/arch/x86/sgx.rst
> index 2bcbffacbed5..f6ca5594dcf2 100644
> --- a/Documentation/arch/x86/sgx.rst
> +++ b/Documentation/arch/x86/sgx.rst
> @@ -300,3 +300,80 @@ to expected failures and handle them as follows:
> first call. It indicates a bug in the kernel or the userspace client
> if any of the second round of ``SGX_IOC_VEPC_REMOVE_ALL`` calls has
> a return code other than 0.
> +
> +
> +Cgroup Support
> +==============
> +
> +The "sgx_epc" resource within the Miscellaneous cgroup controller regulates
> +distribution of SGX EPC memory, which is a subset of system RAM that
> +is used to provide SGX-enabled applications with protected memory,
> +and is otherwise inaccessible, i.e. shows up as reserved in
> +/proc/iomem and cannot be read/written outside of an SGX enclave.
> +
> +Although current systems implement EPC by stealing memory from RAM,
> +for all intents and purposes the EPC is independent from normal system
> +memory, e.g. must be reserved at boot from RAM and cannot be converted
> +between EPC and normal memory while the system is running. The EPC is
> +managed by the SGX subsystem and is not accounted by the memory
> +controller. Note that this is true only for EPC memory itself, i.e.
> +normal memory allocations related to SGX and EPC memory, e.g. the
> +backing memory for evicted EPC pages, are accounted, limited and
> +protected by the memory controller.
> +
> +Much like normal system memory, EPC memory can be overcommitted via
> +virtual memory techniques and pages can be swapped out of the EPC
> +to their backing store (normal system memory allocated via shmem).
> +The SGX EPC subsystem is analogous to the memory subsytem, and
> +it implements limit and protection models for EPC memory.
> +
> +SGX EPC Interface Files
> +-----------------------
> +
> +For a generic description of the Miscellaneous controller interface
> +files, please see Documentation/admin-guide/cgroup-v2.rst
> +
> +All SGX EPC memory amounts are in bytes unless explicitly stated
> +otherwise. If a value which is not PAGE_SIZE aligned is written,
> +the actual value used by the controller will be rounded down to
> +the closest PAGE_SIZE multiple.
> +
> + misc.capacity
> + A read-only flat-keyed file shown only in the root cgroup.
> + The sgx_epc resource will show the total amount of EPC
> + memory available on the platform.
> +
> + misc.current
> + A read-only flat-keyed file shown in the non-root cgroups.
> + The sgx_epc resource will show the current active EPC memory
> + usage of the cgroup and its descendants. EPC pages that are
> + swapped out to backing RAM are not included in the current count.
> +
> + misc.max
> + A read-write single value file which exists on non-root
> + cgroups. The sgx_epc resource will show the EPC usage
> + hard limit. The default is "max".
> +
> + If a cgroup's EPC usage reaches this limit, EPC allocations,
> + e.g. for page fault handling, will be blocked until EPC can
> + be reclaimed from the cgroup. If EPC cannot be reclaimed in
> + a timely manner, reclaim will be forced, e.g. by ignoring LRU.
Document the behavior when reclaim cannot happen, e.g., for the vEPC
pages when a VMM tries to allocate more than misc.max.
> +
> + misc.events
> + A read-write flat-keyed file which exists on non-root cgroups.
> + Writes to the file reset the event counters to zero. A value
> + change in this file generates a file modified event.
> +
> + max
> + The number of times the cgroup has triggered a reclaim
> + due to its EPC usage approaching (or exceeding) its max
> + EPC boundary.
> +
> +Migration
> +---------
> +
> +Once an EPC page is charged to a cgroup (during allocation), it
> +remains charged to the original cgroup until the page is released
> +or reclaimed. Migrating a process to a different cgroup doesn't
> +move the EPC charges that it incurred while in the previous cgroup
> +to its new cgroup.
> --
> 2.25.1
>
next prev parent reply other threads:[~2023-08-17 15:19 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-12 23:01 [PATCH v3 00/28] Add Cgroup support for SGX EPC memory Haitao Huang
2023-07-12 23:01 ` [PATCH v3 01/28] x86/sgx: Store struct sgx_encl when allocating new VA pages Haitao Huang
2023-07-17 11:14 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 02/28] x86/sgx: Add EPC page flags to identify owner type Haitao Huang
2023-07-17 12:41 ` Jarkko Sakkinen
2023-07-17 12:43 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 03/28] x86/sgx: Add 'struct sgx_epc_lru_lists' to encapsulate lru list(s) Haitao Huang
2023-07-17 12:45 ` Jarkko Sakkinen
2023-07-17 13:23 ` Haitao Huang
2023-07-17 14:39 ` Jarkko Sakkinen
2023-07-24 10:04 ` Huang, Kai
2023-07-24 14:55 ` Haitao Huang
2023-07-24 23:31 ` Huang, Kai
2023-07-31 20:35 ` Haitao Huang
2023-07-12 23:01 ` [PATCH v3 04/28] x86/sgx: Use sgx_epc_lru_lists for existing active page list Haitao Huang
2023-07-17 12:47 ` Jarkko Sakkinen
2023-07-31 20:43 ` Haitao Huang
2023-07-12 23:01 ` [PATCH v3 05/28] x86/sgx: Store reclaimable epc pages in sgx_epc_lru_lists Haitao Huang
2023-07-12 23:01 ` [PATCH v3 06/28] x86/sgx: store unreclaimable EPC " Haitao Huang
2023-07-12 23:01 ` [PATCH v3 07/28] x86/sgx: Introduce EPC page states Haitao Huang
2023-07-12 23:01 ` [PATCH v3 08/28] x86/sgx: Introduce RECLAIM_IN_PROGRESS state Haitao Huang
2023-07-12 23:01 ` [PATCH v3 09/28] x86/sgx: Use a list to track to-be-reclaimed pages Haitao Huang
2023-07-12 23:01 ` [PATCH v3 10/28] x86/sgx: Allow reclaiming up to 32 pages, but scan 16 by default Haitao Huang
2023-07-12 23:01 ` [PATCH v3 11/28] x85/sgx: Return the number of EPC pages that were successfully reclaimed Haitao Huang
2023-07-29 12:47 ` Pavel Machek
2023-07-31 11:10 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 12/28] x86/sgx: Add option to ignore age of page during EPC reclaim Haitao Huang
2023-07-12 23:01 ` [PATCH v3 13/28] x86/sgx: Prepare for multiple LRUs Haitao Huang
2023-07-12 23:01 ` [PATCH v3 14/28] x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgroup Haitao Huang
2023-07-12 23:01 ` [PATCH v3 15/28] x86/sgx: Add helper to grab pages from an arbitrary EPC LRU Haitao Huang
2023-07-12 23:01 ` [PATCH v3 16/28] x86/sgx: Add EPC OOM path to forcefully reclaim EPC Haitao Huang
2023-07-12 23:01 ` [PATCH v3 17/28] x86/sgx: fix a NULL pointer Haitao Huang
2023-07-17 12:48 ` Jarkko Sakkinen
2023-07-17 12:49 ` Jarkko Sakkinen
2023-07-17 13:14 ` Haitao Huang
2023-07-17 14:33 ` Jarkko Sakkinen
2023-07-17 15:49 ` Dave Hansen
2023-07-17 18:49 ` Haitao Huang
2023-07-17 18:52 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 18/28] cgroup/misc: Fix an overflow Haitao Huang
2023-07-17 13:15 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 19/28] cgroup/misc: Add per resource callbacks for CSS events Haitao Huang
2023-07-17 13:16 ` Jarkko Sakkinen
2023-07-12 23:01 ` [PATCH v3 20/28] cgroup/misc: Add SGX EPC resource type and export APIs for SGX driver Haitao Huang
2023-07-12 23:01 ` [PATCH v3 21/28] x86/sgx: Limit process EPC usage with misc cgroup controller Haitao Huang
2023-07-13 0:03 ` Randy Dunlap
2023-08-17 15:12 ` Mikko Ylinen
2023-07-12 23:01 ` [PATCH v3 22/28] Docs/x86/sgx: Add description for cgroup support Haitao Huang
2023-07-13 0:10 ` Randy Dunlap
2023-07-14 20:01 ` Haitao Huang
2023-07-14 20:26 ` Haitao Huang
2023-08-17 15:18 ` Mikko Ylinen [this message]
2023-07-12 23:01 ` [PATCH v3 23/28] selftests/sgx: Retry the ioctl()'s returned with EAGAIN Haitao Huang
2023-07-12 23:01 ` [PATCH v3 24/28] selftests/sgx: Move ENCL_HEAP_SIZE_DEFAULT to main.c Haitao Huang
2023-07-12 23:01 ` [PATCH v3 25/28] selftests/sgx: Use encl->encl_size in sigstruct.c Haitao Huang
2023-07-12 23:02 ` [PATCH v3 26/28] selftests/sgx: Include the dynamic heap size to the ELRANGE calculation Haitao Huang
2023-07-12 23:02 ` [PATCH v3 27/28] selftests/sgx: Add SGX selftest augment_via_eaccept_long Haitao Huang
2023-07-12 23:02 ` [PATCH v3 28/28] selftests/sgx: Add scripts for epc cgroup testing Haitao Huang
2023-07-17 11:02 ` [PATCH v3 00/28] Add Cgroup support for SGX EPC memory Jarkko Sakkinen
2023-07-24 19:09 ` Sohil Mehta
2023-07-25 17:16 ` Haitao Huang
2023-08-17 15:04 ` Mikko Ylinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZN46Lj8Ctde+QyPD@himmelriiki \
--to=mikko.ylinen@linux.intel.com \
--cc=anakrish@microsoft.com \
--cc=bagasdotme@gmail.com \
--cc=bp@alien8.de \
--cc=cgroups@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=dave.hansen@linux.intel.com \
--cc=haitao.huang@linux.intel.com \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=kai.huang@intel.com \
--cc=kristen@linux.intel.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sgx@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=reinette.chatre@intel.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
--cc=zhanb@microsoft.com \
--cc=zhiquan1.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).