From: Guorui Yu <GuoRui.Yu@linux.alibaba.com>
To: Andi Kleen <ak@linux.intel.com>,
linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org,
konrad.wilk@oracle.com, linux-coco@lists.linux.dev
Cc: robin.murphy@arm.com
Subject: Re: [PATCH 2/4] swiotlb: Add a new cc-swiotlb implementation for Confidential VMs
Date: Wed, 1 Feb 2023 10:08:48 +0800 [thread overview]
Message-ID: <bdeef865-a326-75ce-a1d0-b5d0c5a44e14@linux.alibaba.com> (raw)
In-Reply-To: <ee5d8c26-a453-678c-be48-d586271573d6@linux.intel.com>
在 2023/2/1 01:16, Andi Kleen 写道:
> >No, this cannot guarantee we always have sufficient TLB caches, so we
> can also have a "No memory for cc-swiotlb buffer" warning.
>
> It's not just a warning, it will be IO errors, right?
>
Yes, they are IO errors, but unsustainable such IO errors are not fatal
in my limited testing so far, and the system can survive after through
them. Again, legacy swiotlb occasionally suffers from TLB starvation.
However, if dynamic allocation of TLB is not allowed at all, the system
will be more likely to be overwhelmed by a large of bursting IOs and
unable to respond. Such problems are generally transient, so it is
difficult to reproduce and debug in a production environment. Users can
only set an unreasonably large fixed size and REBOOT to mitigate this
problem as much as possible.
>>
>> But I want to emphasize that in this case, the current implementation
>> is no worse than the legacy implementation. Moreover, dynamic TLB
>> allocation is more suitable for situations where more disks/network
>> devices will be hotplugged, in which case you cannot pre-set a
>> reasonable value.
>
> That's a reasonable stand point, but have to emphasize that is
> "probabilistic" in all the descriptions and comments.
>
Agreed, but one point to add is that the user can adjust the water level
setting to reduce the possibility of interrupt context allocation TLB
failure.
According to the current design, the kthread will be awaken to allocate
new TLBs when it is lower than half of the water level, so more flexible
room can be left by increasing the water level.
> I assume you did some stress testing (E.g. all cores submitting at full
> bandwidth) to validate that it works for you?
>
> -Andi
>
Yes, I tested by fio with different block sizes, iodepths and job
numbers on my testbed.
And I have noticed that there are some "IO errors" of `No memory for
cc-swiotlb buffer` in the beginning of the test, but it will be
eventually disappeared as long as there are enough free memory.
Thanks for your time,
Guorui
next prev parent reply other threads:[~2023-02-01 2:14 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-28 8:32 [RFC] swiotlb: Add a new cc-swiotlb implementation for Confidential VMs GuoRui.Yu
2023-01-28 8:32 ` [PATCH 1/4] swiotlb: Split common code from swiotlb.{c,h} GuoRui.Yu
2023-01-28 8:32 ` [PATCH 2/4] swiotlb: Add a new cc-swiotlb implementation for Confidential VMs GuoRui.Yu
2023-01-28 12:03 ` kernel test robot
2023-01-28 16:41 ` Randy Dunlap
2023-01-29 1:54 ` Guorui Yu
2023-01-29 16:58 ` Andi Kleen
2023-01-30 2:25 ` Guorui Yu
2023-01-30 6:46 ` Andi Kleen
2023-01-30 13:45 ` Guorui Yu
2023-01-31 17:16 ` Andi Kleen
2023-02-01 2:08 ` Guorui Yu [this message]
2023-01-28 8:32 ` [PATCH 3/4] swiotlb: Add tracepoint swiotlb_unbounced GuoRui.Yu
2023-01-28 8:32 ` [PATCH 4/4] cc-swiotlb: Allow set swiotlb watermark from cmdline GuoRui.Yu
2023-01-28 20:19 ` kernel test robot
2023-01-28 9:03 ` [RFC] swiotlb: Add a new cc-swiotlb implementation for Confidential VMs Guorui Yu
2023-01-30 6:54 ` Christoph Hellwig
2023-01-30 13:03 ` Robin Murphy
2023-01-30 14:37 ` Guorui Yu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bdeef865-a326-75ce-a1d0-b5d0c5a44e14@linux.alibaba.com \
--to=guorui.yu@linux.alibaba.com \
--cc=ak@linux.intel.com \
--cc=iommu@lists.linux-foundation.org \
--cc=konrad.wilk@oracle.com \
--cc=linux-coco@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=robin.murphy@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).