From: Dongsheng Yang <dongsheng.yang@easystack.cn>
To: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
Cc: John Groves <John@groves.net>,
Dan Williams <dan.j.williams@intel.com>,
Gregory Price <gregory.price@memverge.com>,
axboe@kernel.dk, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org,
nvdimm@lists.linux.dev
Subject: Re: [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device)
Date: Thu, 9 May 2024 21:03:41 +0800 [thread overview]
Message-ID: <a571be12-2fd3-e0ee-a914-0a6e2c46bdbc@easystack.cn> (raw)
In-Reply-To: <20240509132134.00000ae9@Huawei.com>
在 2024/5/9 星期四 下午 8:21, Jonathan Cameron 写道:
> On Thu, 9 May 2024 19:24:28 +0800
> Dongsheng Yang <dongsheng.yang@easystack.cn> wrote:
>
...
>>> Yes. I think we are going to have to wait on architecture specific clarifications
>>> before any software coherent use case can be guaranteed to work beyond the 3.1 ones
>>> for temporal sharing (only one accessing host at a time) and read only sharing where
>>> writes are dropped anyway so clean write back is irrelevant beyond some noise in
>>> logs possibly (if they do get logged it is considered so rare we don't care!).
>>
>> Hi Jonathan,
>> Allow me to discuss further. As described in CXL 3.1:
>> ```
>> Software-managed coherency schemes are complicated by any host or device
>> whose caching agents generate clean writebacks. A “No Clean Writebacks”
>> capability bit is available for a host in the CXL System Description
>> Structure (CSDS; see Section 9.18.1.6) or for a device in the DVSEC CXL
>> Capability2 register (see Section 8.1.3.7).
>> ```
>>
>> If we check and find that the "No clean writeback" bit in both CSDS and
>> DVSEC is set, can we then assume that software cache-coherency is
>> feasible, as outlined below:
>>
>> (1) Both the writer and reader ensure cache flushes. Since there are no
>> clean writebacks, there will be no background data writes.
>>
>> (2) The writer writes data to shared memory and then executes a cache
>> flush. If we trust the "No clean writeback" bit, we can assume that the
>> data in shared memory is coherent.
>>
>> (3) Before reading the data, the reader performs cache invalidation.
>> Since there are no clean writebacks, this invalidation operation will
>> not destroy the data written by the writer. Therefore, the data read by
>> the reader should be the data written by the writer, and since the
>> writer's cache is clean, it will not write data to shared memory during
>> the reader's reading process. Additionally, data integrity can be ensured.
>>
>> The first step for CBD should depend on hardware cache coherence, which
>> is clearer and more feasible. Here, I am just exploring the possibility
>> of software cache coherence, not insisting on implementing software
>> cache-coherency right away. :)
>
> Yes, if a platform sets that bit, you 'should' be fine. What exact flush
> is needed is architecture specific however and the DMA related ones
> may not be sufficient. I'd keep an eye open for arch doc update from the
> various vendors.
>
> Also, the architecture that motivated that bit existing is a 'moderately
> large' chip vendor so I'd go so far as to say adoption will be limited
> unless they resolve that in a future implementation :)
Great, I think we've had a good discussion and reached a consensus on
this issue. The remaining aspect will depend on hardware updates. Thank
you for the information, that helps a lot.
Thanx
>
> Jonathan
>
>>
>> Thanx
>>>
>>>>>
>>>>>>
>>>>>> CBD can initially support (3), and then transition to (1) when hardware
>>>>>> supports cache-coherency. If there's sufficient market demand, we can
>>>>>> also consider supporting (2).
>>>>> I'd assume only (3) works. The others rely on assumptions I don't think
>>>>
>>>> I guess you mean (1), the hardware cache-coherency way, right?
>>>
>>> Indeed - oops!
>>> Hardware coherency is the way to go, or a well defined and clearly document
>>> description of how to play with the various host architectures.
>>>
>>> Jonathan
>>>
>>>
>>>>
>>>> :)
>>>> Thanx
>>>>
>>>>> you can rely on.
>>>>>
>>>>> Fun fun fun,
>>>>>
>>>>> Jonathan
>>>>>
>>>>>>
>>>>>> How does this approach sound?
>>>>>>
>>>>>> Thanx
>>>>>>>
>>>>>>> J
>>>>>>>
>>>>>>>>
>>>>>>>> Keep in mind that I don't think anybody has cxl 3 devices or CPUs yet, and
>>>>>>>> shared memory is not explicitly legal in cxl 2, so there are things a cpu
>>>>>>>> could do (or not do) in a cxl 2 environment that are not illegal because
>>>>>>>> they should not be observable in a no-shared-memory environment.
>>>>>>>>
>>>>>>>> CBD is interesting work, though for some of the reasons above I'm somewhat
>>>>>>>> skeptical of shared memory as an IPC mechanism.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> John
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> .
>>>>>>>
>>>>>
>>>>> .
>>>>>
>>>
>>>
>
next prev parent reply other threads:[~2024-05-09 19:02 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-22 7:15 [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device) Dongsheng Yang
2024-04-22 7:16 ` [PATCH 1/7] block: Init for CBD(CXL " Dongsheng Yang
2024-04-22 18:39 ` Randy Dunlap
2024-04-22 22:41 ` Dongsheng Yang
2024-04-24 3:58 ` Chaitanya Kulkarni
2024-04-24 8:36 ` Dongsheng Yang
2024-04-22 7:16 ` [PATCH 2/7] cbd: introduce cbd_transport Dongsheng Yang
2024-04-24 4:08 ` Chaitanya Kulkarni
2024-04-24 8:43 ` Dongsheng Yang
2024-04-22 7:16 ` [PATCH 3/7] cbd: introduce cbd_channel Dongsheng Yang
2024-04-22 7:16 ` [PATCH 4/7] cbd: introduce cbd_host Dongsheng Yang
2024-04-25 5:51 ` [EXTERNAL] " Bharat Bhushan
2024-04-22 7:16 ` [PATCH 5/7] cbd: introuce cbd_backend Dongsheng Yang
2024-04-24 5:03 ` Chaitanya Kulkarni
2024-04-24 8:36 ` Dongsheng Yang
2024-04-25 5:46 ` [EXTERNAL] " Bharat Bhushan
2024-04-22 7:16 ` [PATCH 7/7] cbd: add related sysfs files in transport register Dongsheng Yang
2024-04-25 5:24 ` [EXTERNAL] " Bharat Bhushan
2024-04-22 22:42 ` [PATCH 6/7] cbd: introduce cbd_blkdev Dongsheng Yang
2024-04-23 7:27 ` Dongsheng Yang
2024-04-24 4:29 ` [PATCH RFC 0/7] block: Introduce CBD (CXL Block Device) Dan Williams
2024-04-24 6:33 ` Dongsheng Yang
2024-04-24 15:14 ` Gregory Price
2024-04-26 1:25 ` Dongsheng Yang
2024-04-26 13:48 ` Gregory Price
2024-04-26 14:53 ` Dongsheng Yang
2024-04-26 16:14 ` Gregory Price
2024-04-28 5:47 ` Dongsheng Yang
2024-04-28 16:44 ` Gregory Price
2024-04-28 16:55 ` John Groves
2024-05-03 9:52 ` Jonathan Cameron
2024-05-08 11:39 ` Dongsheng Yang
2024-05-08 12:11 ` Jonathan Cameron
2024-05-08 13:03 ` Dongsheng Yang
2024-05-08 15:44 ` Jonathan Cameron
2024-05-09 11:24 ` Dongsheng Yang
2024-05-09 12:21 ` Jonathan Cameron
2024-05-09 13:03 ` Dongsheng Yang [this message]
2024-05-21 18:41 ` Dan Williams
2024-05-22 6:17 ` Dongsheng Yang
2024-05-29 15:25 ` Gregory Price
2024-05-30 6:59 ` Dongsheng Yang
2024-05-30 13:38 ` Jonathan Cameron
2024-06-01 3:22 ` Dan Williams
2024-06-03 12:48 ` Jonathan Cameron
2024-06-03 17:28 ` James Morse
2024-06-04 14:26 ` Jonathan Cameron
2024-05-31 14:23 ` Gregory Price
2024-06-03 1:33 ` Dongsheng Yang
2024-04-30 0:34 ` Dan Williams
2024-04-24 18:08 ` Dan Williams
[not found] ` <539c1323-68f9-d753-a102-692b69049c20@easystack.cn>
2024-04-30 0:10 ` Dan Williams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a571be12-2fd3-e0ee-a914-0a6e2c46bdbc@easystack.cn \
--to=dongsheng.yang@easystack.cn \
--cc=John@groves.net \
--cc=Jonathan.Cameron@Huawei.com \
--cc=axboe@kernel.dk \
--cc=dan.j.williams@intel.com \
--cc=gregory.price@memverge.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nvdimm@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).