From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E2DDC48BD1 for ; Wed, 9 Jun 2021 08:14:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A4B861359 for ; Wed, 9 Jun 2021 08:14:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237487AbhFIIQv (ORCPT ); Wed, 9 Jun 2021 04:16:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:48244 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237285AbhFIIQs (ORCPT ); Wed, 9 Jun 2021 04:16:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623226494; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BJdAIhBhaJHDTxxgy9Ey5Eqpozz7WMzGVnuX/u268J8=; b=B4WmiwIXLq2Ut0hykR2DN0UkrBmgsPK5hpC+3poG+rM862icUCA9wstjy9hCS3h2Q54xO1 QdkRxFl79toWPX4aQGat22StmDt/KTnmNgQsNxTlqCxDEWSdCZ3mC8vmw8RheGybJgIMxx bDfwFU6qPJ3YijwXt/h2LULkTCbzamQ= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-251-j8m_qZQNMma8EOVdAFZA5A-1; Wed, 09 Jun 2021 04:14:52 -0400 X-MC-Unique: j8m_qZQNMma8EOVdAFZA5A-1 Received: by mail-wm1-f71.google.com with SMTP id g14-20020a05600c4eceb02901b609849650so1796132wmq.6 for ; Wed, 09 Jun 2021 01:14:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=BJdAIhBhaJHDTxxgy9Ey5Eqpozz7WMzGVnuX/u268J8=; b=EkfWpYmTwkvEAwV9HiXIvFixRvpvV3QuNqlhSeCxUMn3cUgXZq1gv8ZyV1SCabvWS+ iO5cWZb/VC6f06cyrSULHN8147KAqoixgp3VeURYW3jaq2VQ1ISABQABU/y1TH+AYQkS SuFHjHZUufu9Jelf23gcf+AaBO5aoAgVgGjEJ3Dlgkspj/GWspu446I2Rl8Z4ihWQwWL SO4izDouGOnDxKGjhite9GrquFGF3FZZPD4Ro1oBbL6Lunk9Egi9wAdU86KXTAhmkjE2 cJog4yFK6W+tF4LM51qDzdYX4Ktf7Nd+hNWxOMvXDVauVE6rstBkFguiCGqQYkO3YETI ZFPQ== X-Gm-Message-State: AOAM530NsMVCUJvXN51U86mxJ2T+M4mZYHcWe87GBqyShoG3UdBImV3t et0I8SI2OUT2iNZem+31iBG8Y/TZkYkLAtHveMowUtUzq/jQw3R9Y2IERdvkqIEwm95deApNmfP sSyrY2JdkXxBH4GKW8xh4pGNA X-Received: by 2002:a1c:5452:: with SMTP id p18mr25986470wmi.176.1623226491776; Wed, 09 Jun 2021 01:14:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwHztkUAyI+BmnE5wcOSMiNwdss3hfpHeT1nOccAa3B1HkID/3sYMZnDdskmY46CLurO/tQVA== X-Received: by 2002:a1c:5452:: with SMTP id p18mr25986454wmi.176.1623226491547; Wed, 09 Jun 2021 01:14:51 -0700 (PDT) Received: from ?IPv6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id w13sm24559323wrc.31.2021.06.09.01.14.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 09 Jun 2021 01:14:50 -0700 (PDT) Reply-To: eric.auger@redhat.com Subject: Re: Plan for /dev/ioasid RFC v2 To: "Tian, Kevin" , Jason Gunthorpe , "Alex Williamson (alex.williamson@redhat.com)" , Jean-Philippe Brucker , David Gibson , Jason Wang , "parav@mellanox.com" , "Enrico Weigelt, metux IT consult" , Paolo Bonzini , Shenming Lu Cc: Jonathan Corbet , "Raj, Ashok" , "Liu, Yi L" , "Wu, Hao" , "Jiang, Dave" , Jacob Pan , Kirti Wankhede , Robin Murphy , "kvm@vger.kernel.org" , "iommu@lists.linux-foundation.org" , David Woodhouse , Joerg Roedel , LKML , Lu Baolu References: From: Eric Auger Message-ID: Date: Wed, 9 Jun 2021 10:14:48 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Kevin, On 6/7/21 4:58 AM, Tian, Kevin wrote: > Hi, all, > > We plan to work on v2 now, given many good comments already received > and substantial changes envisioned. This is a very complex topic with > many sub-threads being discussed. To ensure that I didn't miss valuable > suggestions (and also keep everyone on the same page), here I'd like to > provide a list of planned changes in my mind. Please let me know if > anything important is lost. :) > > -- > > (Remaining opens in v1) > > - Protocol between kvm/vfio/ioasid for wbinvd/no-snoop. I'll see how > much can be refined based on discussion progress when v2 is out; > > - Device-centric (Jason) vs. group-centric (David) uAPI. David is not fully > convinced yet. Based on discussion v2 will continue to have ioasid uAPI > being device-centric (but it's fine for vfio to be group-centric). A new > section will be added to elaborate this part; > > - PASID virtualization (section 4) has not been thoroughly discussed yet. > Jason gave some suggestion on how to categorize intended usages. > I will rephrase this section and hope more discussions can be held for > it in v2; > > (Adopted suggestions) > > - (Jason) Rename /dev/ioasid to /dev/iommu (so does uAPI e.g. IOASID > _XXX to IOMMU_XXX). One suggestion (Jason) was to also rename > RID+PASID to SID+SSID. But given the familiarity of the former, I will > still use RID+PASID in v2 to ease the discussoin; > > - (Jason) v1 prevents one device from binding to multiple ioasid_fd's. This > will be fixed in v2; > > - (Jean/Jason) No need to track guest I/O page tables on ARM/AMD. When > a pasid table is bound, it becomes a container for all guest I/O page tables; while I am totally in line with that change, I guess we need to revisit the invalidate ioctl to support PASID table invalidation. > > - (Jean/Jason) Accordingly a device label is required so iotlb invalidation > and fault handling can both support per-device operation. Per Jean's > suggestion, this label will come from userspace (when VFIO_BIND_ > IOASID_FD); what is not totally clear to me is the correspondance between this label and the SID/SSID tuple. My understanding is it rather maps to the SID because you can attach several ioasids to the device. So it is not clear to me how you reconstruct the SSID info Thanks Eric > > - (Jason) Addition of device label allows per-device capability/format > check before IOASIDs are created. This leads to another major uAPI > change in v2 - specify format info when creating an IOASID (mapping > protocol, nesting, coherent, etc.). User is expected to check per-device > format and then set proper format for IOASID upon to-be-attached > device; > - (Jason/David) No restriction on map/unmap vs. bind/invalidate. They > can be used in either parent or child; > > - (David) Change IOASID_GET_INFO to report permitted range instead of > reserved IOVA ranges. This works better for PPC; > > - (Jason) For helper functions, expect to have explicit bus-type wrappers > e.g. ioasid_pci_device_attach; > > (Not adopted) > > - (Parav) Make page pinning a syscall; > - (Jason. W/Enrico) one I/O page table per fd; > - (David) Replace IOASID_REGISTER_MEMORY through another ioasid > nesting (sort of passthrough mode). Need more thinking. v2 will not > change this part; > > Thanks > Kevin > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_RED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D7D4C48BDF for ; Wed, 9 Jun 2021 08:14:59 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B8F961285 for ; Wed, 9 Jun 2021 08:14:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B8F961285 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 18BDF83BEF; Wed, 9 Jun 2021 08:14:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9DupcV1_V3lE; Wed, 9 Jun 2021 08:14:58 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id F122D83A73; Wed, 9 Jun 2021 08:14:57 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C47B8C000D; Wed, 9 Jun 2021 08:14:57 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 76DF8C000B for ; Wed, 9 Jun 2021 08:14:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 653C540297 for ; Wed, 9 Jun 2021 08:14:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gvufEYaAY4ME for ; Wed, 9 Jun 2021 08:14:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7CBBF400D4 for ; Wed, 9 Jun 2021 08:14:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623226494; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BJdAIhBhaJHDTxxgy9Ey5Eqpozz7WMzGVnuX/u268J8=; b=B4WmiwIXLq2Ut0hykR2DN0UkrBmgsPK5hpC+3poG+rM862icUCA9wstjy9hCS3h2Q54xO1 QdkRxFl79toWPX4aQGat22StmDt/KTnmNgQsNxTlqCxDEWSdCZ3mC8vmw8RheGybJgIMxx bDfwFU6qPJ3YijwXt/h2LULkTCbzamQ= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-303-tHrz8mLWNpWeCNvRjOmZLw-1; Wed, 09 Jun 2021 04:14:52 -0400 X-MC-Unique: tHrz8mLWNpWeCNvRjOmZLw-1 Received: by mail-wr1-f71.google.com with SMTP id q15-20020adfc50f0000b0290111f48b865cso10418099wrf.4 for ; Wed, 09 Jun 2021 01:14:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=BJdAIhBhaJHDTxxgy9Ey5Eqpozz7WMzGVnuX/u268J8=; b=cQzbwiueirz8xkaLrjdw3ED9cVgymWjlwJOVt9Vbl4vRpP78TqDD+knAwOsxwUGKak Vn1bUtUroulxmMkxnk/xPNbQ7UpYfb0Fhs+shLOAdZCDFR0USNF8fkevxL6Ntk3wc45j 4qQxlMrFBynaM3osyEFB3mrngVDCU3tdLgJDs5ipgsKemITt37a+5bwWikfoxQSM56r3 i26zY1RAOfQEOyAwPjmvjlYLQAWXJ6ERKDS15iYLdjFuuDJDu/h0ehRCJCPUbkH0yPCE bnC9sg5aFkGdKIMu1peEMIASijT//vOHeMZ3C8zW3N81RoG5rxT+VnD9UpDWNIeuJOeQ h6vA== X-Gm-Message-State: AOAM530qvi2IuGeuWybcFVuxds/h6gZl9zOFSpM9oyWw7ibbqqfhrKZY wqJqQw5+DeLLLsMgMykQbrNa3i4KIdbrneKxS6cPpmIj1/LJ0UPQdeDnqCmcCXR2NA/oZjQYZpX cEfc1v0nsH8kxYUbtN31xtVigRJLWAQ== X-Received: by 2002:a1c:5452:: with SMTP id p18mr25986469wmi.176.1623226491776; Wed, 09 Jun 2021 01:14:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwHztkUAyI+BmnE5wcOSMiNwdss3hfpHeT1nOccAa3B1HkID/3sYMZnDdskmY46CLurO/tQVA== X-Received: by 2002:a1c:5452:: with SMTP id p18mr25986454wmi.176.1623226491547; Wed, 09 Jun 2021 01:14:51 -0700 (PDT) Received: from ?IPv6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id w13sm24559323wrc.31.2021.06.09.01.14.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 09 Jun 2021 01:14:50 -0700 (PDT) Subject: Re: Plan for /dev/ioasid RFC v2 To: "Tian, Kevin" , Jason Gunthorpe , "Alex Williamson (alex.williamson@redhat.com)" , Jean-Philippe Brucker , David Gibson , Jason Wang , "parav@mellanox.com" , "Enrico Weigelt, metux IT consult" , Paolo Bonzini , Shenming Lu References: From: Eric Auger Message-ID: Date: Wed, 9 Jun 2021 10:14:48 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eric.auger@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: "Jiang, Dave" , "Raj, Ashok" , "kvm@vger.kernel.org" , Jonathan Corbet , David Woodhouse , "iommu@lists.linux-foundation.org" , LKML , Kirti Wankhede , Robin Murphy X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: eric.auger@redhat.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" Hi Kevin, On 6/7/21 4:58 AM, Tian, Kevin wrote: > Hi, all, > > We plan to work on v2 now, given many good comments already received > and substantial changes envisioned. This is a very complex topic with > many sub-threads being discussed. To ensure that I didn't miss valuable > suggestions (and also keep everyone on the same page), here I'd like to > provide a list of planned changes in my mind. Please let me know if > anything important is lost. :) > > -- > > (Remaining opens in v1) > > - Protocol between kvm/vfio/ioasid for wbinvd/no-snoop. I'll see how > much can be refined based on discussion progress when v2 is out; > > - Device-centric (Jason) vs. group-centric (David) uAPI. David is not fully > convinced yet. Based on discussion v2 will continue to have ioasid uAPI > being device-centric (but it's fine for vfio to be group-centric). A new > section will be added to elaborate this part; > > - PASID virtualization (section 4) has not been thoroughly discussed yet. > Jason gave some suggestion on how to categorize intended usages. > I will rephrase this section and hope more discussions can be held for > it in v2; > > (Adopted suggestions) > > - (Jason) Rename /dev/ioasid to /dev/iommu (so does uAPI e.g. IOASID > _XXX to IOMMU_XXX). One suggestion (Jason) was to also rename > RID+PASID to SID+SSID. But given the familiarity of the former, I will > still use RID+PASID in v2 to ease the discussoin; > > - (Jason) v1 prevents one device from binding to multiple ioasid_fd's. This > will be fixed in v2; > > - (Jean/Jason) No need to track guest I/O page tables on ARM/AMD. When > a pasid table is bound, it becomes a container for all guest I/O page tables; while I am totally in line with that change, I guess we need to revisit the invalidate ioctl to support PASID table invalidation. > > - (Jean/Jason) Accordingly a device label is required so iotlb invalidation > and fault handling can both support per-device operation. Per Jean's > suggestion, this label will come from userspace (when VFIO_BIND_ > IOASID_FD); what is not totally clear to me is the correspondance between this label and the SID/SSID tuple. My understanding is it rather maps to the SID because you can attach several ioasids to the device. So it is not clear to me how you reconstruct the SSID info Thanks Eric > > - (Jason) Addition of device label allows per-device capability/format > check before IOASIDs are created. This leads to another major uAPI > change in v2 - specify format info when creating an IOASID (mapping > protocol, nesting, coherent, etc.). User is expected to check per-device > format and then set proper format for IOASID upon to-be-attached > device; > - (Jason/David) No restriction on map/unmap vs. bind/invalidate. They > can be used in either parent or child; > > - (David) Change IOASID_GET_INFO to report permitted range instead of > reserved IOVA ranges. This works better for PPC; > > - (Jason) For helper functions, expect to have explicit bus-type wrappers > e.g. ioasid_pci_device_attach; > > (Not adopted) > > - (Parav) Make page pinning a syscall; > - (Jason. W/Enrico) one I/O page table per fd; > - (David) Replace IOASID_REGISTER_MEMORY through another ioasid > nesting (sort of passthrough mode). Need more thinking. v2 will not > change this part; > > Thanks > Kevin > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu