From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01302C11F65 for ; Mon, 28 Jun 2021 22:31:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D5C3F61CB4 for ; Mon, 28 Jun 2021 22:31:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235949AbhF1WeT (ORCPT ); Mon, 28 Jun 2021 18:34:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36696 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232323AbhF1WeQ (ORCPT ); Mon, 28 Jun 2021 18:34:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624919509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f8XGtEdANUVt2mXeyRNxrhe0L7TYkYXWNRbYkehzeNA=; b=fmDdCcYJAbkdCVPL0xcmjVG4hSJzTsykrtsfdV7pqHUDHa+yqmdscpITp9vcvcUDeyxKoJ z4MKAw/XzIbVsWqb5l5Ef1Q7NbDhjqDgV4ca+Yp0WB+le88kKnB5w7eeHbB4179LsBCGiy CaAoa/Agb23QV/OT/lheMU7Pxz5Xm0U= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-279-7OhdvdBNNQiIbKCkbRK-1A-1; Mon, 28 Jun 2021 18:31:48 -0400 X-MC-Unique: 7OhdvdBNNQiIbKCkbRK-1A-1 Received: by mail-ot1-f69.google.com with SMTP id 59-20020a9d0dc10000b02902a57e382ca1so14083492ots.7 for ; Mon, 28 Jun 2021 15:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f8XGtEdANUVt2mXeyRNxrhe0L7TYkYXWNRbYkehzeNA=; b=M+z8CFuj+s8zscuQZr7QNIdBLNldeKdy0qd1Bvlffx5NeCDofRNiSQBkWYQSGRueTK nYZl8tjkgSpZCtPl/J6bYOlQel+t7a9+uVMneqPijHrUv/c3TgoncJRnMaC3Foqn73q8 h/Y8kovON0ReLcfZLfA56hdM6/MKiw8lh9Xp6X3WfrdHKMsw7M5gXPqPCFNNSA/psBiu x9Nhbvc/BmUVfLDU5b4nZ8XrWuZFm7vvQXKvq9E+QVuxIQQIxeAZDGykpk8r6c+UiaM5 72Egt2S+69gu+7tB9MMNFuvBaYjH9qhCLoRG76i6CFBcWvuw0JRS4n3wDR0HkIw4+Zy6 lU1Q== X-Gm-Message-State: AOAM533wHG+wNPUVxhfrOBokrCLjYE+858MST6N7lq9A/L8cWheSXbsb pwP06bQNjwciWjJv+xRHRQVCSlg6YADNGM8GwF4MHcprvTcdx3u5wk6NTKW2b1owiB4v/1PZ4gO HD4K4cxs5GStHZdS4gf6X8KTv X-Received: by 2002:a9d:6219:: with SMTP id g25mr1586542otj.262.1624919507662; Mon, 28 Jun 2021 15:31:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyseKceU6IT0HP1k2nx0wDzLm7HSSZMWZSAeWgAdz0mPZZLAzQ4VusQyiwpVah22Rg3tvX5pA== X-Received: by 2002:a9d:6219:: with SMTP id g25mr1586520otj.262.1624919507429; Mon, 28 Jun 2021 15:31:47 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id p18sm1091723oth.60.2021.06.28.15.31.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:31:47 -0700 (PDT) Date: Mon, 28 Jun 2021 16:31:45 -0600 From: Alex Williamson To: "Tian, Kevin" Cc: Jason Gunthorpe , Joerg Roedel , "Jean-Philippe Brucker" , David Gibson , Jason Wang , "parav@mellanox.com" , "Enrico Weigelt, metux IT consult" , Paolo Bonzini , Shenming Lu , Eric Auger , Jonathan Corbet , "Raj, Ashok" , "Liu, Yi L" , "Wu, Hao" , "Jiang, Dave" , Jacob Pan , "Kirti Wankhede" , Robin Murphy , "kvm@vger.kernel.org" , "iommu@lists.linux-foundation.org" , "David Woodhouse" , LKML , "Lu Baolu" Subject: Re: Plan for /dev/ioasid RFC v2 Message-ID: <20210628163145.1a21cca9.alex.williamson@redhat.com> In-Reply-To: References: <20210615101215.4ba67c86.alex.williamson@redhat.com> <20210616133937.59050e1a.alex.williamson@redhat.com> <20210617151452.08beadae.alex.williamson@redhat.com> <20210618001956.GA1987166@nvidia.com> <20210618182306.GI1002214@nvidia.com> <20210625143616.GT2371267@nvidia.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 28 Jun 2021 01:09:18 +0000 "Tian, Kevin" wrote: > > From: Jason Gunthorpe > > Sent: Friday, June 25, 2021 10:36 PM > >=20 > > On Fri, Jun 25, 2021 at 10:27:18AM +0000, Tian, Kevin wrote: > > =20 > > > - When receiving the binding call for the 1st device in a group, io= mmu_fd > > > calls iommu_group_set_block_dma(group, dev->driver) which does > > > several things: =20 > >=20 > > The whole problem here is trying to match this new world where we want > > devices to be in charge of their own IOMMU configuration and the old > > world where groups are in charge. > >=20 > > Inserting the group fd and then calling a device-centric > > VFIO_GROUP_GET_DEVICE_FD_NEW doesn't solve this conflict, and isn't > > necessary. =20 >=20 > No, this was not what I meant. There is no group fd required when > calling this device-centric interface. I was actually talking about: >=20 > iommu_group_set_block_dma(dev->group, dev->driver) >=20 > just because current iommu layer API is group-centric. Whether this > should be improved could be next-level thing. Sorry for not making > it clear in the first place. >=20 > > We can always get the group back from the device at any > > point in the sequence do to a group wide operation. =20 >=20 > yes. >=20 > >=20 > > What I saw as the appeal of the sort of idea was to just completely > > leave all the difficult multi-device-group scenarios behind on the old > > group centric API and then we don't have to deal with them at all, or > > least not right away. =20 >=20 > yes, this is the staged approach that we discussed earlier. and > the reason why I refined this proposal about multi-devices group=20 > here is because you want to see some confidence along this > direction. Thus I expanded your idea and hope to achieve consensus > with Alex/Joerg who obviously have not been convinced yet. >=20 > >=20 > > I'd see some progression where iommu_fd only works with 1:1 groups at > > the start. Other scenarios continue with the old API. =20 >=20 > One uAPI open after completing this new sketch. v1 proposed to > conduct binding (VFIO_BIND_IOMMU_FD) after device_fd is acquired. > With this sketch we need a new VFIO_GROUP_GET_DEVICE_FD_NEW > to complete both in one step. I want to get Alex's confirmation whether > it sounds good to him, since it's better to unify the uAPI between 1:1=20 > group and 1:N group even if we don't support 1:N in the start.=20 I don't like it. It doesn't make sense to me. You have the group-centric world, which must continue to exist and cannot change because we cannot break the vfio uapi. We can make extensions, we can define a new parallel uapi, we can deprecate the uapi, but in the short term, it can't change. AIUI, the new device-centric model starts with vfio device files that can be opened directly. So what then is the purpose of a *GROUP* get device fd? Why is a vfio uapi involved in setting a device cookie for another subsystem? I'd expect that /dev/iommu will be used by multiple subsystems. All will want to bind devices to address spaces, so shouldn't binding a device to an iommufd be an ioctl on the iommufd, ie. IOMMU_BIND_VFIO_DEVICE_FD. Maybe we don't even need "VFIO" in there and the iommufd code can figure it out internally. You're essentially trying to reduce vfio to the device interface. That necessarily implies that ioctls on the container, group, or passed through the container to the iommu no longer exist. From my perspective, there should ideally be no new vfio ioctls. The user gets a limited access vfio device fd and enables full access to the device by registering it to the iommufd subsystem (100% this needs to be enforced until close() to avoid revoke issues). The user interacts exclusively with vfio via the device fd and performs all DMA address space related operations through the iommufd. > > Then maybe groups where all devices use the same IOASID. > >=20 > > Then 1:N groups if the source device is reliably identifiable, this > > requires iommu subystem work to attach domains to sub-group objects - > > not sure it is worthwhile. > >=20 > > But at least we can talk about each step with well thought out patches > >=20 > > The only thing that needs to be done to get the 1:1 step is to broadly > > define how the other two cases will work so we don't get into trouble > > and set some way to exclude the problematic cases from even getting to > > iommu_fd in the first place. > >=20 > > For instance if we go ahead and create /dev/vfio/device nodes we could > > do this only if the group was 1:1, otherwise the group cdev has to be > > used, along with its API. =20 >=20 > I feel for VFIO possibly we don't need significant change to its uAPI=20 > sequence, since it anyway needs to support existing semantics for=20 > backward compatibility. With this sketch we can keep vfio container/ > group by introducing an external iommu type which implies a different > GET_DEVICE_FD semantics. /dev/iommu can report a fd-wide capability > for whether 1:N group is supported to vfio user. Ideally vfio would also at least be able to register a type1 IOMMU backend through the existing uapi, backed by this iommu code, ie. we'd create a new "iommufd" (but without the user visible fd), bind all the group devices to it, generating our own device cookies, create a single ioasid and attach all the devices to it (all internal). When using the compatibility mode, userspace doesn't get device cookies, doesn't get an iommufd, they do mappings through the container, where vfio owns the cookies and ioasid. =20 > For new subsystems they can directly create device nodes and rely on > iommu fd to manage group isolation, without introducing any group=20 > semantics in its uAPI. Create device nodes, bind them to iommufd, associate cookies, attach ioasids, etc. That should be the same for all subsystems, including vfio, it's just the magic internal handshake between the device subsystem and the iommufd subsystem that changes. > > > a) Check group viability. A group is viable only when all dev= ices in > > > the group are in one of below states: > > > > > > * driver-less > > > * bound to a driver which is same as dev->driver (vfi= o in this case) > > > * bound to an otherwise allowed driver (same list as = in vfio) =20 > >=20 > > This really shouldn't use hardwired driver checks. Attached drivers > > should generically indicate to the iommu layer that they are safe for > > iommu_fd usage by calling some function around probe() =20 >=20 > good idea. >=20 > >=20 > > Thus a group must contain only iommu_fd safe drivers, or drivers-less > > devices before any of it can be used. It is the more general > > refactoring of what VFIO is doing. > > =20 > > > c) The iommu layer also verifies group viability on BUS_NOTIF= Y_ > > > BOUND_DRIVER event. BUG_ON if viability is broken while = =20 > > block_dma =20 > > > is set. =20 > >=20 > > And with this concept of iommu_fd safety being first-class maybe we > > can somehow eliminate this gross BUG_ON (and the 100's of lines of > > code that are used to create it) by denying probe to non-iommu-safe > > drivers, somehow. =20 >=20 > yes. >=20 > > =20 > > > - Binding other devices in the group to iommu_fd just succeeds since > > > the group is already in block_dma. =20 > >=20 > > I think the rest of this more or less describes the device centric > > logic for multi-device groups we've already talked about. I don't > > think it benifits from having the group fd > > =20 >=20 > sure. All of this new sketch doesn't have group fd in any iommu fd > API. Just try to elaborate a full sketch to sync the base. >=20 > Alex/Joerg, look forward to your thoughts now. =F0=9F=98=8A Some provided. Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14A21C11F66 for ; Mon, 28 Jun 2021 22:31:57 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6AEC761CF8 for ; Mon, 28 Jun 2021 22:31:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AEC761CF8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2F53A402AC; Mon, 28 Jun 2021 22:31:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6YoGr5Jhutjg; Mon, 28 Jun 2021 22:31:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id B49BB402C1; Mon, 28 Jun 2021 22:31:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6889AC001A; Mon, 28 Jun 2021 22:31:54 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 551C1C000E for ; Mon, 28 Jun 2021 22:31:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3979982BF5 for ; Mon, 28 Jun 2021 22:31:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e0stLHUufDoq for ; Mon, 28 Jun 2021 22:31:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id ED8298293F for ; Mon, 28 Jun 2021 22:31:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624919509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oG+hy7F4y4G5Ki5ZrHDrTCYzleu6iGOvnyW6nh8v7HI=; b=BywU7u4q2jBopHd4d7u2WPqOup6WF0SrQXmE8b1EZTGl1YC5ZkTPMQiUInDVG+QAdfJ4xH 5fAL1HRcwnWUlKX8KW73x0/w+4EwTL34EPfl0INqzi1YmsgqztGfmviHS0D4M7O3YlWFUe 2eiXKzQxIHqpBdBLd1CE/c4rAuBplgU= Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-258-xuWkA1mDNr-wAqNhU9rdxg-1; Mon, 28 Jun 2021 18:31:48 -0400 X-MC-Unique: xuWkA1mDNr-wAqNhU9rdxg-1 Received: by mail-ot1-f72.google.com with SMTP id v5-20020a05683011c5b02903cb28b38d0aso14072301otq.19 for ; Mon, 28 Jun 2021 15:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f8XGtEdANUVt2mXeyRNxrhe0L7TYkYXWNRbYkehzeNA=; b=thvUfcLmYa7N6QtrK7J3GoEsVHFiHUj1cNwTbI+TmMzPEgXGbHhM4STFZ08RuS+wGF ojuYiNKV6OAzAURxsiaVURzEpXoz9ZaH2PLJ7bk8LVXJezLafqUeFtKxc9xgx3IPxIVM iEUJXllO5XkbIHqS0I0moz0NxeJwJ1e2psJOFn0c1fT/1ocuvhJh/vzBasyABs6yObBU ehqTH5OrJkPIY+rXkMeOpD8uLnt8RnpEorqLTvc+PQHghj8X6OrIUIbrZC5sWJuTXjLo YbSKSQpVBh84evXaUSYMfkM5eeYwDgWG3iVyh64i1kZI3lFj9iXqx/Lifb3BCBhounj6 XGcQ== X-Gm-Message-State: AOAM531qZeRDY3+5i+GNqgGma/Ra4VbmLdlUojud0O7chntpLx5JRskO Y81YaXL14swnO2mTYp7j7YyvGUpy0QHSVYddpzK/Nnn2SINMNEzYfuK9dmlzVrNa0c8WxXlRrXT zjUYIT/4pLiYxzLeHM7TOlv8mt98Y6A== X-Received: by 2002:a9d:6219:: with SMTP id g25mr1586551otj.262.1624919507719; Mon, 28 Jun 2021 15:31:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyseKceU6IT0HP1k2nx0wDzLm7HSSZMWZSAeWgAdz0mPZZLAzQ4VusQyiwpVah22Rg3tvX5pA== X-Received: by 2002:a9d:6219:: with SMTP id g25mr1586520otj.262.1624919507429; Mon, 28 Jun 2021 15:31:47 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id p18sm1091723oth.60.2021.06.28.15.31.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:31:47 -0700 (PDT) Date: Mon, 28 Jun 2021 16:31:45 -0600 From: Alex Williamson To: "Tian, Kevin" Subject: Re: Plan for /dev/ioasid RFC v2 Message-ID: <20210628163145.1a21cca9.alex.williamson@redhat.com> In-Reply-To: References: <20210615101215.4ba67c86.alex.williamson@redhat.com> <20210616133937.59050e1a.alex.williamson@redhat.com> <20210617151452.08beadae.alex.williamson@redhat.com> <20210618001956.GA1987166@nvidia.com> <20210618182306.GI1002214@nvidia.com> <20210625143616.GT2371267@nvidia.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: "kvm@vger.kernel.org" , Jason Wang , Kirti Wankhede , Jean-Philippe Brucker , "Jiang, Dave" , "Raj, Ashok" , Jonathan Corbet , Jason Gunthorpe , "parav@mellanox.com" , "Enrico Weigelt, metux IT consult" , David Gibson , Robin Murphy , LKML , Shenming Lu , "iommu@lists.linux-foundation.org" , Paolo Bonzini , David Woodhouse X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" T24gTW9uLCAyOCBKdW4gMjAyMSAwMTowOToxOCArMDAwMAoiVGlhbiwgS2V2aW4iIDxrZXZpbi50 aWFuQGludGVsLmNvbT4gd3JvdGU6Cgo+ID4gRnJvbTogSmFzb24gR3VudGhvcnBlIDxqZ2dAbnZp ZGlhLmNvbT4KPiA+IFNlbnQ6IEZyaWRheSwgSnVuZSAyNSwgMjAyMSAxMDozNiBQTQo+ID4gCj4g PiBPbiBGcmksIEp1biAyNSwgMjAyMSBhdCAxMDoyNzoxOEFNICswMDAwLCBUaWFuLCBLZXZpbiB3 cm90ZToKPiA+ICAgCj4gPiA+IC0gICBXaGVuIHJlY2VpdmluZyB0aGUgYmluZGluZyBjYWxsIGZv ciB0aGUgMXN0IGRldmljZSBpbiBhIGdyb3VwLCBpb21tdV9mZAo+ID4gPiAgICAgY2FsbHMgaW9t bXVfZ3JvdXBfc2V0X2Jsb2NrX2RtYShncm91cCwgZGV2LT5kcml2ZXIpIHdoaWNoIGRvZXMKPiA+ ID4gICAgIHNldmVyYWwgdGhpbmdzOiAgCj4gPiAKPiA+IFRoZSB3aG9sZSBwcm9ibGVtIGhlcmUg aXMgdHJ5aW5nIHRvIG1hdGNoIHRoaXMgbmV3IHdvcmxkIHdoZXJlIHdlIHdhbnQKPiA+IGRldmlj ZXMgdG8gYmUgaW4gY2hhcmdlIG9mIHRoZWlyIG93biBJT01NVSBjb25maWd1cmF0aW9uIGFuZCB0 aGUgb2xkCj4gPiB3b3JsZCB3aGVyZSBncm91cHMgYXJlIGluIGNoYXJnZS4KPiA+IAo+ID4gSW5z ZXJ0aW5nIHRoZSBncm91cCBmZCBhbmQgdGhlbiBjYWxsaW5nIGEgZGV2aWNlLWNlbnRyaWMKPiA+ IFZGSU9fR1JPVVBfR0VUX0RFVklDRV9GRF9ORVcgZG9lc24ndCBzb2x2ZSB0aGlzIGNvbmZsaWN0 LCBhbmQgaXNuJ3QKPiA+IG5lY2Vzc2FyeS4gICAKPiAKPiBObywgdGhpcyB3YXMgbm90IHdoYXQg SSBtZWFudC4gVGhlcmUgaXMgbm8gZ3JvdXAgZmQgcmVxdWlyZWQgd2hlbgo+IGNhbGxpbmcgdGhp cyBkZXZpY2UtY2VudHJpYyBpbnRlcmZhY2UuIEkgd2FzIGFjdHVhbGx5IHRhbGtpbmcgYWJvdXQ6 Cj4gCj4gCWlvbW11X2dyb3VwX3NldF9ibG9ja19kbWEoZGV2LT5ncm91cCwgZGV2LT5kcml2ZXIp Cj4gCj4ganVzdCBiZWNhdXNlIGN1cnJlbnQgaW9tbXUgbGF5ZXIgQVBJIGlzIGdyb3VwLWNlbnRy aWMuIFdoZXRoZXIgdGhpcwo+IHNob3VsZCBiZSBpbXByb3ZlZCBjb3VsZCBiZSBuZXh0LWxldmVs IHRoaW5nLiBTb3JyeSBmb3Igbm90IG1ha2luZwo+IGl0IGNsZWFyIGluIHRoZSBmaXJzdCBwbGFj ZS4KPiAKPiA+IFdlIGNhbiBhbHdheXMgZ2V0IHRoZSBncm91cCBiYWNrIGZyb20gdGhlIGRldmlj ZSBhdCBhbnkKPiA+IHBvaW50IGluIHRoZSBzZXF1ZW5jZSBkbyB0byBhIGdyb3VwIHdpZGUgb3Bl cmF0aW9uLiAgCj4gCj4geWVzLgo+IAo+ID4gCj4gPiBXaGF0IEkgc2F3IGFzIHRoZSBhcHBlYWwg b2YgdGhlIHNvcnQgb2YgaWRlYSB3YXMgdG8ganVzdCBjb21wbGV0ZWx5Cj4gPiBsZWF2ZSBhbGwg dGhlIGRpZmZpY3VsdCBtdWx0aS1kZXZpY2UtZ3JvdXAgc2NlbmFyaW9zIGJlaGluZCBvbiB0aGUg b2xkCj4gPiBncm91cCBjZW50cmljIEFQSSBhbmQgdGhlbiB3ZSBkb24ndCBoYXZlIHRvIGRlYWwg d2l0aCB0aGVtIGF0IGFsbCwgb3IKPiA+IGxlYXN0IG5vdCByaWdodCBhd2F5LiAgCj4gCj4geWVz LCB0aGlzIGlzIHRoZSBzdGFnZWQgYXBwcm9hY2ggdGhhdCB3ZSBkaXNjdXNzZWQgZWFybGllci4g YW5kCj4gdGhlIHJlYXNvbiB3aHkgSSByZWZpbmVkIHRoaXMgcHJvcG9zYWwgYWJvdXQgbXVsdGkt ZGV2aWNlcyBncm91cCAKPiBoZXJlIGlzIGJlY2F1c2UgeW91IHdhbnQgdG8gc2VlIHNvbWUgY29u ZmlkZW5jZSBhbG9uZyB0aGlzCj4gZGlyZWN0aW9uLiBUaHVzIEkgZXhwYW5kZWQgeW91ciBpZGVh IGFuZCBob3BlIHRvIGFjaGlldmUgY29uc2Vuc3VzCj4gd2l0aCBBbGV4L0pvZXJnIHdobyBvYnZp b3VzbHkgaGF2ZSBub3QgYmVlbiBjb252aW5jZWQgeWV0Lgo+IAo+ID4gCj4gPiBJJ2Qgc2VlIHNv bWUgcHJvZ3Jlc3Npb24gd2hlcmUgaW9tbXVfZmQgb25seSB3b3JrcyB3aXRoIDE6MSBncm91cHMg YXQKPiA+IHRoZSBzdGFydC4gT3RoZXIgc2NlbmFyaW9zIGNvbnRpbnVlIHdpdGggdGhlIG9sZCBB UEkuICAKPiAKPiBPbmUgdUFQSSBvcGVuIGFmdGVyIGNvbXBsZXRpbmcgdGhpcyBuZXcgc2tldGNo LiB2MSBwcm9wb3NlZCB0bwo+IGNvbmR1Y3QgYmluZGluZyAoVkZJT19CSU5EX0lPTU1VX0ZEKSBh ZnRlciBkZXZpY2VfZmQgaXMgYWNxdWlyZWQuCj4gV2l0aCB0aGlzIHNrZXRjaCB3ZSBuZWVkIGEg bmV3IFZGSU9fR1JPVVBfR0VUX0RFVklDRV9GRF9ORVcKPiB0byBjb21wbGV0ZSBib3RoIGluIG9u ZSBzdGVwLiBJIHdhbnQgdG8gZ2V0IEFsZXgncyBjb25maXJtYXRpb24gd2hldGhlcgo+IGl0IHNv dW5kcyBnb29kIHRvIGhpbSwgc2luY2UgaXQncyBiZXR0ZXIgdG8gdW5pZnkgdGhlIHVBUEkgYmV0 d2VlbiAxOjEgCj4gZ3JvdXAgYW5kIDE6TiBncm91cCBldmVuIGlmIHdlIGRvbid0IHN1cHBvcnQg MTpOIGluIHRoZSBzdGFydC4gCgpJIGRvbid0IGxpa2UgaXQuICBJdCBkb2Vzbid0IG1ha2Ugc2Vu c2UgdG8gbWUuICBZb3UgaGF2ZSB0aGUKZ3JvdXAtY2VudHJpYyB3b3JsZCwgd2hpY2ggbXVzdCBj b250aW51ZSB0byBleGlzdCBhbmQgY2Fubm90IGNoYW5nZQpiZWNhdXNlIHdlIGNhbm5vdCBicmVh ayB0aGUgdmZpbyB1YXBpLiAgV2UgY2FuIG1ha2UgZXh0ZW5zaW9ucywgd2UgY2FuCmRlZmluZSBh IG5ldyBwYXJhbGxlbCB1YXBpLCB3ZSBjYW4gZGVwcmVjYXRlIHRoZSB1YXBpLCBidXQgaW4gdGhl IHNob3J0CnRlcm0sIGl0IGNhbid0IGNoYW5nZS4KCkFJVUksIHRoZSBuZXcgZGV2aWNlLWNlbnRy aWMgbW9kZWwgc3RhcnRzIHdpdGggdmZpbyBkZXZpY2UgZmlsZXMgdGhhdApjYW4gYmUgb3BlbmVk IGRpcmVjdGx5LiAgU28gd2hhdCB0aGVuIGlzIHRoZSBwdXJwb3NlIG9mIGEgKkdST1VQKiBnZXQK ZGV2aWNlIGZkPyAgV2h5IGlzIGEgdmZpbyB1YXBpIGludm9sdmVkIGluIHNldHRpbmcgYSBkZXZp Y2UgY29va2llIGZvcgphbm90aGVyIHN1YnN5c3RlbT8KCkknZCBleHBlY3QgdGhhdCAvZGV2L2lv bW11IHdpbGwgYmUgdXNlZCBieSBtdWx0aXBsZSBzdWJzeXN0ZW1zLiAgQWxsCndpbGwgd2FudCB0 byBiaW5kIGRldmljZXMgdG8gYWRkcmVzcyBzcGFjZXMsIHNvIHNob3VsZG4ndCBiaW5kaW5nIGEK ZGV2aWNlIHRvIGFuIGlvbW11ZmQgYmUgYW4gaW9jdGwgb24gdGhlIGlvbW11ZmQsIGllLgpJT01N VV9CSU5EX1ZGSU9fREVWSUNFX0ZELiAgTWF5YmUgd2UgZG9uJ3QgZXZlbiBuZWVkICJWRklPIiBp biB0aGVyZSBhbmQKdGhlIGlvbW11ZmQgY29kZSBjYW4gZmlndXJlIGl0IG91dCBpbnRlcm5hbGx5 LgoKWW91J3JlIGVzc2VudGlhbGx5IHRyeWluZyB0byByZWR1Y2UgdmZpbyB0byB0aGUgZGV2aWNl IGludGVyZmFjZS4gIFRoYXQKbmVjZXNzYXJpbHkgaW1wbGllcyB0aGF0IGlvY3RscyBvbiB0aGUg Y29udGFpbmVyLCBncm91cCwgb3IgcGFzc2VkCnRocm91Z2ggdGhlIGNvbnRhaW5lciB0byB0aGUg aW9tbXUgbm8gbG9uZ2VyIGV4aXN0LiAgRnJvbSBteQpwZXJzcGVjdGl2ZSwgdGhlcmUgc2hvdWxk IGlkZWFsbHkgYmUgbm8gbmV3IHZmaW8gaW9jdGxzLiAgVGhlIHVzZXIgZ2V0cwphIGxpbWl0ZWQg YWNjZXNzIHZmaW8gZGV2aWNlIGZkIGFuZCBlbmFibGVzIGZ1bGwgYWNjZXNzIHRvIHRoZSBkZXZp Y2UKYnkgcmVnaXN0ZXJpbmcgaXQgdG8gdGhlIGlvbW11ZmQgc3Vic3lzdGVtICgxMDAlIHRoaXMg bmVlZHMgdG8gYmUKZW5mb3JjZWQgdW50aWwgY2xvc2UoKSB0byBhdm9pZCByZXZva2UgaXNzdWVz KS4gIFRoZSB1c2VyIGludGVyYWN0cwpleGNsdXNpdmVseSB3aXRoIHZmaW8gdmlhIHRoZSBkZXZp Y2UgZmQgYW5kIHBlcmZvcm1zIGFsbCBETUEgYWRkcmVzcwpzcGFjZSByZWxhdGVkIG9wZXJhdGlv bnMgdGhyb3VnaCB0aGUgaW9tbXVmZC4KCj4gPiBUaGVuIG1heWJlIGdyb3VwcyB3aGVyZSBhbGwg ZGV2aWNlcyB1c2UgdGhlIHNhbWUgSU9BU0lELgo+ID4gCj4gPiBUaGVuIDE6TiBncm91cHMgaWYg dGhlIHNvdXJjZSBkZXZpY2UgaXMgcmVsaWFibHkgaWRlbnRpZmlhYmxlLCB0aGlzCj4gPiByZXF1 aXJlcyBpb21tdSBzdWJ5c3RlbSB3b3JrIHRvIGF0dGFjaCBkb21haW5zIHRvIHN1Yi1ncm91cCBv YmplY3RzIC0KPiA+IG5vdCBzdXJlIGl0IGlzIHdvcnRod2hpbGUuCj4gPiAKPiA+IEJ1dCBhdCBs ZWFzdCB3ZSBjYW4gdGFsayBhYm91dCBlYWNoIHN0ZXAgd2l0aCB3ZWxsIHRob3VnaHQgb3V0IHBh dGNoZXMKPiA+IAo+ID4gVGhlIG9ubHkgdGhpbmcgdGhhdCBuZWVkcyB0byBiZSBkb25lIHRvIGdl dCB0aGUgMToxIHN0ZXAgaXMgdG8gYnJvYWRseQo+ID4gZGVmaW5lIGhvdyB0aGUgb3RoZXIgdHdv IGNhc2VzIHdpbGwgd29yayBzbyB3ZSBkb24ndCBnZXQgaW50byB0cm91YmxlCj4gPiBhbmQgc2V0 IHNvbWUgd2F5IHRvIGV4Y2x1ZGUgdGhlIHByb2JsZW1hdGljIGNhc2VzIGZyb20gZXZlbiBnZXR0 aW5nIHRvCj4gPiBpb21tdV9mZCBpbiB0aGUgZmlyc3QgcGxhY2UuCj4gPiAKPiA+IEZvciBpbnN0 YW5jZSBpZiB3ZSBnbyBhaGVhZCBhbmQgY3JlYXRlIC9kZXYvdmZpby9kZXZpY2Ugbm9kZXMgd2Ug Y291bGQKPiA+IGRvIHRoaXMgb25seSBpZiB0aGUgZ3JvdXAgd2FzIDE6MSwgb3RoZXJ3aXNlIHRo ZSBncm91cCBjZGV2IGhhcyB0byBiZQo+ID4gdXNlZCwgYWxvbmcgd2l0aCBpdHMgQVBJLiAgCj4g Cj4gSSBmZWVsIGZvciBWRklPIHBvc3NpYmx5IHdlIGRvbid0IG5lZWQgc2lnbmlmaWNhbnQgY2hh bmdlIHRvIGl0cyB1QVBJIAo+IHNlcXVlbmNlLCBzaW5jZSBpdCBhbnl3YXkgbmVlZHMgdG8gc3Vw cG9ydCBleGlzdGluZyBzZW1hbnRpY3MgZm9yIAo+IGJhY2t3YXJkIGNvbXBhdGliaWxpdHkuIFdp dGggdGhpcyBza2V0Y2ggd2UgY2FuIGtlZXAgdmZpbyBjb250YWluZXIvCj4gZ3JvdXAgYnkgaW50 cm9kdWNpbmcgYW4gZXh0ZXJuYWwgaW9tbXUgdHlwZSB3aGljaCBpbXBsaWVzIGEgZGlmZmVyZW50 Cj4gR0VUX0RFVklDRV9GRCBzZW1hbnRpY3MuIC9kZXYvaW9tbXUgY2FuIHJlcG9ydCBhIGZkLXdp ZGUgY2FwYWJpbGl0eQo+IGZvciB3aGV0aGVyIDE6TiBncm91cCBpcyBzdXBwb3J0ZWQgdG8gdmZp byB1c2VyLgoKSWRlYWxseSB2ZmlvIHdvdWxkIGFsc28gYXQgbGVhc3QgYmUgYWJsZSB0byByZWdp c3RlciBhIHR5cGUxIElPTU1VCmJhY2tlbmQgdGhyb3VnaCB0aGUgZXhpc3RpbmcgdWFwaSwgYmFj a2VkIGJ5IHRoaXMgaW9tbXUgY29kZSwgaWUuIHdlJ2QKY3JlYXRlIGEgbmV3ICJpb21tdWZkIiAo YnV0IHdpdGhvdXQgdGhlIHVzZXIgdmlzaWJsZSBmZCksIGJpbmQgYWxsIHRoZQpncm91cCBkZXZp Y2VzIHRvIGl0LCBnZW5lcmF0aW5nIG91ciBvd24gZGV2aWNlIGNvb2tpZXMsIGNyZWF0ZSBhIHNp bmdsZQppb2FzaWQgYW5kIGF0dGFjaCBhbGwgdGhlIGRldmljZXMgdG8gaXQgKGFsbCBpbnRlcm5h bCkuICBXaGVuIHVzaW5nIHRoZQpjb21wYXRpYmlsaXR5IG1vZGUsIHVzZXJzcGFjZSBkb2Vzbid0 IGdldCBkZXZpY2UgY29va2llcywgZG9lc24ndCBnZXQKYW4gaW9tbXVmZCwgdGhleSBkbyBtYXBw aW5ncyB0aHJvdWdoIHRoZSBjb250YWluZXIsIHdoZXJlIHZmaW8gb3ducyB0aGUKY29va2llcyBh bmQgaW9hc2lkLgogCj4gRm9yIG5ldyBzdWJzeXN0ZW1zIHRoZXkgY2FuIGRpcmVjdGx5IGNyZWF0 ZSBkZXZpY2Ugbm9kZXMgYW5kIHJlbHkgb24KPiBpb21tdSBmZCB0byBtYW5hZ2UgZ3JvdXAgaXNv bGF0aW9uLCB3aXRob3V0IGludHJvZHVjaW5nIGFueSBncm91cCAKPiBzZW1hbnRpY3MgaW4gaXRz IHVBUEkuCgpDcmVhdGUgZGV2aWNlIG5vZGVzLCBiaW5kIHRoZW0gdG8gaW9tbXVmZCwgYXNzb2Np YXRlIGNvb2tpZXMsIGF0dGFjaAppb2FzaWRzLCBldGMuICBUaGF0IHNob3VsZCBiZSB0aGUgc2Ft ZSBmb3IgYWxsIHN1YnN5c3RlbXMsIGluY2x1ZGluZwp2ZmlvLCBpdCdzIGp1c3QgdGhlIG1hZ2lj IGludGVybmFsIGhhbmRzaGFrZSBiZXR3ZWVuIHRoZSBkZXZpY2UKc3Vic3lzdGVtIGFuZCB0aGUg aW9tbXVmZCBzdWJzeXN0ZW0gdGhhdCBjaGFuZ2VzLgoKPiA+ID4gICAgICAgICBhKSBDaGVjayBn cm91cCB2aWFiaWxpdHkuIEEgZ3JvdXAgaXMgdmlhYmxlIG9ubHkgd2hlbiBhbGwgZGV2aWNlcyBp bgo+ID4gPiAgICAgICAgICAgICB0aGUgZ3JvdXAgYXJlIGluIG9uZSBvZiBiZWxvdyBzdGF0ZXM6 Cj4gPiA+Cj4gPiA+ICAgICAgICAgICAgICAgICAqIGRyaXZlci1sZXNzCj4gPiA+ICAgICAgICAg ICAgICAgICAqIGJvdW5kIHRvIGEgZHJpdmVyIHdoaWNoIGlzIHNhbWUgYXMgZGV2LT5kcml2ZXIg KHZmaW8gaW4gdGhpcyBjYXNlKQo+ID4gPiAgICAgICAgICAgICAgICAgKiBib3VuZCB0byBhbiBv dGhlcndpc2UgYWxsb3dlZCBkcml2ZXIgKHNhbWUgbGlzdCBhcyBpbiB2ZmlvKSAgCj4gPiAKPiA+ IFRoaXMgcmVhbGx5IHNob3VsZG4ndCB1c2UgaGFyZHdpcmVkIGRyaXZlciBjaGVja3MuIEF0dGFj aGVkIGRyaXZlcnMKPiA+IHNob3VsZCBnZW5lcmljYWxseSBpbmRpY2F0ZSB0byB0aGUgaW9tbXUg bGF5ZXIgdGhhdCB0aGV5IGFyZSBzYWZlIGZvcgo+ID4gaW9tbXVfZmQgdXNhZ2UgYnkgY2FsbGlu ZyBzb21lIGZ1bmN0aW9uIGFyb3VuZCBwcm9iZSgpICAKPiAKPiBnb29kIGlkZWEuCj4gCj4gPiAK PiA+IFRodXMgYSBncm91cCBtdXN0IGNvbnRhaW4gb25seSBpb21tdV9mZCBzYWZlIGRyaXZlcnMs IG9yIGRyaXZlcnMtbGVzcwo+ID4gZGV2aWNlcyBiZWZvcmUgYW55IG9mIGl0IGNhbiBiZSB1c2Vk LiBJdCBpcyB0aGUgbW9yZSBnZW5lcmFsCj4gPiByZWZhY3RvcmluZyBvZiB3aGF0IFZGSU8gaXMg ZG9pbmcuCj4gPiAgIAo+ID4gPiAgICAgICAgIGMpIFRoZSBpb21tdSBsYXllciBhbHNvIHZlcmlm aWVzIGdyb3VwIHZpYWJpbGl0eSBvbiBCVVNfTk9USUZZXwo+ID4gPiAgICAgICAgICAgICBCT1VO RF9EUklWRVIgZXZlbnQuIEJVR19PTiBpZiB2aWFiaWxpdHkgaXMgYnJva2VuIHdoaWxlICAKPiA+ IGJsb2NrX2RtYSAgCj4gPiA+ICAgICAgICAgICAgIGlzIHNldC4gIAo+ID4gCj4gPiBBbmQgd2l0 aCB0aGlzIGNvbmNlcHQgb2YgaW9tbXVfZmQgc2FmZXR5IGJlaW5nIGZpcnN0LWNsYXNzIG1heWJl IHdlCj4gPiBjYW4gc29tZWhvdyBlbGltaW5hdGUgdGhpcyBncm9zcyBCVUdfT04gKGFuZCB0aGUg MTAwJ3Mgb2YgbGluZXMgb2YKPiA+IGNvZGUgdGhhdCBhcmUgdXNlZCB0byBjcmVhdGUgaXQpIGJ5 IGRlbnlpbmcgcHJvYmUgdG8gbm9uLWlvbW11LXNhZmUKPiA+IGRyaXZlcnMsIHNvbWVob3cuICAK PiAKPiB5ZXMuCj4gCj4gPiAgIAo+ID4gPiAtICAgQmluZGluZyBvdGhlciBkZXZpY2VzIGluIHRo ZSBncm91cCB0byBpb21tdV9mZCBqdXN0IHN1Y2NlZWRzIHNpbmNlCj4gPiA+ICAgICB0aGUgZ3Jv dXAgaXMgYWxyZWFkeSBpbiBibG9ja19kbWEuICAKPiA+IAo+ID4gSSB0aGluayB0aGUgcmVzdCBv ZiB0aGlzIG1vcmUgb3IgbGVzcyBkZXNjcmliZXMgdGhlIGRldmljZSBjZW50cmljCj4gPiBsb2dp YyBmb3IgbXVsdGktZGV2aWNlIGdyb3VwcyB3ZSd2ZSBhbHJlYWR5IHRhbGtlZCBhYm91dC4gSSBk b24ndAo+ID4gdGhpbmsgaXQgYmVuaWZpdHMgZnJvbSBoYXZpbmcgdGhlIGdyb3VwIGZkCj4gPiAg IAo+IAo+IHN1cmUuIEFsbCBvZiB0aGlzIG5ldyBza2V0Y2ggZG9lc24ndCBoYXZlIGdyb3VwIGZk IGluIGFueSBpb21tdSBmZAo+IEFQSS4gSnVzdCB0cnkgdG8gZWxhYm9yYXRlIGEgZnVsbCBza2V0 Y2ggdG8gc3luYyB0aGUgYmFzZS4KPiAKPiBBbGV4L0pvZXJnLCBsb29rIGZvcndhcmQgdG8geW91 ciB0aG91Z2h0cyBub3cuIPCfmIoKClNvbWUgcHJvdmlkZWQuICBUaGFua3MsCgpBbGV4CgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwppb21tdSBtYWlsaW5n IGxpc3QKaW9tbXVAbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51 eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vaW9tbXU=