From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEBA7C54E58 for ; Thu, 21 Mar 2024 05:09:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20B416B0082; Thu, 21 Mar 2024 01:09:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BADC6B0083; Thu, 21 Mar 2024 01:09:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AAA26B0085; Thu, 21 Mar 2024 01:09:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EF3906B0082 for ; Thu, 21 Mar 2024 01:09:38 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A60B1161139 for ; Thu, 21 Mar 2024 05:09:38 +0000 (UTC) X-FDA: 81919868436.17.712D41F Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf24.hostedemail.com (Postfix) with ESMTP id D8BE118000C for ; Thu, 21 Mar 2024 05:09:35 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fW38651H; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710997776; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9TXmIZ4J/OL9cJz5zr5guiHFHZin3l3Ex4kkkRLZ9TA=; b=vrA3sEcPKiAbC8CGfuJKjZHO0O6B7GQRuS15msyDTiaT32XNUgambPJvOB3IIhkL4EGO7V sY2UtT27dTPKe79Gei12VGrukep0rifKrgQnkJ2GZNsP7FyceuvHwaswsVjqsaeU52lyHo 2G/q+epAKuhzUb1PV6oD0vD6aejXlpQ= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fW38651H; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf24.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710997776; a=rsa-sha256; cv=none; b=DqgqskmjXdXdsnmhn4AWvxuVvT4Qk0+y+uC+KhY4H6Z+pn3XdGd0VBrr8gaOmE/4LFcn7I hAT1sM4eljJmvw+vwropJieeEJcv4XHa1Q45gm9QV7Gk0VsZjADiNi4+jR5ZgYPJ85BJDu g34+2mjnW8eo6HyQzGrD2q5k0ywidJ8= Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2d29aad15a5so7354691fa.3 for ; Wed, 20 Mar 2024 22:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710997774; x=1711602574; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=9TXmIZ4J/OL9cJz5zr5guiHFHZin3l3Ex4kkkRLZ9TA=; b=fW38651HqZCcpZU0cyD5u6lGUCCANpBvb0qbeFnTr6O0Yljd8sk+vuNbIaJ7m3OBLT gj9X2kCKZRa0CedATwF8Dg3kO6H8CabgmmuyqrvwYHgO0N1rmppdderTzFlAzQNZEsgn CMBVRmkBNN7sBjdygq8n1k4p4Hwry5wTtpkMWZZj1nr6J3yfalrcLrakVB7ZGqoK7cae u0+o8bNAaWSVXjwRxSVyQufSMO4skGMu2H6d3x+GxPBVivJqpZtIuh6Mpb7BfXYVvsEZ 6i1xSb4aPpieYPZqUXXxMfgr0w4gZuNak9S/GoQ8bFb1Jln+3vpCd0BPvcBL6JkcNmQ6 0CoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710997774; x=1711602574; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9TXmIZ4J/OL9cJz5zr5guiHFHZin3l3Ex4kkkRLZ9TA=; b=T9zUntW2NYLlTbUhB5GFYU4Zvz2RYKfFLE2lM/9gs/FDxwXa7q/r1+Cvs5xYe8eu6x D9odPrVwm5Ws8CtReeBqXD+uzQ3fc/biH7tgu2MVcjQ8WWJ7wVJ42ItU/UP6BSwZySqW /L3VWWIcU5YKzdubC8+ZqeQxFYNa61k04LdvtIfYizLffUiVPSvBh+uEp+e0yPBRUhGK H4DgZGDAnJbOpxQ6iiAclZffRMTCJNvLIOsRKJAwqbVIJHQlVobULE6Fy3L/5+++8/Lf 9kIc81rg2jhsYnrmVeFJBTD3dC7QSna4HL1s7qgZByjx0qnIKhqsYaumNEYZEn+Ay3y2 ydBA== X-Forwarded-Encrypted: i=1; AJvYcCVe9q33p4OZd1VNdDSHb8BZilV7oE0GILRJVtrb7YPzr4wP477VBOTR8IwYafUkJScX9bAaosswu8NYzmyS/bfNUfo= X-Gm-Message-State: AOJu0YyJbHkoi69tUUexg8ijwarnTycE/rkzwbx0YOcIzzgs5DhaAfSp knClIUAGO9HqhOQzlihZib2zgSF5ZxvT3vLP9gYLYOa9aRlOK6c2VUWFBzYxqkAVeDFOOZAOWfz eIWtWNhwpWXXx/cWx8xpv27qLUFA66U95HuVpSg== X-Google-Smtp-Source: AGHT+IFuV1/ye/Y3rxLyMR/2PhOiGHgPM0yAElAFcdK9I6B331CRw2F/fYojyETytMdWTFEh88bj+CYf8KBXCivrAOQ= X-Received: by 2002:a2e:824f:0:b0:2d2:dac3:90f with SMTP id j15-20020a2e824f000000b002d2dac3090fmr542469ljh.52.1710997773896; Wed, 20 Mar 2024 22:09:33 -0700 (PDT) MIME-Version: 1.0 References: <01b0b8e8-af1d-4fbe-951e-278e882283fd@linux.dev> In-Reply-To: <01b0b8e8-af1d-4fbe-951e-278e882283fd@linux.dev> From: Zhongkun He Date: Thu, 21 Mar 2024 13:09:22 +0800 Message-ID: Subject: Re: [External] Re: [bug report] mm/zswap :memory corruption after zswap_load(). To: Chengming Zhou Cc: Johannes Weiner , Yosry Ahmed , Andrew Morton , linux-mm , wuyun.abel@bytedance.com, zhouchengming@bytedance.com, Nhat Pham Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: D8BE118000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 97tgof8pqiz8nze9n39fkc8jne6z61e9 X-HE-Tag: 1710997775-837927 X-HE-Meta: U2FsdGVkX18Zb+otM72epv4YoOFCGUol7x2bAmjwWW5HX3d7fDRE9CqiXMlCIi/6NHdf3+C563pmFsHtJmm2h6uPtMyQ2d7x0Ykd5bDnB7SbrHdNUk8Z7RjCrd5vd3hgMJVu0+hgBUxdsffKBaXW2aGt/jHt06oliUiANph1NE/BQRoqIreWJA1hUbfj1MESsC0b6kgxIf1K4aF5XwPeily4Ue7IBH76Pv8kipyWzyX/DdTqIxCSX7FXaTyyNixKppin3tu1nF2uyx2TgWjWpKi6/chKTblXsk6oXt6SOLGFDMIKnHXy1f+YpRdfJOPL5GA6bTVRSd98flFpHKriezOoOizA1uTygMse2Be05jq24LROKI0dBdzASxrYGKLm8gLxXcQYZJle8HwEuOFQrQcUaSBYJg5DCxU9khkt/l7RKKrXZfuBYZ1TuKfWf6ldlVOjIF2TFq376OhK9mhyAz+++OK9uiuiWAXWwlbhBDXtkgrnoXlx/QbIjJa/g+bdlLsMoWA58YvufSCQLdMeu2rEeroudohrCTASmJAy4MavjMyQiyGpf7jgsl0dAZEyiMLQzY4dxpQc6DSVwMJOaJB+YYzpQ5o8BrqQO9z9dgiME8tLwwo7daFzgTpmZBjWltwq+6I7+YMnxfnRL4UxD7zeJPUf6OppxaUBK8n9JS+Gj+OxHHAKPxrVeldcFRB4XyWjeTUlGEmZ6sSVLU9gyt97eWDy82LxQpI6/PQY7KGNBewJ742BKUbSwiEBcbXgxSo5JPt351a3rt4HojGXjvVB1FChxZwLzXvAe6PSu+wPoT7l2kYxJchCeUe5aU6RqWY6NOBIyVLhfSJNozpiu9yd1hWE2FbmQSkRTsr5QhxrvQXrViovQd7XgFxu2eae+E/O9LuFmxyGEhRWLHtbFmNgceQIDnThc61CJOnf/0JIgzLKA8+zys8xzbxRhNdOKtJXfNj7+pnoPYP04Fl gbSKl/5h 9dYUMR+zzi+WR3YZWyr06aJpzF0alwWz70qYLp/et8Dhm5pbUkR5A2xjcU5WBTmwD2TcPBp1XAwK2H9LlneHL8RkVvNgKYr2DCC38uO5428teIvmqCVSsOAXlkm4W+e/GlQhoGNpDAMoYOKOLwz0Any+ASOorBernWAiTT6cD6X3hrhD1Bem/7t0UR/qk+OIK0VoA5Eo2B2teGfyooGwz4FtoQDxxwyO5NcTyBKTqdTCc2UNf9tU7H4CXn0lw67MJdFpjwrJ8sqZxVPVyB+wYR3NoaMLneY54mAxk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 21, 2024 at 12:42=E2=80=AFPM Chengming Zhou wrote: > > On 2024/3/21 12:34, Zhongkun He wrote: > > Hey folks, > > > > Recently, I tested the zswap with memory reclaiming in the mainline > > (6.8) and found a memory corruption issue related to exclusive loads. > > Is this fix included? 13ddaf26be32 ("mm/swap: fix race when skipping swap= cache") > This fix avoids concurrent swapin using the same swap entry. > Yes, This fix avoids concurrent swapin from different cpu, but the reported issue occurs on the same cpu. Thanks. > Thanks. > > > > > > > root@**:/sys/fs/cgroup/zz# stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm= -keep > > stress: info: [31753] dispatching hogs: 0 cpu, 0 io, 5 vm, 0 hdd > > stress: FAIL: [31758] (522) memory corruption at: 0x7f347ed1a010 > > stress: FAIL: [31753] (394) <-- worker 31758 returned error 1 > > stress: WARN: [31753] (396) now reaping child worker processes > > stress: FAIL: [31753] (451) failed run completed in 14s > > > > > > 1. Test step(the frequency of memory reclaiming has been accelerated): > > ------------------------- > > a. set up the zswap, zram and cgroup V2 > > b. echo 0 > /sys/kernel/mm/lru_gen/enabled > > (Increase the probability of problems occurring) > > c. mkdir /sys/fs/cgroup/zz > > echo $$ > /sys/fs/cgroup/zz/cgroup.procs > > cd /sys/fs/cgroup/zz/ > > stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep > > > > e. in other shell: > > while :;do for i in {1..5};do echo 20g > > > /sys/fs/cgroup/zz/memory.reclaim & done;sleep 1;done > > > > 2. Root cause: > > -------------------------- > > With a small probability, the page fault will occur twice with the > > original pte, even if a new pte has been successfully set. > > Unfortunately, zswap_entry has been released during the first page faul= t > > with exclusive loads, so zswap_load will fail, and there is no correspo= nding > > data in swap space, memory corruption occurs. > > > > bpftrace -e'k:zswap_load {printf("%lld, %lld\n", ((struct page > > *)arg0)->private,nsecs)}' > > --include linux/mm_types.h > a.txt > > > > look up the same index: > > > > index nsecs > > 1318876, 8976040736819 > > 1318876, 8976040746078 > > > > 4123110, 8976234682970 > > 4123110, 8976234689736 > > > > 2268896, 8976660124792 > > 2268896, 8976660130607 > > > > 4634105, 8976662117938 > > 4634105, 8976662127596 > > > > 3. Solution > > > > Should we free zswap_entry in batches so that zswap_entry will be > > valid when the next page fault occurs with the > > original pte? It would be great if there are other better solutions. > > >