From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE387C54E68 for ; Thu, 21 Mar 2024 04:34:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CF166B0087; Thu, 21 Mar 2024 00:34:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3584F6B0088; Thu, 21 Mar 2024 00:34:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D2056B0089; Thu, 21 Mar 2024 00:34:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 089956B0087 for ; Thu, 21 Mar 2024 00:34:59 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A7EE8A1E21 for ; Thu, 21 Mar 2024 04:34:58 +0000 (UTC) X-FDA: 81919781076.21.9EA80D1 Received: from mail-lj1-f174.google.com (mail-lj1-f174.google.com [209.85.208.174]) by imf25.hostedemail.com (Postfix) with ESMTP id A3954A0016 for ; Thu, 21 Mar 2024 04:34:55 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=axY5iZW8; spf=pass (imf25.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710995696; a=rsa-sha256; cv=none; b=VyOqxHuSzmP8OVjOJQ7aEOOyOSR5Bah9GgU0VgP2Rpm9y/Vv995EdXN99uvbNANsLuKQib PlaZe8PXVCR54Eo1V+A17WNjZTVBQESimEGWf+P+x1ub7KsR5idqr1hTS47GWEBJlkJ0Ag /40EgLLzSVj71iWGMeMXMfbj2/OKsvk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=axY5iZW8; spf=pass (imf25.hostedemail.com: domain of hezhongkun.hzk@bytedance.com designates 209.85.208.174 as permitted sender) smtp.mailfrom=hezhongkun.hzk@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710995696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=mdRjOMTSOKaQ58IP6OY8ykf+yFfyQE/bel/I262WfEE=; b=VhJUDoaxwxtvqu4mhqvDC/LMLc6/NVAd1L98l/4WI5kGfiBqVYmhAr6Tq6anlfOrTvryqJ v9kuTOPlagWvdHbPWsDu1EQ9sZx9LSdpUAF4Lkbn+XCO8ymBEKjQ+i1fqi/Dy1JQdbiIPj h+sSHan7EEHveXot4CGAAUy5lCmjaWQ= Received: by mail-lj1-f174.google.com with SMTP id 38308e7fff4ca-2d094bc2244so7902701fa.1 for ; Wed, 20 Mar 2024 21:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1710995693; x=1711600493; darn=kvack.org; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=mdRjOMTSOKaQ58IP6OY8ykf+yFfyQE/bel/I262WfEE=; b=axY5iZW8ufufnVMZv8aMIw9iM6TQhvfMIXigzRooCvfS59in4urRvEjBsikufT0sjd U0e7cmXt7oM6wwp9zRWFnrMeFWZCOOwXOufuSx1C69IvWnls9zPUqGpEKIchgBnAq811 P9WJFE/vCCCBbLJsPMoJDO4dZpnfflQQichBPhzFUk7RyBlFFe8isEv8eo/CvoPi2oMc bquqYdKFgT+6z2GphGfJXszbhF2mQYM3l2gZQXcwZyU3V8q02L4Ij9eQyKME5Sx0/HWI EJ7MkaTaDzWqcG0GTU2BHnoCc0CRhp5kjR2QmsNDbweuOqydvBkXMXa2ao1l33pOy2Ae 0RNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710995693; x=1711600493; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=mdRjOMTSOKaQ58IP6OY8ykf+yFfyQE/bel/I262WfEE=; b=PXA0Bn284BzYjjZmWGw5ilw+5mB4+M1YL94xsq5np2y2JHZsgOOuX16CQO1vEJmMiP Hc13UMkp48B47EeXRZQgBUXbH49wouvg949NyxKHKRo/3wvxEXi18gdpTZ4FhLBKAhjG G5xnX9U+xXsagUSfziWwr7Heki4fEPnIEtm/YUxNjJXckiBV/Hw+yXcgAKGzarh3tS4p iRl0LEm3mBQnmcOA+dDAz9JVs+hNtg8Hrg+qtvS3OgghSS8FRld7L79OwY8iSa1rTUBC SBxxfXhb2x9qX5qmZ50ITk8/kG8B9nao2thwEsKpWhUTndiz5P7M6lfM4pgbNGGkP2iD 3M8g== X-Gm-Message-State: AOJu0YzBLwX+22oJxb3W7d2BeGvN69vxO2A04vMflCH9k752tEBedaBK PqRVvL6DCvrMwFhrgNABZJaycTs8qo/R8STRD0xomLt2zbw+WiKLitkzmgju6QGZIuXcwl0HV+E jTMv05O/Mv/vwEK/gRnRIDzYkwRRNwzz6kqxWbg== X-Google-Smtp-Source: AGHT+IFKdaB8JVSf2kYeNhN4mHzbrdWrTQsesDyx2hUfp3AlqusVTe1OgCpZUl3CKPETLj7O2uZwRQnM/g+SDrh/55Q= X-Received: by 2002:a2e:9150:0:b0:2d6:9d3b:d767 with SMTP id q16-20020a2e9150000000b002d69d3bd767mr641372ljg.7.1710995693560; Wed, 20 Mar 2024 21:34:53 -0700 (PDT) MIME-Version: 1.0 From: Zhongkun He Date: Thu, 21 Mar 2024 12:34:41 +0800 Message-ID: Subject: [bug report] mm/zswap :memory corruption after zswap_load(). To: Johannes Weiner , Yosry Ahmed , Andrew Morton Cc: linux-mm , wuyun.abel@bytedance.com, zhouchengming@bytedance.com, Nhat Pham Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A3954A0016 X-Stat-Signature: o6b1zptztxxexc33stfmd5z7nbjzc6c6 X-Rspam-User: X-HE-Tag: 1710995695-360800 X-HE-Meta: U2FsdGVkX1+Et6DM0HbPaASLisI3hBzFQMRTgQC+T5Yz0BJEcgjvIQSAOJtZ04z0oA7iRIRmgKc+Q5MV3HzwBUWU+HOZpJZ6Tg0I7gM/ncq2l6FWoxpDyzbuOfDlCmRepn446VGNB3zmqylHp36YRXEYaeTIkOlhNq/8fDNcbJVl6kkrjM5I60+X8J11eFqNTxy69w4mIcmclZpj7cEDbLKwLvvFxjClrKkaVCR3ZyoMKC3PkUW216xuncuMAHYStmcJ6au/cFk7lAHXXI/zMi6i1Fh/Nm0MbSIZWFC4K/6jRhXC8pAsuzd/eZtDSb/m10i6+Rfz0iPfVCixes/sLwhIuudGd7Ozl7LjMmer1/0Oivd+otpisNGN+d74wb4LtmRV3OvdMlRI3VxlIgwRKwEo64FXyxAGj4706Z+DYKd75qdVmWLPrBM5FmwAncqInGZWVMTL3CyvR+ALcYbUQRO3SSqqqNnpSyysj2nQ0YBwE5V7QLluUYMJef/nu7akzwFHv7KT1GnFSva/FuRQU49tR87gfR4PGWWzthsMoD9VhSi7qdjSq7gmgCeYOS0TWKzOnQUenb+4KsAQ2RtUChGyZZaJygbHTWfac7c2Bkhb6adZpI9b5Kr/VISTM/YM/LIKVXxb4slCrkgAN2cJx0KBdIF5uO8iIMM0EaQEzWU6CIseXvaQNOldoW4+1n7QWoJeqovW0YKu38vLg2/tSadUDd0OfN6ochgR9OrOIqENJXtx5tINgvAAsrM4QvQp1qLKF/JlJ1dWpRqt82Q+kBF97Qn8//s4wMbi+I4ERHwGX0TMTwMZ1QBrHCjWKEa54RT1llC3pHosj2lyiy4meAoGFf/uVKZjbA7MZ90Y4VFCEV+GqSc5iKPQ8/d+ZQruYaZs78L2d3DW6mYivGHZo9mucZc32yTlel7ME++rpTZrZupVGocmV2GYx2d1fCA5KtRQtNLXyu1ip4M0Be2 fvE+self HvWqQXdEb7y79qGv1mmo4SGNONnL2PbTW78IyAADQ9mQgf0rHNKdoYKW+jw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000050, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hey folks, Recently, I tested the zswap with memory reclaiming in the mainline (6.8) and found a memory corruption issue related to exclusive loads. root@**:/sys/fs/cgroup/zz# stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep stress: info: [31753] dispatching hogs: 0 cpu, 0 io, 5 vm, 0 hdd stress: FAIL: [31758] (522) memory corruption at: 0x7f347ed1a010 stress: FAIL: [31753] (394) <-- worker 31758 returned error 1 stress: WARN: [31753] (396) now reaping child worker processes stress: FAIL: [31753] (451) failed run completed in 14s 1. Test step(the frequency of memory reclaiming has been accelerated): ------------------------- a. set up the zswap, zram and cgroup V2 b. echo 0 > /sys/kernel/mm/lru_gen/enabled (Increase the probability of problems occurring) c. mkdir /sys/fs/cgroup/zz echo $$ > /sys/fs/cgroup/zz/cgroup.procs cd /sys/fs/cgroup/zz/ stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep e. in other shell: while :;do for i in {1..5};do echo 20g > /sys/fs/cgroup/zz/memory.reclaim & done;sleep 1;done 2. Root cause: -------------------------- With a small probability, the page fault will occur twice with the original pte, even if a new pte has been successfully set. Unfortunately, zswap_entry has been released during the first page fault with exclusive loads, so zswap_load will fail, and there is no corresponding data in swap space, memory corruption occurs. bpftrace -e'k:zswap_load {printf("%lld, %lld\n", ((struct page *)arg0)->private,nsecs)}' --include linux/mm_types.h > a.txt look up the same index: index nsecs 1318876, 8976040736819 1318876, 8976040746078 4123110, 8976234682970 4123110, 8976234689736 2268896, 8976660124792 2268896, 8976660130607 4634105, 8976662117938 4634105, 8976662127596 3. Solution Should we free zswap_entry in batches so that zswap_entry will be valid when the next page fault occurs with the original pte? It would be great if there are other better solutions.