From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 619D8C54E58 for ; Thu, 21 Mar 2024 05:24:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E73E56B0089; Thu, 21 Mar 2024 01:24:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E236A6B0092; Thu, 21 Mar 2024 01:24:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D11E06B0093; Thu, 21 Mar 2024 01:24:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C2DA66B0089 for ; Thu, 21 Mar 2024 01:24:20 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 83FC18154F for ; Thu, 21 Mar 2024 05:24:20 +0000 (UTC) X-FDA: 81919905480.30.0AF7B5A Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf25.hostedemail.com (Postfix) with ESMTP id 724CEA0011 for ; Thu, 21 Mar 2024 05:24:18 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=IE3HX9QP; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710998658; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=axkEJ+ZEA1LJ5tFH46QHNzXKeZoUps06XvLRgwgWqDk=; b=N+JkG41Bc0j+8s4Ytu1ABYnDWWkOLWLOTLg/ckL4UdyLjYWRikR8Yp8QzfZlTVnvlyH5qR p63oF/+j0CpyRmMlWdRbCC9tS2FR7/42VG73YHauV6X0OA6cQUwoOIAeUh+Zwyn+UMcldA iX1WtDRQihnVojQGAc/bDTODFmbnjx0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=IE3HX9QP; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710998658; a=rsa-sha256; cv=none; b=SFOVb/QnoBzomdbC2BDGnlsJb+o0R28SD/H/prBMUiQ/URukLfcpPrvr6+kpL2CgIAzzNC aIGLPHiJFxu5v4IFwN4TDhJuxWXJtOc/xD1q1TuJfslq562Pi2b+LgtiiPEDHVVLAC2gqZ fvMs6tT+8tSoQVzvOkMtcpspeKtaGE0= Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1710998653; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=axkEJ+ZEA1LJ5tFH46QHNzXKeZoUps06XvLRgwgWqDk=; b=IE3HX9QPwOPN7rOJIAye/uG1BG6XD6zDt7kHkZP8D1swuDM4sbii2UM5xWUq45PXBjFyCo d9oF3eSg7tn6nbeF4MURG3GJCaw59nLfDbuxXpH1yP7oDrcPb1ra+z0QzMA3S+dUoxK9Z7 XtExz357lDqrSnSZJ90sGHSwxGUg0f4= Date: Thu, 21 Mar 2024 13:24:07 +0800 MIME-Version: 1.0 Subject: Re: [External] Re: [bug report] mm/zswap :memory corruption after zswap_load(). Content-Language: en-US To: Zhongkun He Cc: Johannes Weiner , Yosry Ahmed , Andrew Morton , linux-mm , wuyun.abel@bytedance.com, zhouchengming@bytedance.com, Nhat Pham References: <01b0b8e8-af1d-4fbe-951e-278e882283fd@linux.dev> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 724CEA0011 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: k9r3pe9ed4rd86f9zhk43sckw885bnpd X-HE-Tag: 1710998658-721578 X-HE-Meta: U2FsdGVkX18Qdw0B2Dou9BrGt38vYym/Z1Kb1KG/N1ft2k9o6mlCqaYOS5NpRLceVqmmdYMbsQkv/vP25kIyNkom49oRM/tg6uYf748vJWE8ha0K6SwsJ5xFQvpZjyEfB9h4kp8k9xc0vIcBS1WJ9vIuZU4uXJtxjD/KGqzATWHoB4qXvH2ZvMHbgmATew28VffkS4bb7tfliCWz0ZH3EP7/FV6UhnLD3w0t+Z7xw22R2CtvFuJ5t/k4w5UM1l9WjD8qB7LX0oehBxeba4Y6gC8VUlaFwG/phedZt57NkIAfBR8/jg2/dNQW3wugfdtG4VHF3Vk1wpQMsbu4Gc/Xr3e49j+6XAJOHD/Y1dwFAUP6qk1IyM7xjs4adZaKYTnnbNM32rg22HaQueykjq4X9kkVh2ZbI9/YHz+JwDZHnxM+MZM7aEebWOkgJnSRODsn+1KmAV/JayrV8SHWTCfXYF1oCE6pfU7kEOcBIXq5YvaE6I7M6pGfVBXvZz3dhR/jewbDulDHApIX85AOlVz5chQtR/EIpUPbZn7n6WfUifObxF7cAyGzO0nRBZ7QHaunpE38CoZlkED0E0yEn4Pffzt6HdlZi4p+DdBmPzf+o8zbmkfC6cnmdQxOvIQ/NXii0k0X5uH8NJpGaoRZJm+zYvflfpTugutGKvHVhMMm/Bfx1YKHU2E05lDGen6oioTbk1OU0EIKIgBRsvNcwM0wBnkjiwfer+yIvEQxHZCzvnZ4nfzwX0uSk1s7s5WzKEjgn6NqdfGS7SYZ66u7u7rR2J/Q0i3+ASvCgEh4D8jfxDH674HPlaEk8Iz4Hy7EK1kQXvxRQ4/KzXm6rAICJOLSO7lhAxD528yPeBAE5wn4v26AZPo3x8c5/TzcDpEriR5CP/b+ptc2K9/pRbQ++t+0yTPz5U7KxJcon7aEiPk7zkB3nWLegLjPQRwtmKrhItJrPWcYvYbcQgL5R1gtEwS 5ar/bukA qPoLGwdkeIG+PzlSqsEghsoUyYFhFLE2sHkY3s68wqe9qDCixjoj8A5mKx3LDRRR19g4lTKxlFqkPAUnuh0vGrzwAuByGBASs7b7mVchGK5NGSPwmTtjQoXKHEcCMsc0SATzWN7Cq0kNJxGlD7WCwHcTqmla+CvaJtZ9z X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/3/21 13:09, Zhongkun He wrote: > On Thu, Mar 21, 2024 at 12:42 PM Chengming Zhou > wrote: >> >> On 2024/3/21 12:34, Zhongkun He wrote: >>> Hey folks, >>> >>> Recently, I tested the zswap with memory reclaiming in the mainline >>> (6.8) and found a memory corruption issue related to exclusive loads. >> >> Is this fix included? 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") >> This fix avoids concurrent swapin using the same swap entry. >> > > Yes, This fix avoids concurrent swapin from different cpu, but the > reported issue occurs > on the same cpu. I think you may misunderstand the race description in this fix changelog, the CPU0 and CPU1 just mean two concurrent threads, not real two CPUs. Could you verify if the problem still exists with this fix? > > Thanks. > >> Thanks. >> >>> >>> >>> root@**:/sys/fs/cgroup/zz# stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep >>> stress: info: [31753] dispatching hogs: 0 cpu, 0 io, 5 vm, 0 hdd >>> stress: FAIL: [31758] (522) memory corruption at: 0x7f347ed1a010 >>> stress: FAIL: [31753] (394) <-- worker 31758 returned error 1 >>> stress: WARN: [31753] (396) now reaping child worker processes >>> stress: FAIL: [31753] (451) failed run completed in 14s >>> >>> >>> 1. Test step(the frequency of memory reclaiming has been accelerated): >>> ------------------------- >>> a. set up the zswap, zram and cgroup V2 >>> b. echo 0 > /sys/kernel/mm/lru_gen/enabled >>> (Increase the probability of problems occurring) >>> c. mkdir /sys/fs/cgroup/zz >>> echo $$ > /sys/fs/cgroup/zz/cgroup.procs >>> cd /sys/fs/cgroup/zz/ >>> stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep >>> >>> e. in other shell: >>> while :;do for i in {1..5};do echo 20g > >>> /sys/fs/cgroup/zz/memory.reclaim & done;sleep 1;done >>> >>> 2. Root cause: >>> -------------------------- >>> With a small probability, the page fault will occur twice with the >>> original pte, even if a new pte has been successfully set. >>> Unfortunately, zswap_entry has been released during the first page fault >>> with exclusive loads, so zswap_load will fail, and there is no corresponding >>> data in swap space, memory corruption occurs. >>> >>> bpftrace -e'k:zswap_load {printf("%lld, %lld\n", ((struct page >>> *)arg0)->private,nsecs)}' >>> --include linux/mm_types.h > a.txt >>> >>> look up the same index: >>> >>> index nsecs >>> 1318876, 8976040736819 >>> 1318876, 8976040746078 >>> >>> 4123110, 8976234682970 >>> 4123110, 8976234689736 >>> >>> 2268896, 8976660124792 >>> 2268896, 8976660130607 >>> >>> 4634105, 8976662117938 >>> 4634105, 8976662127596 >>> >>> 3. Solution >>> >>> Should we free zswap_entry in batches so that zswap_entry will be >>> valid when the next page fault occurs with the >>> original pte? It would be great if there are other better solutions. >>> >>