From: zhenwei pi <pizhenwei@bytedance.com>
To: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
virtualization@lists.linux.dev
Cc: mst@redhat.com, david@redhat.com, jasowang@redhat.com,
xuanzhuo@linux.alibaba.com, akpm@linux-foundation.org,
zhenwei pi <pizhenwei@bytedance.com>
Subject: [RFC 0/3] Improve memory statistics for virtio balloon
Date: Mon, 15 Apr 2024 16:41:10 +0800 [thread overview]
Message-ID: <20240415084113.1203428-1-pizhenwei@bytedance.com> (raw)
Hi,
When the guest runs under critial memory pressure, the guest becomss
too slow, even sshd turns D state(uninterruptible) on memory
allocation. We can't login this VM to do any work on trouble shooting.
Guest kernel log via virtual TTY(on host side) only provides a few
necessary log after OOM. More detail memory statistics are required,
then we can know explicit memory events and estimate the pressure.
I'm going to introduce several VM counters for virtio balloon:
- oom-kill
- alloc-stall
- scan-async
- scan-direct
- reclaim-async
- reclaim-direct
Then we have a metric to analyze the memory performance:
[also describe this metric in patch
'virtio_balloon: introduce memory scan/reclaim info']
y: counter increases
n: counter does not changes
h: the rate of counter change is high
l: the rate of counter change is low
OOM: VIRTIO_BALLOON_S_OOM_KILL
STALL: VIRTIO_BALLOON_S_ALLOC_STALL
ASCAN: VIRTIO_BALLOON_S_SCAN_ASYNC
DSCAN: VIRTIO_BALLOON_S_SCAN_DIRECT
ARCLM: VIRTIO_BALLOON_S_RECLAIM_ASYNC
DRCLM: VIRTIO_BALLOON_S_RECLAIM_DIRECT
- OOM[y], STALL[*], ASCAN[*], DSCAN[*], ARCLM[*], DRCLM[*]:
the guest runs under really critial memory pressure
- OOM[n], STALL[h], ASCAN[*], DSCAN[l], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to cgroup, not the global memory
pressure.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[h]:
the memory allocation stalls due to global memory pressure. The
performance gets hurt a lot. A high ratio between DRCLM/DSCAN shows
quite effective memory reclaiming.
- OOM[n], STALL[h], ASCAN[*], DSCAN[h], ARCLM[*], DRCLM[l]:
the memory allocation stalls due to global memory pressure.
the ratio between DRCLM/DSCAN gets low, the guest OS is thrashing
heavily, the serious case leads poor performance and difficult
trouble shooting. Ex, sshd may block on memory allocation when
accepting new connections, a user can't login a VM by ssh command.
- OOM[n], STALL[n], ASCAN[h], DSCAN[n], ARCLM[l], DRCLM[n]:
the low ratio between ARCLM/ASCAN shows that the guest tries to
reclaim more memory, but it can't. Once more memory is required in
future, it will struggle to reclaim memory.
zhenwei pi (3):
virtio_balloon: introduce oom-kill invocations
virtio_balloon: introduce memory allocation stall counter
virtio_balloon: introduce memory scan/reclaim info
drivers/virtio/virtio_balloon.c | 30 ++++++++++++++++++++++++++++-
include/uapi/linux/virtio_balloon.h | 16 +++++++++++++--
2 files changed, 43 insertions(+), 3 deletions(-)
--
2.34.1
next reply other threads:[~2024-04-15 8:41 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-15 8:41 zhenwei pi [this message]
2024-04-15 8:41 ` [RFC 1/3] virtio_balloon: introduce oom-kill invocations zhenwei pi
2024-04-15 15:04 ` David Hildenbrand
2024-04-15 8:41 ` [RFC 2/3] virtio_balloon: introduce memory allocation stall counter zhenwei pi
2024-04-15 15:04 ` David Hildenbrand
2024-04-15 8:41 ` [RFC 3/3] virtio_balloon: introduce memory scan/reclaim info zhenwei pi
2024-04-15 15:06 ` David Hildenbrand
2024-04-15 15:01 ` [RFC 0/3] Improve memory statistics for virtio balloon David Hildenbrand
2024-04-16 3:20 ` zhenwei pi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240415084113.1203428-1-pizhenwei@bytedance.com \
--to=pizhenwei@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=david@redhat.com \
--cc=jasowang@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mst@redhat.com \
--cc=virtualization@lists.linux.dev \
--cc=xuanzhuo@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).