$ git log --pretty=format:'%h %s (%cs)%d'
6bc94f0 Update the version (2024-04-29)
(HEAD -> master, tag: v2.3.3, next)
69e467f release note: Update (2024-04-28)
0be57a9 tests/report/test: Use realpath for expect/result files (2024-04-28)
a5624bf tests/report/expects/report-wss*: Update for changed report wss behavior (2024-04-28)
d5bfca3 TODO: Update (2024-04-28)
5378208 damo_record: Do '--include_child_tasks' by default (2024-04-28)
7f18bec damo_wss: Do '--collapse_targets' by default (2024-04-28)
0dfe917 TODO: Remove quota goals testing todo item (2024-04-27)
c662a1f _damon_args: Wordsmith --damos_quota_goal help message (2024-04-27)
ef745d2 _damon: Use variables for quota goal metrics (2024-04-27)
...
$ git cat-file blob HEAD:README.md
DAMO: Data Access Monitoring Operator
=====================================
`damo` is a user space tool for [DAMON](https://damonitor.github.io). Using
this, you can monitor the data access patterns of your system or workloads and
make data access-aware memory management optimizations.
Demo Video
==========
Please click the below thumbnail to show the short demo video.
[](
http://www.youtube.com/watch?v=l63eqbVBZRY
"DAMON: a demo for the Kernel Summit 2020")
Getting Started
===============
[](https://repology.org/project/damo/versions)
Follow below instructions and commands to monitor and visualize the access
pattern of your workload.
$ # ensure DAMON is enabled on your kernel
$ # install damo from PyPI, or use your distribution's package manager
$ sudo pip3 install damo
$ sudo damo record $(pidof <your workload>)
$ sudo damo report heats --heatmap stdout --stdout_heatmap_color emotion
The last command will show the access pattern of your workload, like below:
FAQs
====
How can I ensure DAMON is enabled on my kernel?
-----------------------------------------------
Please refer to 'Install'
[section](https://sjp38.github.io/post/damon/#install) of the project webpage.
Where can I get more detailed usage?
------------------------------------
The below sections provide quick introductions for `damo`'s major features.
For more detailed usage, please refer to [USAGE.md](USAGE.md) file.
What does the version numbers mean?
-----------------------------------
Nothing at all but indicate which version is more fresh. A higher version
number means it is more recently released.
Will `pip3 install damo` install the latest version of `damo`?
--------------------------------------------------------------
It will install the latest _stable_ version of `damo`. If you want, you can
also install less stable but more fresh `damo` from source code. For that,
fetch the `next` branch of the source tree and use `damo` executable file in
the tree.
$ git clone https://github.com/awslabs/damo -b next
How can I participate in the development of `damo`?
---------------------------------------------------
Please refer to
[CONTRIBUTING](https://github.com/awslabs/damo/blob/next/CONTRIBUTING) file.
Why some subcommands are not documented on [USAGE.md](USAGE.md) file?
---------------------------------------------------------------------
Only sufficiently stabilized features are documented there. In other words,
any feature that not documented on [USAGE.md](USAGE.md) are in experimental
stage. Such experimental features could be deprecated and removed without any
notice and grace periods. The documented features could also be deprecated,
but those will provide some notification and grace periods.
Quick Intro for Major Features
==============================
Below are quick introductions for `damo`'s major features.
For more detailed usage, please refer to [USAGE.md](USAGE.md) file.
Snapshot Data Access Pattern
----------------------------
Below commands repeatedly get a snapshot of the access pattern of a program for
every second.
$ git clone https://github.com/sjp38/masim
$ cd masim; make; ./masim ./configs/zigzag.cfg --silent &
$ sudo damo start --target_pid $(pidof masim)
$ while :; do sudo damo show; sleep 1; done
The first two lines of the commands get an artificial memory access generator
program and run it in the background. It will repeatedly access two 100
MiB-sized memory regions one by one. You can substitute this with your real
workload.
The third line asks ``damo`` to start monitoring the access pattern of the
process. Finally, the last line retries a snapshot of the monitoring results
every second and show on terminal.
Recording Data Access Patterns
------------------------------
Below commands record memory access patterns of a program and save the
monitoring results in `damon.data` file.
$ git clone https://github.com/sjp38/masim
$ cd masim; make; ./masim ./configs/zigzag.cfg --silent &
$ sudo damo record -o damon.data $(pidof masim)
The first two lines of the commands get an artificial memory access generator
program and run it in the background. It will repeatedly access two 100
MiB-sized memory regions one by one. You can substitute this with your real
workload. The last line asks ``damo`` to record the access pattern in
``damon.data`` file.
Visualizing Recorded Patterns
-----------------------------
Below three commands visualize the recorded access patterns into three
image files.
$ damo report heats --heatmap stdout
$ damo report wss --range 0 101 1
$ damo report wss --range 0 101 1 --sortby time --plot
- ``access_pattern_heatmap.png`` will show the data access pattern in a
heatmap, which shows when (x-axis) what memory region (y-axis) is how
frequently accessed (color).
- ``wss_dist.png`` will show the distribution of the working set size.
- ``wss_chron_change.png`` will show how the working set size has
chronologically changed.
You can show the images on a web page [1]. Those made with other realistic
workloads are also available [2,3,4].
[1] https://damonitor.github.io/doc/html/latest/admin-guide/mm/damon/start.html#visualizing-recorded-patterns<br>
[2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html<br>
[3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html<br>
[4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
Data Access Pattern Aware Memory Management
-------------------------------------------
Below command makes every memory region of size >=4K that hasn't accessed for
>=60 seconds in your workload to be swapped out. By doing this, you can make
your workload more memory efficient with only modest performance overhead.
$ sudo damo start --damos_access_rate 0 0 --damos_sz_region 4K max \
--damos_age 60s max --damos_action pageout \
<pid of your workload>
Deprecated, or Will be Deprecated Features
==========================================
Below are features that recently deprecated, or will be deprecated. If you
depend on any of those, please report your usecase to the community via github
issue, sj@kernel.org, damon@lists.linux.dev, and/or linux-mm@kvack.org.
`damo translate_damos`
----------------------
Deprecated. Use the command of v2.0.2 or lower version of DAMO instead.
DAMON record binary format
--------------------------
Deprecated. Use `json_compressed` format instead.
At the beginning, DAMO used its special binary format, namely `record`. It is
designed for lightweight saving of the monitoring results. It is difficult to
read, and not that efficient compared to fancy compression techniques. `json`
based monitoring results can be easier to read, and more efficient when
compression technique is used. Hence, the format is deprecated. You may
use `damo convert_record_format` of v2.0.2 or lower version of DAMO to convert
your old record binary format monitoring results files to the new format.
Python2 support
---------------
Deprecated. Use Python3.
For some old distros, DAMO initially supported Python2. Because Python2 is
really old now, the support has deprecated. Please use Python3 or newer.
DAMOS single line format
------------------------
Deprecated. Use `--damos_*` command line options or json format input.
A simple DAMOS scheme specification format called one-line scheme specification
was initially supported. Because it is not flexible for extension of features,
it has deprecated now. You may use `--damos_*` command line options or json
format instead. You could use `damo translate_damos` ov v2.0.2 or lower
version of DAMO to convert your old single line DAMOS schemes specification to
the new json format.
--rbuf option of `damo record`
------------------------------
Deprecated.
Early versions of DAMON supported in-kernel direct monitoring results record
file generation. To control the overhead of it, DAMO allowed user to specify
the size of buffer for the work. The feature has not merged into the mainline,
and discarded. Hence the option was available for only few kernels that ported
the feature. For most of kernels, tracepoint based record file generation is
being used, and the overhead of the approach is subtle. Hence, the option has
deprecated.
# heads (aka `branches'):
$ git for-each-ref --sort=-creatordate refs/heads \
--format='%(HEAD) %(refname:short) %(subject) (%(creatordate:short))'
* master Update the version (2024-04-29)
next Update the version (2024-04-29)
# tags:
$ git for-each-ref --sort=-creatordate refs/tags \
--format='%(refname:short) %(subject) (%(creatordate:short))'
v2.3.3 DAMO 2.3.3 (2024-04-29) tar.gz
v2.3.2 DAMO 2.3.2 (2024-04-22) tar.gz
v2.3.0 DAMO 2.3.0 (2024-04-15) tar.gz
v2.2.9 DAMO 2.2.9 (2024-04-08) tar.gz
v2.2.8 DAMO 2.2.8 (2024-04-01) tar.gz
v2.2.7 DAMO 2.2.7 (2024-03-25) tar.gz
v2.2.6 DAMO 2.2.6 (2024-03-18) tar.gz
v2.2.5 DAMO 2.2.5 (2024-03-11) tar.gz
v2.2.4 DAMO 2.2.4 (2024-03-04) tar.gz
v2.2.2 DAMO 2.2.2 (2024-02-19) tar.gz
...
# associated public inboxes:
# (number on the left is used for dev purposes)
63 lkml
27 linux-mm
24 linux-doc
14 qemu-devel
14 u-boot
10 linux-arm-kernel
10 linuxppc-dev
9 dpdk-dev
8 netdev
7 alsa-devel
7 kvm
6 git
5 linux-wireless
5 xen-devel
5 linux-scsi
5 linux-btrfs
5 intel-gfx
5 linux-omap
4 linux-nfs
4 netfilter-devel
4 linux-fsdevel
4 linux-devicetree
4 linux-media
4 stable
4 linux-mips
4 kvmarm
4 linux-bluetooth
4 dri-devel
4 openbmc
4 buildroot
4 openembedded-devel
3 linux-mtd
3 bpf
3 linux-mediatek
3 linux-iommu
3 linux-input
3 driverdev-devel
3 selinux
3 linux-iio
3 linux-gpio
3 linux-rdma
3 linux-ext4
3 linux-security-module
3 linux-ide
3 linux-watchdog
3 linux-trace-devel
3 util-linux
3 amd-gfx
3 linux-api
3 linux-spi
3 linux-i2c
3 linux-kbuild
3 linux-arch
3 containers
3 linux-s390
3 yocto-meta-freescale
3 yocto-meta-arago
3 outreachy
3 intel-wired-lan
3 poky
3 grub-devel
2 linux-samsung-soc
2 linux-block
2 linux-riscv
2 linux-nvme
2 linux-xfs
2 linux-crypto
2 linux-acpi
2 linux-edac
2 linux-cifs
2 linux-arm-msm
2 linux-amlogic
2 linux-erofs
2 linux-nvdimm
2 fstests
2 linux-pm
2 linux-rtc
2 linux-usb
2 linux-renesas-soc
2 linux-kselftest
2 linux-clk
2 dmaengine
2 linux-pci
2 linux-f2fs-devel
2 linux-modules
2 linux-rt-users
2 linux-serial
2 backports
2 io-uring
2 linux-man
2 linux-mmc
2 linux-sparse
2 linux-tegra
2 lttng-dev
2 virtualization
2 linux-fpga
2 linux-can
2 linux-rockchip
2 linux-raid
2 platform-driver-x86
2 dm-devel
2 linux-fbdev
2 linux-cxl
2 linux-perf-users
2 lustre-devel
2 linux-sh
2 nouveau
2 linux-staging
2 linux-coco
2 nvdimm
2 fio
2 iwd
2 ltp
2 yocto
2 openembedded-core
2 yocto-toaster
2 yocto-meta-arm
2 bitbake-devel
2 yocto-meta-virtualization
2 chrome-platform
2 yocto-meta-ti
2 openrisc
2 kexec
2 intel-xe
2 linux-um
2 virtio-comment
2 qemu-riscv
2 cluster-devel
2 cgroups
2 kvm-ppc
2 linux-hexagon
2 reiserfs-devel
2 lvm-devel
2 virtio-fs
2 ../../../../../../../fuego
2 ../../../../../../../igt-dev
1 linux-fscrypt
1 cocci
1 live-patching
1 linux-efi
1 linux-integrity
1 linux-snps-arc
1 linux-hwmon
1 kernel-hardening
1 linux-next
1 linux-parisc
1 linux-leds
1 kernelnewbies
1 linux-sgx
1 linux-hyperv
1 workflows
1 rcu
1 selinux-refpolicy
1 linux-m68k
1 linux-i3c
1 linux-spdx
1 linux-wpan
1 ksummit-discuss
1 wireguard
1 linux-kernel-mentees
1 linux-csky
1 tpmdd-devel
1 linux-unionfs
1 linux-firmware
1 cip-dev
1 linux-audit
1 linux-remoteproc
1 linux-dash
1 linux-bcache
1 mm-commits
1 linux-pwm
1 ceph-devel
1 xdp-newbies
1 phone-devel
1 linux-hardening
1 dash
1 keyrings
1 linux-sctp
1 target-devel
1 soc
1 ocfs2-devel
1 rust-for-linux
1 ath10k
1 ath11k
1 linux-phy
1 kernel-janitors
1 sparclinux
1 linux-sunxi
1 mptcp
1 regressions
1 ksummit
1 b43-dev
1 linux-nfc
1 linux-bcachefs
1 ath9k-devel
1 ntfs3
1 llvm
1 ell
1 ofono
1 yocto-docs
1 linux-patches
1 ntb
1 xenomai
1 damon
1 asahi
1 loongarch
1 imx
1 ath12k
1 b4-sent
1 linux-trace-kernel
1 oe-linux-nfc
1 oe-kbuild-all
1 oe-chipsec
1 batman
1 virtio-dev
1 v9fs
1 ecryptfs
1 linux-ia64
1 kbd
1 autofs
1 cpufreq
1 dccp
1 devicetree-spec
1 devicetree-compiler
1 initramfs
1 hail-devel
1 kvm-ia64
1 linux-8086
1 kernel-testers
1 linux-alpha
1 linux-btrace
1 linux-embedded
1 linux-hams
1 linux-hotplug
1 linux-laptop
1 linux-sound
1 trinity
1 linux-metag
1 linux-x25
1 linux-nilfs
1 lvs-devel
1 netfilter
1 linux-oxnas
1 u-boot-amlogic
1 lm-sensors
1 acpica-devel
1 perfbook
1 smatch
1 ../../../../../../../wireless-regdb
1 ../../../../../../../powertop
git clone https://80x24.org/lore/pub/scm/linux/kernel/git/sj/damo.git