stgt.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

From: Jelle de Jong <jelledejong@powercraft.nl>
To: stgt@vger.kernel.org
Subject: how to configure a drbd tgtd kvm cluster to make sure one machine doesn't eat up all the IO and hangs other nodes
Date: Sun, 25 May 2014 15:14:22 +0200	[thread overview]
Message-ID: <5381ECAE.60708@powercraft.nl> (raw)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everybody,

I got a drbd cluster with tgtd managed by pacemaker as my iscsi target

I then got three kvm host servers connected with open-iscsi to this
target.

All servers use an dual nic 1G LACP (mode 4 bonding) connection to a
cisco switch.

On the servers the iscsi disk is managed by lvm.

I use the bellow for all the disks in the drbd cluster servers and on
for all the disks on the kvm hosts (so also the iscsi disks).

echo deadline > /sys/block/sdi/queue/scheduler
echo 0 > /sys/block/sdi/queue/iosched/front_merges
echo 150 > /sys/block/sdi/queue/iosched/read_expire
echo 1500 > /sys/block/sdi/queue/iosched/write_expire

My first problem is that if i run dd[1] on one of the KVM host, all
the kvm guests on the other machines drop dead because they cant get
any IO any more.

My second problem is that if I run a heavy IO job[2] inside a KVM
guest it can take all the IO and the KVM guest on an other KVM host
can't get any IO any more

[1] ionice -c 3 dd oflag=direct bs=4M if=/dev/lvm2-vol/kvm07-snapshot
of=/dev/lvm3-vol/kvm07-disk

[2] nice -n 19 ionice -c2 -n7 pvmove /dev/vdb /dev/vdc

Now my guess is this is because, there is no IO scheduling between the
KVM host, they do IO scheduling on there own server but just pass the
load to open-iscsi and tgtd. However if multiple server are connected
to tgtd it seems to be that one server can take it all and the other
connections just have to wait.... (till a heavy dd job that takes
hours runs out...)

So how should I configure my cluster and tgtd so an heavy IO jobs just
doesn’t get all the IO and the other request get handled as well? Is
this an impossibility?

Kind regards,

Jelle de Jong

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iJwEAQECAAYFAlOB7KkACgkQ1WclBW9j5HnZ8gP/faFZk0CHaefPgMlXWHQ8N0YT
7Cv3XFUUTenVYbsAam5KuNMb48QzEMQnNhKeS3LeBELPUBAxMOAh/meXHVxLaDiK
L3ZnhLuxdWOsDCp1hSaIDAekPqhvUdxjfOznqbOfxjLGjXNYdPai1ZZK0AVUog/z
sdDzFWRmTLEPHrTR3HM=
=zVop
-----END PGP SIGNATURE-----

next             reply	other threads:[~2014-05-25 13:14 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-05-25 13:14 Jelle de Jong [this message]
2014-05-25 17:17 ` how to configure a drbd tgtd kvm cluster to make sure one machine doesn't eat up all the IO and hangs other nodes Jelle de Jong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5381ECAE.60708@powercraft.nl \
    --to=jelledejong@powercraft.nl \
    --cc=stgt@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).