From: Brian Baboch <brian.baboch@gmail.com>
To: "Saleem, Shiraz" <shiraz.saleem@intel.com>,
Jason Gunthorpe <jgg@ziepe.ca>, Zhu Yanjun <yanjun.zhu@linux.dev>
Cc: Konstantin Taranov <kotaranov@microsoft.com>,
Leon Romanovsky <leon@kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
"stable@vger.kernel.org" <stable@vger.kernel.org>,
"florent.fourcot@wifirst.fr" <florent.fourcot@wifirst.fr>,
"brian.baboch@wifirst.fr" <brian.baboch@wifirst.fr>
Subject: Re: Excessive memory usage when infiniband config is enabled
Date: Mon, 13 May 2024 15:59:45 +0200 [thread overview]
Message-ID: <6dc1d533-122b-475d-a908-c15026c8b345@gmail.com> (raw)
In-Reply-To: <IA1PR11MB7317B56B84421912ED6E856CE9E52@IA1PR11MB7317.namprd11.prod.outlook.com>
Hello,
Thank you for your answers.
It's unfortunate that by default the irdma module uses an extra 5Gb of
RAM, which is huge (more than 30% of the available RAM) and that there's
practically no way of reducing it without deactivating the module since
the limits_sel parameter is not available in the in-tree driver, as
mentioned by Shiraz (I can confirm that gen1_limits_sel=0 works on
my x722 card, I tested it, but it looks out of topic for me since it's
not in the tree).
For my case, since I don't need the irdma module, I will just blacklist
it as suggested, but I think that it would be better to change the
default value of the resource limit selector so it doesn't consume this
much RAM if it's not used.
On 5/8/24 03:24, Saleem, Shiraz wrote:
>> Subject: Re: Excessive memory usage when infiniband config is enabled
>>
>> On Tue, May 07, 2024 at 05:24:51PM +0200, Zhu Yanjun wrote:
>>> 在 2024/5/7 15:32, Konstantin Taranov 写道:
>>>>> Hello Leon,
>>>>>
>>>>> I feel that it's a bug because I don't understand why is this
>>>>> module/option allocating 6GB of RAM without any explicit configuration or
>> usage from us.
>>>>> It's also worth mentioning that we are using the default
>>>>> linux-image from Debian bookworm, and it took us a long time to
>>>>> understand the reason behind this memory increase by bisecting the
>> kernel's config file.
>>>>> Moreover the documentation of the module doesn't mention anything
>>>>> regarding additional memory usage, we're talking about an increase
>>>>> of 6Gb which is huge since we're not using the option.
>>>>> So is that an expected behavior, to have this much increase in the
>>>>> memory consumption, when activating the RDMA option even if we're
>>>>> not using it ? If that's the case, perhaps it would be good to
>>>>> mention this in the documentation.
>>>>>
>>>>> Thank you
>>>>>
>>>>
>>>> Hi Brian,
>>>>
>>>> I do not think it is a bug. The high memory usage seems to come from these
>> lines:
>>>> rsrc_size = irdma_calc_mem_rsrc_size(rf);
>>>> rf->mem_rsrc = vzalloc(rsrc_size);
>>>
>>> Exactly. The memory usage is related with the number of QP.
>>> When on irdma, the Queue Pairs is 4092, Completion Queues is 8189, the
>>> memory usage is about 4194302.
>>>
>>> The command "modprobe irdma limits_sel" will change QP numbers.
>>> 0 means minimum, up to 124 QPs.
>>>
>>> Please use the command "modprobe irdma limits_sel=0" to make tests.
>>> Please let us know the test results.
>>
>> It seems like a really unfortunate design choice in this driver to not have dynamic
>> memory allocation.
>>
>> Burning 6G on every server that has your HW, regardless if any RDMA apps are
>> run, seems completely excessive.
>>
>
> So the driver requires to pre-allocate backing pages for HW context objects during device initialization. At least for the x722 and e800 series product lines.
>
> And the amount of memory allocated is proportional to the max QP (primarily) setup for the function.
>
> One option is to set a lower default profile upon driver loading; which will reduce the memory footprint; but exposes lower QP and other verb resources per ib_device.
> And provide users with a devlink knob to choose a larger/smaller profile as they see fit.
>
> This is sort of what limits_sel module parameter Yanjun suggested realizes, but it is not available in the in-tree driver.
>
> Between, what is the specific Intel NIC model in use?
> lspci -vv | grep -E 'Eth.*Intel|Product'
>
> Shiraz
>
prev parent reply other threads:[~2024-05-13 13:59 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-06 15:15 Excessive memory usage when infiniband config is enabled Brian Baboch
2024-05-07 11:27 ` Leon Romanovsky
2024-05-07 13:15 ` Brian Baboch
2024-05-07 13:32 ` Konstantin Taranov
2024-05-07 15:24 ` Zhu Yanjun
2024-05-07 16:37 ` Jason Gunthorpe
2024-05-08 1:24 ` Saleem, Shiraz
2024-05-13 13:59 ` Brian Baboch [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6dc1d533-122b-475d-a908-c15026c8b345@gmail.com \
--to=brian.baboch@gmail.com \
--cc=brian.baboch@wifirst.fr \
--cc=florent.fourcot@wifirst.fr \
--cc=jgg@ziepe.ca \
--cc=kotaranov@microsoft.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=shiraz.saleem@intel.com \
--cc=stable@vger.kernel.org \
--cc=yanjun.zhu@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).