* "ip_conntrack_core: Frag of proto 17." error and memory leak?
@ 2003-09-15 16:43 c0g
2003-09-16 8:19 ` Maciej Soltysiak
2003-09-20 12:11 ` Martin Josefsson
0 siblings, 2 replies; 8+ messages in thread
From: c0g @ 2003-09-15 16:43 UTC (permalink / raw
To: netfilter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
I'm getting this error, my kern.log is flooded, it is logged many
hundred of thousands times. I read in ip_conntrack_core.c it should
"never happen" (so it's kind of assertion). Why does this assertion fail?
I'm using 2.4.22 kernel with CVS p-o-m 20030907. I'm using only
following major netfilter patches:
TCPLAG.patch
ipt_REJECT-fake-source.patch
raw.patch
and some minor:
39_ip_conntrack-proc.patch
40_nf-log.patch
54_ip_nat-macro-args.patch
58-ip_conntrack-macro-args.patch
59_ip_nat_h-unused-var.patch
60_nat_tftp-remove-warning.patch
61-remove-memsets.patch
63_getorigdst-tuple-zero.patch
74_nat-range-fix.patch
ip_ct_refresh_optimization.patch
I made rule that blocks fragments, but it doesn't stop messages (perhaps
because the check is performed before rules are evaluated).
Also, I suppose there is big kernel memory leak that happen about 12-24
hours after boot. It eats up to 40 MB of RAM memory in few seconds. If
someone is interested I made some graphs of vmstat (excel) at
http://ng.pl/~jp/vmstat-n5/
Could it be caused by failed "Frag of proto" assertion?
Is there someone who had this problem? And solved it?
Please give me some pointers what should I do.
- --
c0g@wp.pl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/ZewzPqmVt5WhbA8RAq3HAJ9Ahvz19BQNE4MXhLxzCyIuE4fqCwCfUwaw
eEhLSl0iX7j9KZr8kfsPIZ8=
=AiO4
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-15 16:43 "ip_conntrack_core: Frag of proto 17." error and memory leak? c0g
@ 2003-09-16 8:19 ` Maciej Soltysiak
2003-09-17 14:22 ` c0g
2003-09-20 12:11 ` Martin Josefsson
1 sibling, 1 reply; 8+ messages in thread
From: Maciej Soltysiak @ 2003-09-16 8:19 UTC (permalink / raw
To: c0g; +Cc: netfilter
Hello,
On Mon, 15 Sep 2003, c0g wrote:
> I made rule that blocks fragments, but it doesn't stop messages (perhaps
> because the check is performed before rules are evaluated).
How about investigating why are you getting fragments ? Maybe you can
eliminate the problem by avoiding fragmentation.
Anyway, as I remember the conntrack mechanism assembles fragments for
connection tracking purposes, thus it's denoted as /* Never happen */
Maybe you are being flooded, and the conntrack mechanism sees, something
like new fragmented UDP packets that cannot be assembled. Run iptraf,
tcpdump, snort or whatever you wish.
This could explain the memleak: conntrack reserves memory for each udp
fragment and hopes to assemble it, which never happens. I guess that
this should be free'd at some point, after failing to assemble the data,
but I am not sure.
If I am correct, we have a problem in the conntrack mechanism.
And a Netfilter DoS.
> Could it be caused by failed "Frag of proto" assertion?
If I am correct, or at least close, it might.
> Is there someone who had this problem? And solved it?
I never heard of such problems.
Regards,
Maciej
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-16 8:19 ` Maciej Soltysiak
@ 2003-09-17 14:22 ` c0g
2003-09-17 16:56 ` Maciej Soltysiak
0 siblings, 1 reply; 8+ messages in thread
From: c0g @ 2003-09-17 14:22 UTC (permalink / raw
To: Maciej Soltysiak; +Cc: netfilter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
| How about investigating why are you getting fragments ? Maybe you can
| eliminate the problem by avoiding fragmentation.
I tcpdumped it. It seems it is big syslog messages generated by one of
my hosts which don't fit in one packet. Syslog uses udp, so it fragment
these packets. And there are some minor fragmented udp packets from
users (perhaps some p2p program generates them).
| Anyway, as I remember the conntrack mechanism assembles fragments for
| connection tracking purposes, thus it's denoted as /* Never happen */
| Maybe you are being flooded, and the conntrack mechanism sees, something
| like new fragmented UDP packets that cannot be assembled. Run iptraf,
| tcpdump, snort or whatever you wish.
It is strange. These messages are logged even if fragments rate (sniffed
with tcpdump) is very low. It doesn't look like messages are indicating
fragments flood.
| This could explain the memleak: conntrack reserves memory for each udp
| fragment and hopes to assemble it, which never happens. I guess that
| this should be free'd at some point, after failing to assemble the data,
| but I am not sure.
I investigated it more deeply - log messages and memory leaks don't
correlate. The first happens in other time than second. Sorry for bad hint.
I was supposing these "leaks", because my box was hanging every day. I
replaced hardware and it stopped.
RAM "jumps" still happen. But now, when box uptime is > 1day I see RAM
returns to norm, some time after "leak". I spend long time tracking it,
and I know only one: RAM is not eaten by processes for sure. But
increased process activity triggers RAM leak (and I know - increased
process activity triggers cache and buffers grow, but I substracted
these in my tests). After processes end, there is fewer RAM than before.
But this problem appears now to be a little offtopic.
And now, when my box isn't hanging such frequently it isn't so important ;-)
Greets!
- --
c0g@wp.pl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE/aG4tPqmVt5WhbA8RAkJnAJ9snfyKYWJeyXnv9i7WJqFvBacUSQCeNpZt
s9XiHzavRoLQK7AEiSk6lA8=
=cz2P
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-17 14:22 ` c0g
@ 2003-09-17 16:56 ` Maciej Soltysiak
0 siblings, 0 replies; 8+ messages in thread
From: Maciej Soltysiak @ 2003-09-17 16:56 UTC (permalink / raw
To: c0g; +Cc: netfilter
On Wed, 17 Sep 2003, c0g wrote:
> I tcpdumped it. It seems it is big syslog messages generated by one of
> my hosts which don't fit in one packet. Syslog uses udp, so it fragment
> these packets. And there are some minor fragmented udp packets from
> users (perhaps some p2p program generates them).
So I'm puzzled. You can try sending a brief note about it to
netfilter-devel@lists.netfilter.org
> But this problem appears now to be a little offtopic.
> And now, when my box isn't hanging such frequently it isn't so important ;-)
Good for you.
Regards,
Maciej
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-15 16:43 "ip_conntrack_core: Frag of proto 17." error and memory leak? c0g
2003-09-16 8:19 ` Maciej Soltysiak
@ 2003-09-20 12:11 ` Martin Josefsson
2003-09-21 15:04 ` Harald Welte
1 sibling, 1 reply; 8+ messages in thread
From: Martin Josefsson @ 2003-09-20 12:11 UTC (permalink / raw
To: c0g; +Cc: Netfilter, Netfilter-devel
On Mon, 2003-09-15 at 18:43, c0g wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
> I'm getting this error, my kern.log is flooded, it is logged many
> hundred of thousands times. I read in ip_conntrack_core.c it should
> "never happen" (so it's kind of assertion). Why does this assertion fail?
>
> I'm using 2.4.22 kernel with CVS p-o-m 20030907. I'm using only
> following major netfilter patches:
>
> TCPLAG.patch
> ipt_REJECT-fake-source.patch
> raw.patch
Are you using the NOTRACK target in raw/PREROUTING?
This is the only source of this I can see and it is indeed a bug. This
message shouldn't be printed for this case. It doesn't defrag packets
and calls get_tuple() for all packets, even the fragments and
get_tuple() barfs. This causes only the first fragment to be marked as
untracked and leaves the rest of the fragments to conntrack which tries
to defrag them but that won't succeed since conntrack will never see the
first packet. And thus the kernel keeps lots of fragments in memory
until they time out and are dropped.
We'll look into how to best solve this if this is what is causing the
problem for you.
--
/Martin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-20 12:11 ` Martin Josefsson
@ 2003-09-21 15:04 ` Harald Welte
2003-09-21 20:03 ` Martin Josefsson
2003-09-24 5:11 ` Jozsef Kadlecsik
0 siblings, 2 replies; 8+ messages in thread
From: Harald Welte @ 2003-09-21 15:04 UTC (permalink / raw
To: Martin Josefsson; +Cc: c0g, jozsef Netfilter-devel
[-- Attachment #1: Type: text/plain, Size: 2016 bytes --]
On Sat, Sep 20, 2003 at 02:11:16PM +0200, Martin Josefsson wrote:
> Are you using the NOTRACK target in raw/PREROUTING?
>
> This is the only source of this I can see and it is indeed a bug. This
> message shouldn't be printed for this case. It doesn't defrag packets
> and calls get_tuple() for all packets, even the fragments and
> get_tuple() barfs. This causes only the first fragment to be marked as
> untracked and leaves the rest of the fragments to conntrack which tries
> to defrag them but that won't succeed since conntrack will never see the
> first packet. And thus the kernel keeps lots of fragments in memory
> until they time out and are dropped.
Mh. So why does it call get_tuple() anyway? To check if the tuple is
taken (i.e. the packet is part of an already tracked connection)? I
think we can safely split this check, since
1) it tries to work around an obvious misconfiguration by the user
2) it adds significant runtime cost to the NOTRACK target (call
get_tuple, which calls in turn the l4 protocol helper get_tuple, then
tuple_taken(), which in turn locks the conntrack lock, ...)
3) it introduces the problem above.
Yes, we can still try to make it work, even with that check. But I
think we can just skip it and make our life easier.
Jozsef?
btw: I wan to merge the 'raw' table userspace changes into iptables
before 1.2.9 is released... All we need to make it compile even on a
non-rawtable-kernel is:
#ifndef IPT_CONNTRACK_STATE_UNTRACKED
#define IPT_CONNTRACK_STATE_UNTRACKED (1 << (IP_CT_IS_REPLY+3))
#endif
in one of our (userspace) header files.
> /Martin
--
- Harald Welte <laforge@netfilter.org> http://www.netfilter.org/
============================================================================
"Fragmentation is like classful addressing -- an interesting early
architectural error that shows how much experimentation was going
on while IP was being designed." -- Paul Vixie
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-21 15:04 ` Harald Welte
@ 2003-09-21 20:03 ` Martin Josefsson
2003-09-24 5:11 ` Jozsef Kadlecsik
1 sibling, 0 replies; 8+ messages in thread
From: Martin Josefsson @ 2003-09-21 20:03 UTC (permalink / raw
To: Harald Welte; +Cc: c0g, Jozsef Kadlecsik, Netfilter-devel
On Sun, 2003-09-21 at 17:04, Harald Welte wrote:
> Mh. So why does it call get_tuple() anyway? To check if the tuple is
> taken (i.e. the packet is part of an already tracked connection)? I
> think we can safely split this check, since
>
> 1) it tries to work around an obvious misconfiguration by the user
> 2) it adds significant runtime cost to the NOTRACK target (call
> get_tuple, which calls in turn the l4 protocol helper get_tuple, then
> tuple_taken(), which in turn locks the conntrack lock, ...)
> 3) it introduces the problem above.
>
> Yes, we can still try to make it work, even with that check. But I
> think we can just skip it and make our life easier.
That's how it used to work but that wasn't Jozsef's intentions with the
target iirc.
> btw: I wan to merge the 'raw' table userspace changes into iptables
> before 1.2.9 is released... All we need to make it compile even on a
> non-rawtable-kernel is:
>
> #ifndef IPT_CONNTRACK_STATE_UNTRACKED
> #define IPT_CONNTRACK_STATE_UNTRACKED (1 << (IP_CT_IS_REPLY+3))
> #endif
>
> in one of our (userspace) header files.
This creates a problem with p-o-m.
If we release a new p-o-m version before 1.2.9 is released it will
contain the userspace part of the raw patch. Then when 1.2.9 is released
it will already be patched with that patch and p-o-m will fail.
--
/Martin
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: "ip_conntrack_core: Frag of proto 17." error and memory leak?
2003-09-21 15:04 ` Harald Welte
2003-09-21 20:03 ` Martin Josefsson
@ 2003-09-24 5:11 ` Jozsef Kadlecsik
1 sibling, 0 replies; 8+ messages in thread
From: Jozsef Kadlecsik @ 2003-09-24 5:11 UTC (permalink / raw
To: Harald Welte; +Cc: Martin Josefsson, c0g, jozsef Netfilter-devel
On Sun, 21 Sep 2003, Harald Welte wrote:
> Mh. So why does it call get_tuple() anyway? To check if the tuple is
> taken (i.e. the packet is part of an already tracked connection)? I
> think we can safely split this check, since
>
> 1) it tries to work around an obvious misconfiguration by the user
> 2) it adds significant runtime cost to the NOTRACK target (call
> get_tuple, which calls in turn the l4 protocol helper get_tuple, then
> tuple_taken(), which in turn locks the conntrack lock, ...)
> 3) it introduces the problem above.
>
> Yes, we can still try to make it work, even with that check. But I
> think we can just skip it and make our life easier.
>
> Jozsef?
I'll release an updated patch before the weekend (together with the
non-userspace 40_nf_log patch).
> btw: I wan to merge the 'raw' table userspace changes into iptables
> before 1.2.9 is released... All we need to make it compile even on a
> non-rawtable-kernel is:
>
> #ifndef IPT_CONNTRACK_STATE_UNTRACKED
> #define IPT_CONNTRACK_STATE_UNTRACKED (1 << (IP_CT_IS_REPLY+3))
> #endif
>
> in one of our (userspace) header files.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2003-09-24 5:11 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-09-15 16:43 "ip_conntrack_core: Frag of proto 17." error and memory leak? c0g
2003-09-16 8:19 ` Maciej Soltysiak
2003-09-17 14:22 ` c0g
2003-09-17 16:56 ` Maciej Soltysiak
2003-09-20 12:11 ` Martin Josefsson
2003-09-21 15:04 ` Harald Welte
2003-09-21 20:03 ` Martin Josefsson
2003-09-24 5:11 ` Jozsef Kadlecsik
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.