* [ANNOUNCE] Open POSIX Test Suite
@ 2002-11-04 22:48 Geoff Gustafson
2002-11-04 22:58 ` Larry McVoy
` (2 more replies)
0 siblings, 3 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-04 22:48 UTC (permalink / raw
To: linux-kernel
I would like to announce a new project to develop and/or assemble a GPL test
suite for POSIX APIs. The tests will focus on conformance to the IEEE Std
1003.1-2001, but will also include separate functional and stress tests.
The project's current approach to conformance testing is to record
assertions
from a close reading of the POSIX specifications, and write minimal test
cases
that prove or disprove these assertions. The test suite will be independent
of
specific API implementations, and will eventually be easily configurable to
work with different implementations. The project aims for OS independence,
using only POSIX APIs, the autoconf suite, and simple shell support.
However,
it is currently only being tested on Linux.
Ultimately, the plan is to use the test suite to evaluate current support in
Linux, as well as new implementations being considered in the open source
community, and then contribute patches or at least bug reports (with a
minimal
test case) to the appropriate places, like LKML.
Contributions of any test cases, review of the work, discussion of the
approach, etc. are very welcome. Join the development mailing list,
posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
Semaphores, and Clocks & Timers, based on current interests and resources.
You can help in these areas, or start work on another area of the spec.
There
will need to be some uniformity across the suite, but many details have yet
to
be worked out, so your involvement in those decisions help a lot.
For more information, see the project website at
http://posixtest.sourceforge.net
Thanks,
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-04 22:48 [ANNOUNCE] Open POSIX Test Suite Geoff Gustafson
@ 2002-11-04 22:58 ` Larry McVoy
2002-11-04 23:17 ` Geoff Gustafson
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
2002-11-04 23:57 ` [ANNOUNCE] Open POSIX Test Suite Christopher Yeoh
2 siblings, 1 reply; 24+ messages in thread
From: Larry McVoy @ 2002-11-04 22:58 UTC (permalink / raw
To: Geoff Gustafson; +Cc: linux-kernel
On Mon, Nov 04, 2002 at 02:48:47PM -0800, Geoff Gustafson wrote:
> I would like to announce a new project to develop and/or assemble a GPL test
> suite for POSIX APIs.
Great idea. We can help in the following way: BitKeeper has an extremely
simple test harness used for regressions. It's well thought out in that
it is trivial to write simple tests and run them in isolation or to
run the whole suite. If you want the harness, we'll give it to you
under whatever license you want, I assume GPL, but we don't care.
You can see what the tests look like in BK, if you have it installed, we
ship all the tests, they are in `bk bin`/t
A simple test might be
#!/bin/sh
# test that touch creates a file
touch foo
test -f foo || {
echo failed to create foo
exit 1
}
The harness takes care of putting you in a clean isolated environment.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
^ permalink raw reply [flat|nested] 24+ messages in thread
* RFC: A POSIX Linux project?
2002-11-04 22:48 [ANNOUNCE] Open POSIX Test Suite Geoff Gustafson
2002-11-04 22:58 ` Larry McVoy
@ 2002-11-04 23:14 ` Jeff Garzik
2002-11-04 23:31 ` Andreas Dilger
` (2 more replies)
2002-11-04 23:57 ` [ANNOUNCE] Open POSIX Test Suite Christopher Yeoh
2 siblings, 3 replies; 24+ messages in thread
From: Jeff Garzik @ 2002-11-04 23:14 UTC (permalink / raw
To: linux-kernel
I wonder if any vendors, or independent groups, would be interested in
maintaining a POSIX compliancy patchkit for the Linux kernel?
IMO such a "POSIX Linux" project would be useful for several reasons.
Overall, I think there is pressure from several directions to get all
sorts of POSIX APIs into the kernel. On occasion, kernel hackers are
confronted with a situation where complete POSIX compliancy may mean a
compromise in some area, be it performance, security, API issues, code
cleanliness issues, etc. Or simply that the POSIX-related code just
isn't ready to be merged into the mainline kernel yet.
The vendors also benefit by this, because the barrier to entry in
POSIX-related cases would be lowered, which would in turn satisfy the
demands of customers. Which would in turn give the mainline kernel all
the software engineering benefits that come from a more reasoned and
gradual review and merge of new features.
Does something like this already exist? This would need to be an open,
vendor-neutral project...
Jeff
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-04 22:58 ` Larry McVoy
@ 2002-11-04 23:17 ` Geoff Gustafson
0 siblings, 0 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-04 23:17 UTC (permalink / raw
To: Larry McVoy, linux-kernel
Larry McVoy wrote:
> Great idea. We can help in the following way: BitKeeper has an extremely
> simple test harness used for regressions. It's well thought out in that
> it is trivial to write simple tests and run them in isolation or to
> run the whole suite. If you want the harness, we'll give it to you
> under whatever license you want, I assume GPL, but we don't care.
Yeah, that would be great. GPL would be correct.
> You can see what the tests look like in BK, if you have it installed, we
> ship all the tests, they are in `bk bin`/t
I haven't used BK yet, but I'll go try to do that.
Thanks,
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* re: [ANNOUNCE] Open POSIX Test Suite
@ 2002-11-04 23:29 Dan Kegel
2002-11-05 0:04 ` Geoff Gustafson
0 siblings, 1 reply; 24+ messages in thread
From: Dan Kegel @ 2002-11-04 23:29 UTC (permalink / raw
To: Geoff Gustafson, Linux Kernel Mailing List
"Geoff Gustafson" <geoff@linux.co.intel.com> wrote:
> I would like to announce a new project to develop and/or
> assemble a GPL test suite for POSIX APIs.
You are about to duplicate http://ltp.sf.net
I imagine the main difference is you're targeting Posix
compliance rather than LSB compliance. That seems
like a fairly minor difference that could be
accommodated within the framework of the LTP.
Or do you feel the existing test framework
and tests assembled in LTP are so inadequate that you
need to start an entirely new project doing
essentially the same thing?
(Apologies if you've already answered this question. I did
check your web page, but didn't see an answer.)
- Dan
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
@ 2002-11-04 23:31 ` Andreas Dilger
2002-11-04 23:51 ` Jim Freeman
2002-11-05 2:01 ` Geoff Gustafson
2002-11-04 23:37 ` Rik van Riel
2002-11-05 1:14 ` Geoff Gustafson
2 siblings, 2 replies; 24+ messages in thread
From: Andreas Dilger @ 2002-11-04 23:31 UTC (permalink / raw
To: Jeff Garzik; +Cc: linux-kernel
On Nov 04, 2002 18:14 -0500, Jeff Garzik wrote:
> I wonder if any vendors, or independent groups, would be interested in
> maintaining a POSIX compliancy patchkit for the Linux kernel?
>
> IMO such a "POSIX Linux" project would be useful for several reasons.
> Overall, I think there is pressure from several directions to get all
> sorts of POSIX APIs into the kernel. On occasion, kernel hackers are
> confronted with a situation where complete POSIX compliancy may mean a
> compromise in some area, be it performance, security, API issues, code
> cleanliness issues, etc. Or simply that the POSIX-related code just
> isn't ready to be merged into the mainline kernel yet.
>
> The vendors also benefit by this, because the barrier to entry in
> POSIX-related cases would be lowered, which would in turn satisfy the
> demands of customers. Which would in turn give the mainline kernel all
> the software engineering benefits that come from a more reasoned and
> gradual review and merge of new features.
>
> Does something like this already exist? This would need to be an open,
> vendor-neutral project...
What about the existing POSIX test suite from X/Open? I don't know what
the current license is, but it is certainly freely downloadable from
their website. However, it is a pain in the a** to set up and run, so
a new version would definitely be welcome.
I would give you a URL, but I don't have access to my mail archives now.
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
2002-11-04 23:31 ` Andreas Dilger
@ 2002-11-04 23:37 ` Rik van Riel
2002-11-05 1:14 ` Geoff Gustafson
2 siblings, 0 replies; 24+ messages in thread
From: Rik van Riel @ 2002-11-04 23:37 UTC (permalink / raw
To: Jeff Garzik; +Cc: linux-kernel
On Mon, 4 Nov 2002, Jeff Garzik wrote:
> I wonder if any vendors, or independent groups, would be interested in
> maintaining a POSIX compliancy patchkit for the Linux kernel?
I'd be interested in helping. There must be some POSIXisms
that are not in the kernel yet but still useful enough to keep
alive.
regards,
Rik
--
Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/ http://distro.conectiva.com/
Current spamtrap: <a href=mailto:"october@surriel.com">october@surriel.com</a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:31 ` Andreas Dilger
@ 2002-11-04 23:51 ` Jim Freeman
2002-11-05 0:14 ` Geoff Gustafson
2002-11-05 2:01 ` Geoff Gustafson
1 sibling, 1 reply; 24+ messages in thread
From: Jim Freeman @ 2002-11-04 23:51 UTC (permalink / raw
To: Jeff Garzik, linux-kernel
On Mon, Nov 04, 2002 at 04:31:13PM -0700, Andreas Dilger wrote:
...
> What about the existing POSIX test suite from X/Open? I don't know what
> the current license is, but it is certainly freely downloadable from
> their website. However, it is a pain in the a** to set up and run, so
> a new version would definitely be welcome.
>
> I would give you a URL, but I don't have access to my mail archives now.
Howzabout the NIST one ?
http://www.itl.nist.gov/div897/ctg/posix_form.htm
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-04 22:48 [ANNOUNCE] Open POSIX Test Suite Geoff Gustafson
2002-11-04 22:58 ` Larry McVoy
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
@ 2002-11-04 23:57 ` Christopher Yeoh
2002-11-05 0:44 ` Geoff Gustafson
2 siblings, 1 reply; 24+ messages in thread
From: Christopher Yeoh @ 2002-11-04 23:57 UTC (permalink / raw
To: Geoff Gustafson; +Cc: linux-kernel
Hi Geoff,
At 2002/11/4 14:48-0800 Geoff Gustafson writes:
>
> Contributions of any test cases, review of the work, discussion of the
> approach, etc. are very welcome. Join the development mailing list,
> posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
> Semaphores, and Clocks & Timers, based on current interests and resources.
> You can help in these areas, or start work on another area of the spec.
> There
> will need to be some uniformity across the suite, but many details have yet
> to
> be worked out, so your involvement in those decisions help a lot.
Have you looked at the LSB test suites yet? They already cover much of
what you plan on writing tests for, though we would welcome any
volunteers who would like to increase the coverage. Most of the tests
suites are released under the Artistic License, with quite a bit of
the code donated by the Open Group (originally from the Unix
certification tests).
You can find more information about them here:
http://www.linuxbase.org/test
The CVS repository is on SourceForge in the LSB project.
Chris
--
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-04 23:29 Dan Kegel
@ 2002-11-05 0:04 ` Geoff Gustafson
2002-11-05 0:08 ` Dan Kegel
2002-11-05 0:24 ` Dan Kegel
0 siblings, 2 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-05 0:04 UTC (permalink / raw
To: Dan Kegel, Linux Kernel Mailing List; +Cc: sglass
> You are about to duplicate http://ltp.sf.net
My understanding is that LTP is focused on current mainline kernel testing,
while this project's initial concern is areas that are not currently in Linux
like POSIX message queues, semaphores, and full support for POSIX threads. I see
this as being used to evaluate different implementations that are being
considered for inclusion in the kernel, glibc, etc.
This project is concerned with the POSIX APIs regardless of where they are
implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent of
implementation. This project will be more concerned with traceability back to
the POSIX specification, and completeness of coverage, than I would expect from
LTP.
That said, there is some overlap, and an exchange of test cases between the
projects may be very useful.
I've copied Stephanie from LTP to get her reaction.
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:04 ` Geoff Gustafson
@ 2002-11-05 0:08 ` Dan Kegel
2002-11-05 15:44 ` Nathan Straz
2002-11-05 0:24 ` Dan Kegel
1 sibling, 1 reply; 24+ messages in thread
From: Dan Kegel @ 2002-11-05 0:08 UTC (permalink / raw
To: Geoff Gustafson; +Cc: Linux Kernel Mailing List, sglass
Geoff Gustafson wrote:
>>You are about to duplicate http://ltp.sf.net
>
>
> My understanding is that LTP is focused on current mainline kernel testing,
> while this project's initial concern is areas that are not currently in Linux
> like POSIX message queues, semaphores, and full support for POSIX threads. I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
>
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent of
> implementation. This project will be more concerned with traceability back to
> the POSIX specification, and completeness of coverage, than I would expect from
> LTP.
>
> That said, there is some overlap, and an exchange of test cases between the
> projects may be very useful.
>
> I've copied Stephanie from LTP to get her reaction.
Geoff,
thanks for the reply. I have a feeling that LTP would be overjoyed
to have you contribute to the LTP and make it a more accurate Posix
compliance test. The areas that Linux does not currently cover --
message queues, semaphores, etc -- *should* be in the LTP, regardless
of whether Linux currently implements them. Linux's 'expected results'
on these tests would be 'fail' at the moment. That's not a problem.
The LTP would also greatly benefit from better tracability and coverage.
I urge you to consider ways in which you could work within the
framework of the LTP to meet both your goals and the LTP's goals.
They may be more in synch than you originally thought!
- Dan
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:51 ` Jim Freeman
@ 2002-11-05 0:14 ` Geoff Gustafson
0 siblings, 0 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-05 0:14 UTC (permalink / raw
To: Jim Freeman, linux-kernel
Jim Freeman wrote:
> Howzabout the NIST one ?
>
> http://www.itl.nist.gov/div897/ctg/posix_form.htm
I think this is based on the 1990 POSIX spec. The things the new suite is
starting with have been added since then, up to and including the 2001 spec.
It is public domain I believe, so I would like to merge that in at some point,
but I would want to review the test cases individually to make sure they still
apply to the latest spec, etc. A lot of the work is just checking things in the
latest specification and adding/updating references in the code.
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:04 ` Geoff Gustafson
2002-11-05 0:08 ` Dan Kegel
@ 2002-11-05 0:24 ` Dan Kegel
2002-11-05 3:18 ` Christopher Yeoh
1 sibling, 1 reply; 24+ messages in thread
From: Dan Kegel @ 2002-11-05 0:24 UTC (permalink / raw
To: Geoff Gustafson; +Cc: Linux Kernel Mailing List, sglass, cyeoh
Geoff Gustafson wrote:
>Dan wrote:
>>You are about to duplicate http://ltp.sf.net
>Chris wrote:
>[You are about to duplicate http://www.linuxbase.org/test ]
Anyone know what the relationship between the LTP and LSB Posix compliance tests are?
Were they both originally derived from the OpenGroup Posix compliance tests?
- Dan
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-04 23:57 ` [ANNOUNCE] Open POSIX Test Suite Christopher Yeoh
@ 2002-11-05 0:44 ` Geoff Gustafson
2002-11-05 2:26 ` Andreas Dilger
2002-11-05 3:35 ` Christopher Yeoh
0 siblings, 2 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-05 0:44 UTC (permalink / raw
To: Christopher Yeoh, linux-kernel
Christopher Yeoh writes:
> Have you looked at the LSB test suites yet? They already cover much of
> what you plan on writing tests for, though we would welcome any
> volunteers who would like to increase the coverage. Most of the tests
> suites are released under the Artistic License, with quite a bit of
> the code donated by the Open Group (originally from the Unix
> certification tests).
One issue is that this new project is primarily concerned with testing parts of
the spec that have not been fully supported in Linux so far. These are the kind
of things that are not included in LSB yet, so they wouldn't be appropriate in
LSB's test suite.
Another problem is the overhead of the TET framework. One of the goals of this
the new test suite is to have test cases which are utterly minimal. So far, each
test case has its own main() function and a bare minimum of surrounding code.
The idea is that when a bug is found, this one .c file can be sent to the
appropriate developer, and without any learning curve, they have the ability to
find their bug. I don't think LKML wants to see TET code posted here. :)
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
2002-11-04 23:31 ` Andreas Dilger
2002-11-04 23:37 ` Rik van Riel
@ 2002-11-05 1:14 ` Geoff Gustafson
2 siblings, 0 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-05 1:14 UTC (permalink / raw
To: Jeff Garzik, linux-kernel
> I wonder if any vendors, or independent groups, would be interested in
> maintaining a POSIX compliancy patchkit for the Linux kernel?
I agree this sounds very useful. I could do something like this as part of the
test suite project; this would expand the scope to include testing and reporting
the status of the latest patches.
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: RFC: A POSIX Linux project?
2002-11-04 23:31 ` Andreas Dilger
2002-11-04 23:51 ` Jim Freeman
@ 2002-11-05 2:01 ` Geoff Gustafson
1 sibling, 0 replies; 24+ messages in thread
From: Geoff Gustafson @ 2002-11-05 2:01 UTC (permalink / raw
To: Andreas Dilger, linux-kernel
Andreas Dilger wrote:
> What about the existing POSIX test suite from X/Open? I don't know what
> the current license is, but it is certainly freely downloadable from
> their website. However, it is a pain in the a** to set up and run, so
> a new version would definitely be welcome.
If I understand correctly, the test suites that cover POSIX extensions more
recent than 1990 are not free. Also, these use the TET framework, whereas this
project hopes to keep tests very simple and standalone, so the code can be
immediately sent to and warmly received by developers if bugs are found.
http://www.opengroup.org/testing/sales+support/prices.htm#VSTRC
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:44 ` Geoff Gustafson
@ 2002-11-05 2:26 ` Andreas Dilger
2002-11-05 3:35 ` Christopher Yeoh
1 sibling, 0 replies; 24+ messages in thread
From: Andreas Dilger @ 2002-11-05 2:26 UTC (permalink / raw
To: Geoff Gustafson; +Cc: Christopher Yeoh, linux-kernel
On Nov 04, 2002 16:44 -0800, Geoff Gustafson wrote:
> Another problem is the overhead of the TET framework. One of the goals of
> this the new test suite is to have test cases which are utterly minimal.
> So far, each test case has its own main() function and a bare minimum of
> surrounding code. The idea is that when a bug is found, this one .c file
> can be sent to the appropriate developer, and without any learning curve,
> they have the ability to find their bug. I don't think LKML wants to see
> TET code posted here. :)
Having suffered through using the TET framework for the Open Group POSIX
test suite, I would agree that using TET sucks. The code is so convoluted
as to be useless, and it is nearly impossible to see from the output what
it is actually doing.
I agree that having a simple C or shell or perl script which is the entire
test, and the rest of the framework is external to it is very desirable.
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:24 ` Dan Kegel
@ 2002-11-05 3:18 ` Christopher Yeoh
0 siblings, 0 replies; 24+ messages in thread
From: Christopher Yeoh @ 2002-11-05 3:18 UTC (permalink / raw
To: Dan Kegel; +Cc: Geoff Gustafson, Linux Kernel Mailing List, sglass, cyeoh
At 2002/11/4 16:24-0800 Dan Kegel writes:
>
> Anyone know what the relationship between the LTP and LSB Posix
> compliance tests are?
The emphasis on what the test suites are trying to achieve does differ
- eg the LSB doesn't have endurance type tests and concentrates more
on writing tests which can be used for certification purposes. And of
course the LSB test suites only cover the areas that the LSB spec
requires.
> Were they both originally derived from the OpenGroup Posix compliance tests?
I don't think the LTP were tests derived from the OpenGroup (though
I'm not that familiar with them).
Chris
--
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:44 ` Geoff Gustafson
2002-11-05 2:26 ` Andreas Dilger
@ 2002-11-05 3:35 ` Christopher Yeoh
1 sibling, 0 replies; 24+ messages in thread
From: Christopher Yeoh @ 2002-11-05 3:35 UTC (permalink / raw
To: Geoff Gustafson; +Cc: linux-kernel
At 2002/11/4 16:44-0800 Geoff Gustafson writes:
> One issue is that this new project is primarily concerned with
> testing parts of the spec that have not been fully supported in
> Linux so far. These are the kind of things that are not included in
> LSB yet, so they wouldn't be appropriate in LSB's test suite.
Actually we do include some areas that aren't yet in the LSB spec or
fully supported by Linux (eg aio) in the LSB test suites we release -
they just aren't run by default - but it is easy to enable them.
> utterly minimal. So far, each test case has its own main() function
> and a bare minimum of surrounding code. The idea is that when a bug
> is found, this one .c file can be sent to the appropriate developer,
> and without any learning curve, they have the ability to find their
> bug. I don't think LKML wants to see TET code posted here. :)
Yes, I agree TET does have a significant learning curve, and I do end
up writing small test programs that don't include the TET stuff before
sending off bug reports.
I have however seen some advantages - It is nice when you get a test
failure the report tells you exactly which part of the specification
you're violating. Once you do understand the TET/vsxgen library calls
testcases look much simpler - and if you're aiming for complete
functionality coverage including all the tricky corner cases for
various interfaces which can require quite a bit of setup code to get
into the right situation I think you'll end up having to write helper
libraries anyway.
Chris
--
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 0:08 ` Dan Kegel
@ 2002-11-05 15:44 ` Nathan Straz
0 siblings, 0 replies; 24+ messages in thread
From: Nathan Straz @ 2002-11-05 15:44 UTC (permalink / raw
To: Linux Kernel Mailing List
On Mon, Nov 04, 2002 at 04:04:32PM -0800, Geoff Gustafson wrote:
> > You are about to duplicate http://ltp.sf.net
>
> My understanding is that LTP is focused on current mainline kernel testing,
[ .. snip .. ]
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.).
You are correct. LTP isn't trying to be any kind of "compliance" test.
We're just trying to exercise and stress the Linux kernel to find bugs
and guard against regressions.
On Mon, Nov 04, 2002 at 04:08:44PM -0800, Dan Kegel wrote:
> I urge you to consider ways in which you could work within the
> framework of the LTP to meet both your goals and the LTP's goals.
> They may be more in synch than you originally thought!
I don't think LTP's "framework[1]" is going to help Geoff complete his
project. Geoff already created a SourceForge site. As part of the
LTP, I probably would have given him a separate CVS module and a mailing
list. I think he has everything he needs to get started already.
That being said, if Geoff does want his Open POSIX test suite to be part
of the LTP, we'll gladly accomodate him.
[1] By framework I think you mean logistical, not libraries. I think
our test libraries have some things he might like. I just recommend
cleaning them up before using them.
--
Nate Straz nstraz@sgi.com
sgi, inc http://www.sgi.com/
Linux Test Project http://ltp.sf.net/
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
@ 2002-11-05 15:49 Stephanie Glass
2002-11-05 16:43 ` Rusty Lynch
0 siblings, 1 reply; 24+ messages in thread
From: Stephanie Glass @ 2002-11-05 15:49 UTC (permalink / raw
To: Geoff Gustafson; +Cc: Dan Kegel, Linux Kernel Mailing List
Geoff,
The LTP would be happy to have anyone in the Linux community donate test
cases. This includes any POSIX tests.
The LTP would not be advertised as a POSIX compliance test, that would be
up to LSB to handle. These tests
would only increase the overall LTP api coverages.
Does your group own these tests? Do you want to donate them to the LTP?
Stephanie
Linux Technology Center
IBM, 11400 Burnet Road, Austin, TX 78758
Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
E-Mail: sglass@us.ibm.com
"Geoff Gustafson"
<geoff@linux.co.i To: "Dan Kegel" <dkegel@ixiacom.com>, "Linux Kernel Mailing List"
ntel.com> <linux-kernel@vger.kernel.org>
cc: Stephanie Glass/Austin/IBM@IBMUS
11/04/2002 06:04 Subject: Re: [ANNOUNCE] Open POSIX Test Suite
PM
> You are about to duplicate http://ltp.sf.net
My understanding is that LTP is focused on current mainline kernel testing,
while this project's initial concern is areas that are not currently in
Linux
like POSIX message queues, semaphores, and full support for POSIX threads.
I see
this as being used to evaluate different implementations that are being
considered for inclusion in the kernel, glibc, etc.
This project is concerned with the POSIX APIs regardless of where they are
implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent
of
implementation. This project will be more concerned with traceability back
to
the POSIX specification, and completeness of coverage, than I would expect
from
LTP.
That said, there is some overlap, and an exchange of test cases between the
projects may be very useful.
I've copied Stephanie from LTP to get her reaction.
-- Geoff Gustafson
These are my views and not necessarily those of my employer.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 15:49 Stephanie Glass
@ 2002-11-05 16:43 ` Rusty Lynch
0 siblings, 0 replies; 24+ messages in thread
From: Rusty Lynch @ 2002-11-05 16:43 UTC (permalink / raw
To: Geoff Gustafson, Stephanie Glass; +Cc: Dan Kegel, Linux Kernel Mailing List
Stephanie,
All test are GPL, so anyone can do anything they want with them. We would
be happy to donate test to any project.
The truth is that we modeled test cases after LTP, meaning that a test case
is
a simple executable that returns 0 for success and anything else to indicate
failure, so copying a test from posixtest to LTP should be very easy.
I was under the impression that LTP did not want to accept a bunch of test
cases that did not currently have an associated implementation in Linux.
It sounds like this is not exactly correct. How about test cases that will
probably
always be implemented in user space? Isn't LTP specific to kernel testing?
-rusty
----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Geoff Gustafson" <geoff@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 7:49 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite
>
> Geoff,
> The LTP would be happy to have anyone in the Linux community donate test
> cases. This includes any POSIX tests.
> The LTP would not be advertised as a POSIX compliance test, that would be
> up to LSB to handle. These tests
> would only increase the overall LTP api coverages.
>
> Does your group own these tests? Do you want to donate them to the LTP?
>
> Stephanie
>
> Linux Technology Center
> IBM, 11400 Burnet Road, Austin, TX 78758
> Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
> E-Mail: sglass@us.ibm.com
>
>
>
> "Geoff Gustafson"
> <geoff@linux.co.i To: "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
> ntel.com> <linux-kernel@vger.kernel.
org>
> cc: Stephanie
Glass/Austin/IBM@IBMUS
> 11/04/2002 06:04 Subject: Re: [ANNOUNCE]
Open POSIX Test Suite
> PM
>
>
>
>
>
> > You are about to duplicate http://ltp.sf.net
>
> My understanding is that LTP is focused on current mainline kernel
testing,
> while this project's initial concern is areas that are not currently in
> Linux
> like POSIX message queues, semaphores, and full support for POSIX threads.
> I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
>
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent
> of
> implementation. This project will be more concerned with traceability back
> to
> the POSIX specification, and completeness of coverage, than I would expect
> from
> LTP.
>
> That said, there is some overlap, and an exchange of test cases between
the
> projects may be very useful.
>
> I've copied Stephanie from LTP to get her reaction.
>
> -- Geoff Gustafson
>
> These are my views and not necessarily those of my employer.
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
@ 2002-11-05 18:24 Stephanie Glass
2002-11-05 19:05 ` Rusty Lynch
0 siblings, 1 reply; 24+ messages in thread
From: Stephanie Glass @ 2002-11-05 18:24 UTC (permalink / raw
To: Rusty Lynch; +Cc: Dan Kegel, Geoff Gustafson, Linux Kernel Mailing List
Rusty,
We will take them. We may set them up not to run with our runall portion
but as a separate run. This is how we do several areas, such as
networking. Just let us know when you are ready to start contributing.
It doesn't have to be all at once, we will take in stages also. We put out
a new version almost every month so we can get them out there quickly.
Don't most of these test cases deal with things like POSIX timers,
semaphores, threads, etc.? Or are these other types of tests?
Thanks
Stephanie
Linux Technology Center
IBM, 11400 Burnet Road, Austin, TX 78758
Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
E-Mail: sglass@us.ibm.com
"Rusty Lynch"
<rusty@linux.co.i To: "Geoff Gustafson" <geoff@linux.co.intel.com>, Stephanie
ntel.com> Glass/Austin/IBM@IBMUS
cc: "Dan Kegel" <dkegel@ixiacom.com>, "Linux Kernel Mailing List"
11/05/2002 10:43 <linux-kernel@vger.kernel.org>
AM Subject: Re: [ANNOUNCE] Open POSIX Test Suite
Stephanie,
All test are GPL, so anyone can do anything they want with them. We would
be happy to donate test to any project.
The truth is that we modeled test cases after LTP, meaning that a test case
is
a simple executable that returns 0 for success and anything else to
indicate
failure, so copying a test from posixtest to LTP should be very easy.
I was under the impression that LTP did not want to accept a bunch of test
cases that did not currently have an associated implementation in Linux.
It sounds like this is not exactly correct. How about test cases that will
probably
always be implemented in user space? Isn't LTP specific to kernel testing?
-rusty
----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Geoff Gustafson" <geoff@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 7:49 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite
>
> Geoff,
> The LTP would be happy to have anyone in the Linux community donate test
> cases. This includes any POSIX tests.
> The LTP would not be advertised as a POSIX compliance test, that would be
> up to LSB to handle. These tests
> would only increase the overall LTP api coverages.
>
> Does your group own these tests? Do you want to donate them to the LTP?
>
> Stephanie
>
> Linux Technology Center
> IBM, 11400 Burnet Road, Austin, TX 78758
> Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
> E-Mail: sglass@us.ibm.com
>
>
>
> "Geoff Gustafson"
> <geoff@linux.co.i To: "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
> ntel.com>
<linux-kernel@vger.kernel.
org>
> cc: Stephanie
Glass/Austin/IBM@IBMUS
> 11/04/2002 06:04 Subject: Re: [ANNOUNCE]
Open POSIX Test Suite
> PM
>
>
>
>
>
> > You are about to duplicate http://ltp.sf.net
>
> My understanding is that LTP is focused on current mainline kernel
testing,
> while this project's initial concern is areas that are not currently in
> Linux
> like POSIX message queues, semaphores, and full support for POSIX
threads.
> I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
>
> This project is concerned with the POSIX APIs regardless of where they
are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX,
independent
> of
> implementation. This project will be more concerned with traceability
back
> to
> the POSIX specification, and completeness of coverage, than I would
expect
> from
> LTP.
>
> That said, there is some overlap, and an exchange of test cases between
the
> projects may be very useful.
>
> I've copied Stephanie from LTP to get her reaction.
>
> -- Geoff Gustafson
>
> These are my views and not necessarily those of my employer.
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [ANNOUNCE] Open POSIX Test Suite
2002-11-05 18:24 Stephanie Glass
@ 2002-11-05 19:05 ` Rusty Lynch
0 siblings, 0 replies; 24+ messages in thread
From: Rusty Lynch @ 2002-11-05 19:05 UTC (permalink / raw
To: Rusty Lynch, Stephanie Glass
Cc: Dan Kegel, Geoff Gustafson, Linux Kernel Mailing List
We started with a few areas like message queues and signals, but it is just
a starting point.
-rusty (hmm... guess rusty is already taken in this domain. maybe I
should sign rustyl :->)
----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Rusty Lynch" <rusty@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Geoff Gustafson"
<geoff@linux.co.intel.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 10:24 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite
>
> Rusty,
> We will take them. We may set them up not to run with our runall portion
> but as a separate run. This is how we do several areas, such as
> networking. Just let us know when you are ready to start contributing.
> It doesn't have to be all at once, we will take in stages also. We put
out
> a new version almost every month so we can get them out there quickly.
>
> Don't most of these test cases deal with things like POSIX timers,
> semaphores, threads, etc.? Or are these other types of tests?
>
> Thanks
>
> Stephanie
>
> Linux Technology Center
> IBM, 11400 Burnet Road, Austin, TX 78758
> Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
> E-Mail: sglass@us.ibm.com
>
>
>
> "Rusty Lynch"
> <rusty@linux.co.i To: "Geoff Gustafson"
<geoff@linux.co.intel.com>, Stephanie
> ntel.com> Glass/Austin/IBM@IBMUS
> cc: "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
> 11/05/2002 10:43
<linux-kernel@vger.kernel.org>
> AM Subject: Re: [ANNOUNCE]
Open POSIX Test Suite
>
>
>
>
>
>
> Stephanie,
>
> All test are GPL, so anyone can do anything they want with them. We would
> be happy to donate test to any project.
>
> The truth is that we modeled test cases after LTP, meaning that a test
case
> is
> a simple executable that returns 0 for success and anything else to
> indicate
> failure, so copying a test from posixtest to LTP should be very easy.
>
> I was under the impression that LTP did not want to accept a bunch of test
> cases that did not currently have an associated implementation in Linux.
> It sounds like this is not exactly correct. How about test cases that
will
> probably
> always be implemented in user space? Isn't LTP specific to kernel
testing?
>
> -rusty
>
> ----- Original Message -----
> From: "Stephanie Glass" <sglass@us.ibm.com>
> To: "Geoff Gustafson" <geoff@linux.co.intel.com>
> Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
> <linux-kernel@vger.kernel.org>
> Sent: Tuesday, November 05, 2002 7:49 AM
> Subject: Re: [ANNOUNCE] Open POSIX Test Suite
>
>
> >
> > Geoff,
> > The LTP would be happy to have anyone in the Linux community donate test
> > cases. This includes any POSIX tests.
> > The LTP would not be advertised as a POSIX compliance test, that would
be
> > up to LSB to handle. These tests
> > would only increase the overall LTP api coverages.
> >
> > Does your group own these tests? Do you want to donate them to the LTP?
> >
> > Stephanie
> >
> > Linux Technology Center
> > IBM, 11400 Burnet Road, Austin, TX 78758
> > Phone: (512) 838-9284 T/L: 678-9284 Fax: (512) 838-3882
> > E-Mail: sglass@us.ibm.com
> >
> >
> >
> > "Geoff Gustafson"
> > <geoff@linux.co.i To: "Dan Kegel"
> <dkegel@ixiacom.com>, "Linux Kernel Mailing List"
> > ntel.com>
> <linux-kernel@vger.kernel.
> org>
> > cc: Stephanie
> Glass/Austin/IBM@IBMUS
> > 11/04/2002 06:04 Subject: Re: [ANNOUNCE]
> Open POSIX Test Suite
> > PM
> >
> >
> >
> >
> >
> > > You are about to duplicate http://ltp.sf.net
> >
> > My understanding is that LTP is focused on current mainline kernel
> testing,
> > while this project's initial concern is areas that are not currently in
> > Linux
> > like POSIX message queues, semaphores, and full support for POSIX
> threads.
> > I see
> > this as being used to evaluate different implementations that are being
> > considered for inclusion in the kernel, glibc, etc.
> >
> > This project is concerned with the POSIX APIs regardless of where they
> are
> > implemented (kernel, glibc, etc.). Thus it can focus on POSIX,
> independent
> > of
> > implementation. This project will be more concerned with traceability
> back
> > to
> > the POSIX specification, and completeness of coverage, than I would
> expect
> > from
> > LTP.
> >
> > That said, there is some overlap, and an exchange of test cases between
> the
> > projects may be very useful.
> >
> > I've copied Stephanie from LTP to get her reaction.
> >
> > -- Geoff Gustafson
> >
> > These are my views and not necessarily those of my employer.
> >
> >
> >
> >
> >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2002-11-05 18:58 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-11-04 22:48 [ANNOUNCE] Open POSIX Test Suite Geoff Gustafson
2002-11-04 22:58 ` Larry McVoy
2002-11-04 23:17 ` Geoff Gustafson
2002-11-04 23:14 ` RFC: A POSIX Linux project? Jeff Garzik
2002-11-04 23:31 ` Andreas Dilger
2002-11-04 23:51 ` Jim Freeman
2002-11-05 0:14 ` Geoff Gustafson
2002-11-05 2:01 ` Geoff Gustafson
2002-11-04 23:37 ` Rik van Riel
2002-11-05 1:14 ` Geoff Gustafson
2002-11-04 23:57 ` [ANNOUNCE] Open POSIX Test Suite Christopher Yeoh
2002-11-05 0:44 ` Geoff Gustafson
2002-11-05 2:26 ` Andreas Dilger
2002-11-05 3:35 ` Christopher Yeoh
-- strict thread matches above, loose matches on Subject: below --
2002-11-04 23:29 Dan Kegel
2002-11-05 0:04 ` Geoff Gustafson
2002-11-05 0:08 ` Dan Kegel
2002-11-05 15:44 ` Nathan Straz
2002-11-05 0:24 ` Dan Kegel
2002-11-05 3:18 ` Christopher Yeoh
2002-11-05 15:49 Stephanie Glass
2002-11-05 16:43 ` Rusty Lynch
2002-11-05 18:24 Stephanie Glass
2002-11-05 19:05 ` Rusty Lynch
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).