[ntp-GSoC] Testing update

Harlan Stenn stenn at nwtime.org
Mon Jun 1 18:47:32 UTC 2015


Please see this email from Mark Vander Voord, below.

If folks here *want* to use the fixtures code with these capabilities
that will take some extra work.

Mark, I think the primary reason this happened was that we were looking
for ways to have finer-grained control over which test case(s) were
executed when running a block of tests.

There's a concern that as the number of tests grow the time it takes to
run them will also grow, and making sure a test can be quickly run
during its development was seen as a good thing.

If the answer is "Write your tests stand-alone and when they work
#include them in the bigger runner/driver" that may be a fine answer.

It will be trivial for us to not include the fixtures code.

Another thing we're looking at is adding XFAIL/XPASS, because I'd like
to see us also have tests for the issues reported in our bugzilla
system.  Unresolved issues  there would be "expected failures" and would
not stop the testing process until the PACKAGE_VERSION (our software's
version) was at a point where the issue was resolved.

We're thinking that XPASS (unexpected pass) would be reported but would
not stop the test run, either.

So, to quote an earlier message about this, the test code would contain
lines like:

- for code earlier than 4.2: FAIL
- for code in the 4.2 branch: XFAIL@<4.2.8p2,PASS
- for code in the 4.3 branch,:XFAIL@<4.3.20,PASS
- else: PASS

That's off the top of my head. It means that if we're running the test
on something earlier than 4.2 (EOL'd code) we expect the test to fail,
but this is not an XFAIL case.  If we're in the 4.2 branch, code before
4.2.8p2 is expected to fail, and code from >= 4.2.8p2 is expected to
pass.  I suspect the other cases are now obvious...

Reading this, I can see we might want to make even more changes, or at
least be sure that people understood that an XFAIL is a known (possibly
very serious) bug where we know the problem won't be fixed by changing
that specific branch - if folks want that to be fixed the should update
to the newer branch of the code.

But I'm not a testing guy and this is one of the cases where your input
would be most appreciated.


On 6/1/15 10:49 AM, Mark Vander Voord wrote:
> Harlan:
> If you have a compelling reason that you need fixtures, then feel free to
> proceed as your message outlines. Otherwise, my humble opinion is that you
> should use Unity directly without the extra layer.
> The fixtures layer was submitted by James Grenning when he released his
> book, primarily because he had received feedback on an early release of his
> book which said that people didn't like using a C++ framework to test C. He
> wanted Unity to look like CppUTest so that he wouldn't have to rewrite all
> the examples in his book.
> The problem is that he hasn't done any maintenance on that submission since
> it was made. None of the main developers of Unity use the extra layer, so
> it tends to lag featurewise and is buggier than the rest of Unity (it
> doesn't get fixes until we hear a number of complaints about something).
> Just my two cents. :)
> Mark
> On Mon, Jun 1, 2015 at 1:34 PM Harlan Stenn <stenn at nwtime.org> wrote:
>> Testing folks,
>> Tomasz pointed me at the unity_framework.h stuff:
>>  This Framework is an optional add-on to Unity.  By including
>>  unity_framework.h in place of unity.h, you may now work with
>>  Unity in a manner similar to CppUTest.
>>  This framework adds the concepts of test groups and gives
>>  finer control of your tests over the command line.
>> I added the code to the libunity.a that we build, so if you #include
>> <unity_framework.h> instead of <unity.h> you should get these extra
>> features.
>> From your unity repos, please try:
>>  bk changes -Rvv ~stenn/ntp-stable-unity
>> to see what changes are in that repo that you don't have, and:
>>  bk changes -Lvv ~stenn/ntp-stable-unity
>> to see what changes are in your repos that are not in the
>> ~stenn/ntp-stable-unity integration area.  If you have changes that are
>> ready to be put in the integration area, please let me know.
>> --
>> Harlan Stenn <stenn at nwtime.org>
>> http://networktimefoundation.org - be a member!
>> _______________________________________________
>> GSoC mailing list
>> GSoC at lists.ntp.org
>> http://lists.ntp.org/listinfo/gsoc

Harlan Stenn <stenn at nwtime.org>
http://networktimefoundation.org - be a member!

More information about the GSoC mailing list