[ntp-GSoC] Testing update
Mark Vander Voord
mvandervoord at gmail.com
Tue Jun 2 12:21:17 UTC 2015
I'm clearly not here to tell you how you should be doing your jobs. I CAN
however tell you how Unity was designed to work, and maybe that will affect
the way you choose to use it. It appears that you are going in a few
non-standard directions which are going to be more work for you, and I
don't believe they gain you anything.
First, Unity is a unit testing framework, therefore designed to test a unit
at a time. I've not seen the makefile setup you have at the moment, so I
don't know how close to the standard you are, but I can explain how Unity
is typically used: Typically you have one or more Test files for each C
module. This test file gets built along with the C module, Unity, and
possibly a few other supporting files (depending on if you are using Mocks
or other features like that). You therefore have a test executable built
for EACH test file, meaning there is at least one test executable for each
C module (sometimes more).
The build system's job is then to do all the dependency tracking so that
only the executables that NEED to be rebuilt are rebuilt and run. This
dependency tracking is similar to a normal release build, obviously, but
you're tracking many more files. Often people decide to make their master
"test" target a summary file, which gets "built" from results files from
each executable, which get "built" from running the executables themselves.
I think you can see how the rest of it fans out from there.
I explain this because there seems to be some concern that the tests are
going to take too long to build. When run a FULL test suite, this will of
course take awhile once you have a lot of tests. But when you are working
and making changes to a c file or two, you will only be triggering the
rebuild and retest of a handful of test files most of the time (the
exception being when you change a global header file or something like
that). In almost all circumstances, building and linking takes longer than
executing the tests afterwards. (This is the main reason fixtures don't
actually add much to a compiled language. You STILL have to recompile the
entire executable even if you just want to retest a single function. Since
the compilation is the long step, this mean you haven't saved much time.
Fixtures encourage people to group more unlike tests into the same test
files as well, so this often results in fixtures taking LONGER to run than
a standard Unity test).
In the end, I usually set up my make (or rake) files to give me three
- build/run a single test file by name
- build/run all tests that have changes (or that rely on source that have
- build/run ALL tests (basically doing a clean or clobber ahead of time,
and then asking for all tests)
I use the first while I am working on a particular feature. I use the
second before I commit anything to a central repo. I run the last on a
continuous integration server so that we are getting a full test of ALL
changes on the test server.
EXPECTED FAILING TESTS
Adding tests for things that you run into along the way is a good idea.
It's good to have a test around to document how a feature SHOULD work, even
well before you implement the feature. I like that you are planning to do
There is a very simple mechanism for handling this situation already. You
write the test and then place a TEST_IGNORE_MESSAGE("reason it is ignored")
on the first line. These ignores become part of the overall report when you
are done. They won't cause your final tests to fail (as they shouldn't) and
they DO remind you that you need to work on these things by providing a
list in the report.
Your versioning sounds unnecessarily complex. Your tests should just be in
a subdirectory of your overall project anyway, so they are already along
for the "versioning" ride. When code on a particular branch now makes a
test valid, you remove the IGNORE line. There is no reason to track
additional revisioning if you are already using version control (which I
assume you are, since you talked about branches).
I don't personally see any advantages to the XPASS/XFAIL idea. It just
seems like adding complexity to the system to accomplish something that can
already be done using your version control naturally.
Those are my opinions on these topics. I hope it didn't sound like a rant
or something. In the end, if you decide you disagree with any of these
things, I'm happy to to help you get there. ;)
On Mon, Jun 1, 2015 at 2:47 PM Harlan Stenn <stenn at nwtime.org> wrote:
> Please see this email from Mark Vander Voord, below.
> If folks here *want* to use the fixtures code with these capabilities
> that will take some extra work.
> Mark, I think the primary reason this happened was that we were looking
> for ways to have finer-grained control over which test case(s) were
> executed when running a block of tests.
> There's a concern that as the number of tests grow the time it takes to
> run them will also grow, and making sure a test can be quickly run
> during its development was seen as a good thing.
> If the answer is "Write your tests stand-alone and when they work
> #include them in the bigger runner/driver" that may be a fine answer.
> It will be trivial for us to not include the fixtures code.
> Another thing we're looking at is adding XFAIL/XPASS, because I'd like
> to see us also have tests for the issues reported in our bugzilla
> system. Unresolved issues there would be "expected failures" and would
> not stop the testing process until the PACKAGE_VERSION (our software's
> version) was at a point where the issue was resolved.
> We're thinking that XPASS (unexpected pass) would be reported but would
> not stop the test run, either.
> So, to quote an earlier message about this, the test code would contain
> lines like:
> - for code earlier than 4.2: FAIL
> - for code in the 4.2 branch: XFAIL@<4.2.8p2,PASS
> - for code in the 4.3 branch,:XFAIL@<4.3.20,PASS
> - else: PASS
> That's off the top of my head. It means that if we're running the test
> on something earlier than 4.2 (EOL'd code) we expect the test to fail,
> but this is not an XFAIL case. If we're in the 4.2 branch, code before
> 4.2.8p2 is expected to fail, and code from >= 4.2.8p2 is expected to
> pass. I suspect the other cases are now obvious...
> Reading this, I can see we might want to make even more changes, or at
> least be sure that people understood that an XFAIL is a known (possibly
> very serious) bug where we know the problem won't be fixed by changing
> that specific branch - if folks want that to be fixed the should update
> to the newer branch of the code.
> But I'm not a testing guy and this is one of the cases where your input
> would be most appreciated.
> On 6/1/15 10:49 AM, Mark Vander Voord wrote:
> > Harlan:
> > If you have a compelling reason that you need fixtures, then feel free to
> > proceed as your message outlines. Otherwise, my humble opinion is that
> > should use Unity directly without the extra layer.
> > The fixtures layer was submitted by James Grenning when he released his
> > book, primarily because he had received feedback on an early release of
> > book which said that people didn't like using a C++ framework to test C.
> > wanted Unity to look like CppUTest so that he wouldn't have to rewrite
> > the examples in his book.
> > The problem is that he hasn't done any maintenance on that submission
> > it was made. None of the main developers of Unity use the extra layer, so
> > it tends to lag featurewise and is buggier than the rest of Unity (it
> > doesn't get fixes until we hear a number of complaints about something).
> > Just my two cents. :)
> > Mark
> > On Mon, Jun 1, 2015 at 1:34 PM Harlan Stenn <stenn at nwtime.org> wrote:
> >> Testing folks,
> >> Tomasz pointed me at the unity_framework.h stuff:
> >> This Framework is an optional add-on to Unity. By including
> >> unity_framework.h in place of unity.h, you may now work with
> >> Unity in a manner similar to CppUTest.
> >> This framework adds the concepts of test groups and gives
> >> finer control of your tests over the command line.
> >> I added the code to the libunity.a that we build, so if you #include
> >> <unity_framework.h> instead of <unity.h> you should get these extra
> >> features.
> >> From your unity repos, please try:
> >> bk changes -Rvv ~stenn/ntp-stable-unity
> >> to see what changes are in that repo that you don't have, and:
> >> bk changes -Lvv ~stenn/ntp-stable-unity
> >> to see what changes are in your repos that are not in the
> >> ~stenn/ntp-stable-unity integration area. If you have changes that are
> >> ready to be put in the integration area, please let me know.
> >> --
> >> Harlan Stenn <stenn at nwtime.org>
> >> http://networktimefoundation.org - be a member!
> >> _______________________________________________
> >> GSoC mailing list
> >> GSoC at lists.ntp.org
> >> http://lists.ntp.org/listinfo/gsoc
> Harlan Stenn <stenn at nwtime.org>
> http://networktimefoundation.org - be a member!
More information about the GSoC