[ntp-GSoC] Testing update

Harlan Stenn stenn at ntp.org
Wed Jun 3 03:36:58 UTC 2015

Mark Vander Voord writes:
> Hi everyone,
> I'm clearly not here to tell you how you should be doing your jobs. I
> CAN however tell you how Unity was designed to work, and maybe that
> will affect the way you choose to use it. It appears that you are
> going in a few non-standard directions which are going to be more work
> for you, and I don't believe they gain you anything.
> First, Unity is a unit testing framework, therefore designed to test a
> unit at a time. I've not seen the makefile setup you have at the
> moment, so I don't know how close to the standard you are, but I can
> explain how Unity is typically used: Typically you have one or more
> Test files for each C module. This test file gets built along with the
> C module, Unity, and possibly a few other supporting files (depending
> on if you are using Mocks or other features like that). You therefore
> have a test executable built for EACH test file, meaning there is at
> least one test executable for each C module (sometimes more).

We use GNU automake.  As I look more closely at how things pan out with
Unity we'll be looking to see if we feel lucky and want to use wildcard
expansion, or if we want an explicit list of test modules and drive from

> The build system's job is then to do all the dependency tracking so
> that only the executables that NEED to be rebuilt are rebuilt and
> run. This dependency tracking is similar to a normal release build,
> obviously, but you're tracking many more files. Often people decide to
> make their master "test" target a summary file, which gets "built"
> from results files from each executable, which get "built" from
> running the executables themselves.  I think you can see how the rest
> of it fans out from there.

Yes, we have this already.

> I explain this because there seems to be some concern that the tests
> are going to take too long to build. When run a FULL test suite, this
> will of course take awhile once you have a lot of tests. But when you
> are working and making changes to a c file or two, you will only be
> triggering the rebuild and retest of a handful of test files most of
> the time (the exception being when you change a global header file or
> something like that). In almost all circumstances, building and
> linking takes longer than executing the tests afterwards. (This is the
> main reason fixtures don't actually add much to a compiled
> language. You STILL have to recompile the entire executable even if
> you just want to retest a single function. Since the compilation is
> the long step, this mean you haven't saved much time.  Fixtures
> encourage people to group more unlike tests into the same test files
> as well, so this often results in fixtures taking LONGER to run than a
> standard Unity test).

I'm not worried about the build/link time, I'm worried about the time it
takes to run a huge number of tests.

I'm still not sure how many tests we'll want to call from each
"runner".  If it's a small number, that's not a big deal, and in that
case I'm assuming there will eventually be a huge number of executables.

> In the end, I usually set up my make (or rake) files to give me three
> options:
>  - build/run a single test file by name
>  - build/run all tests that have changes (or that rely on source that have
> changed)
>  - build/run ALL tests (basically doing a clean or clobber ahead of time,
> and then asking for all tests)

In automake, there is a "make check" target that builds any out-of-date
tests and then runs *all* the tests for the given subtree.

If it was me, I'd just use that and interrupt when the tests start
running and run the one I want manually.  We could just as easily create
a new target that would only build the tests.

> I use the first while I am working on a particular feature. I use the
> second before I commit anything to a central repo. I run the last on a
> continuous integration server so that we are getting a full test of ALL
> changes on the test server.

That makes sense, thanks!

> Adding tests for things that you run into along the way is a good
> idea.  It's good to have a test around to document how a feature
> SHOULD work, even well before you implement the feature. I like that
> you are planning to do this.
> There is a very simple mechanism for handling this situation
> already. You write the test and then place a
> TEST_IGNORE_MESSAGE("reason it is ignored") on the first line. These
> ignores become part of the overall report when you are done. They
> won't cause your final tests to fail (as they shouldn't) and they DO
> remind you that you need to work on these things by providing a list
> in the report.

Happy to look at this, and I guess I'm concerned that it will take more
effort to do things this way.

> Your versioning sounds unnecessarily complex. Your tests should just
> be in a subdirectory of your overall project anyway, so they are
> already along for the "versioning" ride. When code on a particular
> branch now makes a test valid, you remove the IGNORE line. There is no
> reason to track additional revisioning if you are already using
> version control (which I assume you are, since you talked about
> branches).

Yes, and the problem I see is when we want to backport tests to older
versions of the code.

This may not be a big issue, but with the package versioning we should
rarely need to modify any appropriate tests, because they generally
won't be changing.

> I don't personally see any advantages to the XPASS/XFAIL idea. It just
> seems like adding complexity to the system to accomplish something
> that can already be done using your version control naturally.

OK, and I suspect we'll learn more as we go forward.

> Those are my opinions on these topics. I hope it didn't sound like a
> rant or something. In the end, if you decide you disagree with any of
> these things, I'm happy to to help you get there.  ;)

Very much appreciated, thanks!

> On Mon, Jun 1, 2015 at 2:47 PM Harlan Stenn <stenn at nwtime.org> wrote:
> > Folks,
> >
> > Please see this email from Mark Vander Voord, below.
> >
> > If folks here *want* to use the fixtures code with these capabilities
> > that will take some extra work.
> >
> > Mark, I think the primary reason this happened was that we were looking
> > for ways to have finer-grained control over which test case(s) were
> > executed when running a block of tests.
> >
> > There's a concern that as the number of tests grow the time it takes to
> > run them will also grow, and making sure a test can be quickly run
> > during its development was seen as a good thing.
> >
> > If the answer is "Write your tests stand-alone and when they work
> > #include them in the bigger runner/driver" that may be a fine answer.
> >
> > It will be trivial for us to not include the fixtures code.
> >
> > Another thing we're looking at is adding XFAIL/XPASS, because I'd like
> > to see us also have tests for the issues reported in our bugzilla
> > system.  Unresolved issues  there would be "expected failures" and would
> > not stop the testing process until the PACKAGE_VERSION (our software's
> > version) was at a point where the issue was resolved.
> >
> > We're thinking that XPASS (unexpected pass) would be reported but would
> > not stop the test run, either.
> >
> > So, to quote an earlier message about this, the test code would contain
> > lines like:
> >
> > - for code earlier than 4.2: FAIL
> > - for code in the 4.2 branch: XFAIL@<4.2.8p2,PASS
> > - for code in the 4.3 branch,:XFAIL@<4.3.20,PASS
> > - else: PASS
> >
> > That's off the top of my head. It means that if we're running the test
> > on something earlier than 4.2 (EOL'd code) we expect the test to fail,
> > but this is not an XFAIL case.  If we're in the 4.2 branch, code before
> > 4.2.8p2 is expected to fail, and code from >= 4.2.8p2 is expected to
> > pass.  I suspect the other cases are now obvious...
> >
> > Reading this, I can see we might want to make even more changes, or at
> > least be sure that people understood that an XFAIL is a known (possibly
> > very serious) bug where we know the problem won't be fixed by changing
> > that specific branch - if folks want that to be fixed the should update
> > to the newer branch of the code.
> >
> > But I'm not a testing guy and this is one of the cases where your input
> > would be most appreciated.
> >
> > H
> >
> > On 6/1/15 10:49 AM, Mark Vander Voord wrote:
> > > Harlan:
> > >
> > > If you have a compelling reason that you need fixtures, then feel free to
> > > proceed as your message outlines. Otherwise, my humble opinion is that
> > you
> > > should use Unity directly without the extra layer.
> > >
> > > The fixtures layer was submitted by James Grenning when he released his
> > > book, primarily because he had received feedback on an early release of
> > his
> > > book which said that people didn't like using a C++ framework to test C.
> > He
> > > wanted Unity to look like CppUTest so that he wouldn't have to rewrite
> > all
> > > the examples in his book.
> > >
> > > The problem is that he hasn't done any maintenance on that submission
> > since
> > > it was made. None of the main developers of Unity use the extra layer, so
> > > it tends to lag featurewise and is buggier than the rest of Unity (it
> > > doesn't get fixes until we hear a number of complaints about something).
> > >
> > > Just my two cents. :)
> > >
> > > Mark
> > >
> > > On Mon, Jun 1, 2015 at 1:34 PM Harlan Stenn <stenn at nwtime.org> wrote:
> > >
> > >> Testing folks,
> > >>
> > >> Tomasz pointed me at the unity_framework.h stuff:
> > >>
> > >>  This Framework is an optional add-on to Unity.  By including
> > >>  unity_framework.h in place of unity.h, you may now work with
> > >>  Unity in a manner similar to CppUTest.
> > >>  This framework adds the concepts of test groups and gives
> > >>  finer control of your tests over the command line.
> > >>
> > >> I added the code to the libunity.a that we build, so if you #include
> > >> <unity_framework.h> instead of <unity.h> you should get these extra
> > >> features.
> > >>
> > >> From your unity repos, please try:
> > >>
> > >>  bk changes -Rvv ~stenn/ntp-stable-unity
> > >>
> > >> to see what changes are in that repo that you don't have, and:
> > >>
> > >>  bk changes -Lvv ~stenn/ntp-stable-unity
> > >>
> > >> to see what changes are in your repos that are not in the
> > >> ~stenn/ntp-stable-unity integration area.  If you have changes that are
> > >> ready to be put in the integration area, please let me know.
> > >> --
> > >> Harlan Stenn <stenn at nwtime.org>
> > >> http://networktimefoundation.org - be a member!
> > >>
> > >> _______________________________________________
> > >> GSoC mailing list
> > >> GSoC at lists.ntp.org
> > >> http://lists.ntp.org/listinfo/gsoc
> > >>
> > >
> >
> > --
> > Harlan Stenn <stenn at nwtime.org>
> > http://networktimefoundation.org - be a member!
> >
> >
> _______________________________________________
> GSoC mailing list
> GSoC at lists.ntp.org
> http://lists.ntp.org/listinfo/gsoc

More information about the GSoC mailing list