[ntp-GSoC] Questions about testing

Mark Vander Voord mvandervoord at gmail.com
Fri Jun 12 13:51:59 UTC 2015


Sorry about the delay in responding. It's been a crazy week.

*1) There is no EXPECT function in Unity, only ASSERT. In Gtest you can
use EXPECT functions for asserts that aren't critical. If an EXPECT
function fails, the test prints a message and continues onto the next
ASSERT / EXPECT without failing the test.Is this a necessary feature? I
could implement EXPECT functions in Unity,if we require them. Unity
currently works like this:If you have 5 asserts in a test, and the 2nd one
fails, Unity prints amessage related to the 2nd assert, stops the test and
marks it as afailiure. In order to see if the 3rd assert returns true, you
would have tofix the 2nd assert first.Should we convert all gtest EXPECT
functions to ASSERT in Unity or? Wecould also split the gtest test to
multiple unity tests, that way we wouldget the same effect.*

You are correct that Unity doesn't support the EXPECT. This is because (a)
it conflicts with the meaning of the expect in CMock and (b) because we
disagree with it fundamentally. The idea of collecting more information
after a failure is good for a system testing framework. A unit testing
framework, however, is intended to test individual use-cases. If it's
broken, it's broken. It should report that and move on to the next case.

My advice, therefore, would be to break the tests up into smaller ones ONLY
when it makes sense that those two tests are separate logical procedures.
Tests should be as small as possible, capturing the functionality needed to
test a detail of a feature. ("Small" is relative, of course, depending on
the complexity of your system.)

*2) All gtest tests start by calling NTP functions init_lib()
andinit_auth(). I've noticed that some Unity tests crash (segmentation
fault)if I don't add these 2 functions.In Gtest, this was solved by calling
these 2 functions in tests-main.cpp(which is a file that starts the gtest
framework)In our case, test runners are automatically generated by a Ruby
script thatanalyzes a  <name>.c file. This leaves me with 2 choices : a)
manually addinit_lib() and/or init_auth() to testRunner-<name>.c when they
require itor b) modify the Ruby script to always call these 2 functions in
it'smain() - doesnt decrease the testing speed by a noticable marginor c)
modify the Ruby script to call these 2 functions if something isdefined in
<name>.c for example #DEFINE REQUIRE_AUTH_FUNI also tried calling
init_auth() in the test setUp() function which runs atthe beginning of
every test, but this crashes the program at the very end,because glibc (at
least I think so) tries to free memory already freed atthe end of the test.*

Curious. My advice was going to be to put it into setUp, until I read the
last paragraph here. The setUp function is meant for things like this, and
it's nice because you're not hiding a requirement away in a script
somewhere... it's self documenting.

I'm not really sure why adding this to setUp would cause a crash at the end
of your program. Even if init_auth() allocates a bunch of memory, there is
nothing in Unity which would require it to be cleaned up unless you ask it
to in tearDown.

Are you running the memory checking stuff in the optional directory of
Unity? It's possible the issue is in there?

*3) Assert functions in Unity have an issue when comparing structs (or
I'mdoing it wrong)In c++ which Gtest uses, you can create a class, override
== operator andwrite your own way to compare objects. However, in C, you
can't do that.For example, let's say we have a struct l_fp.l_fp
a,b;TEST_ASSERT_EQUAL(a,b);won't work, and I'm currently
usingTEST_ASSERT_EQUAL_MEMORY(a,b,sizeof(a));which could pose a problem,
depending on how you you initialize structs. If we don't init structs with
memset(), padding bytes could be different,meaning the ASSERT result could
be wrong.*

:) You are absolutely correct. You should not be using the MEMORY assertion
unless you know there is no padding... which for structs you do not want to
depend on that.

Typically with large unit testing projects, most people create a
unity_helper.h header and make sure it gets included in their tests. In
this they will often create their own helpful macros, like so:

#define TEST_ASSERT_EQUAL_l_fp(a, b) { \
    TEST_ASSERT_EQUAL_UINT32_MESSAGE(a.Xl_ui, b.Xl_ui, "Field Xl_ui"); \
    TEST_ASSERT_EQUAL_INT32_MESSAGE(a.Xl_i, b.Xl_i, "Field Xl_i");

Then you can use these custom assertions from there on. If you're also
using CMock, you'll want to pay careful attention to the naming convention
and it will automatically be able to pull in those types.

> *In gtest, there is a function ASSERT_LE (less or equal). There is no
> equivalent function in Unity. Closest one is
> TEST_ASSERT_DOUBLE_WITHIN(delta, fabs(op1-op3), eps(op2)) . You can't
> specify LE or GE, but a certain range aka delta, which doesn't help us in
> replicating gtest functionality. Should I implement LE, GE, UGE, ULE
> (unsigned less or equal), and similar functions in unity or?*
Currently people use TEST_ASSERT and specify their own operation, but we've
talked about adding a nice version of the macros that would pretty print
the results. We were heading towards something like this:

TEST_ASSERT_OP_UINT(expected, UNITY_LE, actual)
TEST_ASSERT_OP_HEX8(expected, UNITY_GT, actual)

The signedness and size would be handled in the assertion, as is the Unity
standard. We thought we would specify an enum for the operation, to avoid
needing to make so many more macros and to keep it flexible, something like:

UNITY_LT //less than
UNITY_LE //less than or equal to
UNITY_GT //greater than
UNITY_GE //greater than or equal to
UNITY_EQ //equal (redundant, but here for completeness)
UNITY_NE //not equal

Thanks for the great questions! I hope this was helpful.


More information about the GSoC mailing list