[ntp-GSoC] Questions about testing

Damir Tomic viperus.oblivion at gmail.com
Tue Jun 9 07:57:47 UTC 2015


Hello,

I'm one the gsoc students, and I have a few questions related to testing.


I discovered a few issues when converting Google test framework tests to
Unity tests:

1) There is no EXPECT function in Unity, only ASSERT. In Gtest you can use
EXPECT functions for asserts that aren't critical. If an EXPECT function
fails, the test prints a message and continues onto the next ASSERT /
EXPECT wihout failing the test.

Is this a necessary feature? I could implement EXPECT functions in Unity,
if we require them. Unity currently works like this:
If you have 5 asserts in a test, and the 2nd one fails, Unity prints a
message related to the 2nd assert, stops the test and marks it as a
failiure. In order to see if the 3rd assert returns true, you would have to
fix the 2nd assert first.

Should we convert all gtest EXPECT functions to ASSERT in Unity or? We
could also split the gtest test to multiple unity tests, that way we would
get the same effect.


2) All gtest tests start by calling NTP functions init_lib() and
init_auth(). I've noticed that some Unity tests crash (segmentation fault)
if I don't add these 2 functions.

In Gtest, this was solved by calling these 2 functions in tests-main.cpp
(which is a file that starts the gtest framework)

In our case, test runners are automatically generated by a Ruby script that
analyzes a  <name>.c file. This leaves me with 2 choices : a) manually add
init_lib() and/or init_auth() to testRunner-<name>.c when they require it
or b) modify the Ruby script to always call these 2 functions in it's
main() - doesnt decrease the testing speed by a noticable margin
or c) modify the Ruby script to call these 2 functions if something is
defined in <name>.c for example #DEFINE REQUIRE_AUTH_FUN

I also tried calling init_auth() in the test setUp() function which runs at
the beginning of every test, but this crashes the program at the very end,
because glibc (at least I think so) tries to free memory already freed at
the end of the test.


3) Assert functions in Unity have an issue when comparing structs (or I'm
doing it wrong)

In c++ which Gtest uses, you can create a class, override == operator and
write your own way to compare objects. However, in C, you can't do that.

For example, let's say we have a struct l_fp.

l_fp a,b;

TEST_ASSERT_EQUAL(a,b);

won't work, and I'm currently using

TEST_ASSERT_EQUAL_MEMORY(a,b,sizeof(a));

which could pose a problem, depending on how you you initialize structs. If
we don't init structs with memset(), padding bytes could be different,
meaning the ASSERT result could be wrong.



Best regards, Damir Tomić


More information about the GSoC mailing list