Introduction

In a software development project, errors can appear in any of the stages of the life cycle, the later they appear the more effort is needed to fix those errors. So testing can avoid errors in later stages and thus reduce costs. But wait, there are much for more benefits!

Why do I have to test my code? I Know that it works

Thats a question unexperienced developers may ask. In fact I have often heard that question. We already covered the fact that finding errors in early stages helps to reduce costs. But lets see what other benefits there are, the most important ones are:

• More Knowledge about the Software: Tests give confidence in the code. By running tests after the code has been modified, it can be confirmed that the changes did not break existing code as defined by the tests. Thus, bugs and errors are being avoided early on. Additionaly, if bugs occur, then you already know where you to not look for the bug. Because, you know that the tested code works as intended. Testings makes bug less likely and easier to resolve.

• Software Quality: This is probably the most crucial point. Testing increases the software quality. Theres multiple reasons for that, some of them are also mentioned in this list. A good software quality is a must for every bigger project and inevitably needed to ensure a long term stable and maintainable software.

• Software Design: Writing code that is easily testable automatically improves the software design and architecture. Testable code is usually well structured and encapsulated. Knowing that the code must be tested, forces the code to be written in a modular way.

• Accelerate The Development: Software testing helps developers find errors and scenarios to reproduce the error. This truly makes it so simple for the developer to find and fix errors fast.

• Customer Satisfaction: A stable "bug-free" software generates trust and customer satisfaction. Trust is hard to earn, but easy to be destroyed by bugs / errors and unstable software.

Different types of tests

Unit Tests: Unit tests are low-level tests that focus on testing a specific part of the Software. They are usually written very fast with low effort. Test failures should provide enough information to find the source of an occuring error.

Integration Tests: The compatibility and functionality of the interfaces between the different "units" that make up a system are tested.

Validation Tests: These are the tests performed on fully integrated software to evaluate the fulfillment specified requirements.

System Tests: the validated software is integrated with the rest of the system and tested.

Unit tests are cheap to write and easy to automate. Additionaly, the benefit of unit tests is very high. Thats why the number of unit tests should and also generally exceeds the number of other test types.

Tests on the bottom of the pyramide are easy to automate, the ones further to the top are more difficult to automate.

Properties of good tests are: They are easily automated, they run fast and they are deterministic.

When should I use tests?

Short answer: Always!

Long answer: Almost always, I would say that you should use automated tests (at least unit tests) for any software that exceeds a size of 5 files or 500 lines of code. Yes that means that almost any software should be tested. Once you learned how to setup tests and automated build systems, the cost of doing so becomes very very low. However, the befenits grow the bigger and the longer your software lives.

Common software testing tools & How do tests look like?

Now, lets see how we can acutally write and run tests in some of the most commonly used programming languages:

Java

Java has a very powerfull testing environment integrated in the standard tools called JUnit. It is very commonly used all around the Java ecosphere.

A typical unit test can look like the following. This unit test checks some basic geometric math functionality between triangles and lines.

/**
    * Test method for
    * {@link Triangle#lineIntersections(ILine)}
    */
@Test
public void testLineIntersections() {
    Vector2 a = Vector2.fromXY(0.0, 0.0);
    Vector2 b = Vector2.fromXY(1000.0, 0.0);
    Vector2 c = Vector2.fromXY(0.0, 1000.0);
    Triangle triangle = Triangle.fromCorners(a, b, c);

    ILine line1 = edu.tigers.sumatra.math.line.Line.fromPoints(Vector2.fromXY(1, 1), Vector2.fromXY(2, 2));
    assertThat(triangle.lineIntersections(line1)).hasSize(3);

    ILine line4 = edu.tigers.sumatra.math.line.Line.fromPoints(Vector2.fromXY(0, 500), Vector2.fromXY(500, 500));
    assertThat(triangle.lineIntersections(line4)).hasSize(2);

    ILine line2 = edu.tigers.sumatra.math.line.Line.fromPoints(Vector2.fromXY(-1, 0), Vector2.fromXY(-1, 0));
    assertThat(triangle.lineIntersections(line2)).isEmpty();

    ILine line3 = edu.tigers.sumatra.math.line.Line.fromPoints(Vector2.fromXY(0, -1), Vector2.fromXY(1, -1));
    assertThat(triangle.lineIntersections(line3)).isEmpty();
}

Python

In python the unittest module is very common for writing unit tests.

import unittest

class TestSum(unittest.TestCase):

    def test_sum(self):
        self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")

    def test_sum_tuple(self):
        self.assertEqual(sum((1, 2, 2)), 6, "Should be 6")

if __name__ == '__main__':
    unittest.main()

source: https://realpython.com/python-testing/#unit-tests-vs-integration-tests

C/C++

Sadly the C/C++ environment does not offer any standard tools for testing purposes. However, there are still some widely used third party tools. One of them is googletest, https://github.com/google/googletest.

#ifndef CORE_POWERESTIMATIONTEST_H
#define CORE_POWERESTIMATIONTEST_H

#include "gtest/gtest.h"
#include "../sim/PowerEstimation.h"
#include "../model/Vector3.h"
#include "../model/Vector2.h"
#include "../model/Receiver.h"
#include "../model/Heliostat.h"

class PowerEstimationTest : public ::testing::Test
{

protected:

    PowerEstimationTest()
    {
    }

    ~PowerEstimationTest() override
    {
    }

    void SetUp() override
    {
        Test::SetUp();
    }

    void TearDown() override
    {
        Test::TearDown();
    };

};

TEST_F(PowerEstimationTest, TestHeliostatEfficiencyCalculation)
{
    PowerEstimation estimation{};

    Receiver receiver{Vector3(0,0,60)};
    receiver.setWidth(10);
    receiver.setLength(10);
    receiver.initialize();

    Heliostat heliostatBadAngleAndFar{Vector3(0,500,5), Vector2(0,0)};
    Heliostat heliostatMid{Vector3(0,100,5), Vector2(0,0)};
    Heliostat heliostatGoodAngle{Vector3(0,10,5), Vector2(0,0)};

    Vector3 sunVector(0,0,1);

    double efficiencyLow = estimation.estimateHeliostatEfficiency(sunVector, heliostatBadAngleAndFar, receiver);
    double efficiencyMid = estimation.estimateHeliostatEfficiency(sunVector, heliostatMid, receiver);
    double efficiencyHigh = estimation.estimateHeliostatEfficiency(sunVector, heliostatGoodAngle, receiver);

    ASSERT_LE(efficiencyMid, efficiencyHigh);
    ASSERT_LE(efficiencyLow, efficiencyMid);
}

#endif //CORE_POWERESTIMATIONTEST_H

Continuous Integration (CI)

Continuois Integration not only consists of automated testing, but also building, integrating, testing and reporting.

There is a very wide range of different tools and frameworks for CI. In this blog post I will talk more about two of them. Jenkins and the Gitlab envrionment.

Jenkins

The leading open source automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project. Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.

Read more about Jenkins on their webiste: https://jenkins.io/

Gitlab

Besides Gitlab being a very powerful git hosting and issue tracking platform, it also comes with a built in CI environemnt. Cool! (Its my absolute favorite tool!)

The following source code shows a rather complex gitlab-ci yml file (the .yml file define how the build server should build and test the project) that builds and tests a core software written in C++ (with a cmake build system), a GUI software written in java (with a maven build system). The software is built within a docker container. Furthermore, an asciidoc documentation is being built. Thus, this a very powerful setup!

image: fluxsim

stages:
  - build
  - test
  - documentation

build-core-linux:
  stage: build
  tags: 
    - linux
    - fluxsim
  script:
    - cd core
    - mkdir -p build && cd build
    - cmake ..
    - make
  artifacts:
    paths:
      - ./core/build/core

build-gui:
  stage: build
  tags: 
    - linux
    - fluxsim
  script:
    - cd gui
    - mvn package
  artifacts:
    paths:
      - ./gui/target/*.jar

build-doc:
  stage: documentation
  tags: 
    - linux
    - fluxsim
  script:
    - asciidoctor-pdf -r asciidoctor-diagram ./documentation/documentation.adoc
    - mv ./documentation/documentation.pdf ./
  artifacts:
    paths:
      - ./documentation.pdf

unit-test-core:
  stage: test
  tags: 
    - linux
    - fluxsim
  script:
    - cd core
    - mkdir -p build && cd build
    - cmake .. -DUNIT_TEST=ON
    - make
  artifacts:
    reports:
      junit: $CI_PROJECT_DIR/core/build/test/unit_test_report.xml

Gitlab uses so called "gitlab-runners" to execute the build pipeline.

Test Driven Development

What is it? Straight from wikipedia:

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved so that the tests pass.

Pratically this means that the test cases are written and implemented before the actualy functionality. This will lead to a "failing" test at first. After the implementation of the test case the developer starts to implement the actual feature. Until the test fails no more. TDD can lead to more modularized, flexible, and extensible code.