Soft Asserts – Why should you use them for Unit and Integration tests?

Introduction

Assertions are part of the day-by-day activities of any software engineer. There are numerous ways to apply it to support us in finding issues earlier in the development process. One of the best possible ways to apply it for unit and integration layers, even into another layer, is. the usage of the Soft Assertions approach.

Hard vs Soft Assertions

What’s a Hard Assertion

Hard Assertion is the normal assertion we know: when an expected result does not match the test framework will halt the test execution and an assertion error is thrown. The main thing is that the test execution stops at the first failure, even though you have more assertions in the test.

Example:

import org.assertj.core.api.SoftAssertions;
import org.junit.jupiter.api.Test;

class HardAssertionTest {

    @Test
    void hardAssertion() {
        var person = Person.builder().name("John").phoneNumber(null).age(16).build();

        assertThat(person.getName()).isNotBlank();
        assertThat(person.getPhoneNumber()).isNotNull();
        assertThat(person.getAge()).isGreaterThan(18);
    }
}

Let’s imagine a situation where the Person object cannot have null values and the age attribute must be equal to or greater than 18.

• On line 8 you can see the Person object with the phoneNumber as null and the age as 16
• The assertions on lines 11 and 12 will fail, but as it’s a hard assertion the test execution will be stopped before it reaches the assertion on line 11

This is the test output:

java.lang.AssertionError: 
Expecting actual not to be null

Hard Assertions make sense when they stop the execution as, as soon as the test has an error, the execution must stop. But there are cases in which you have an object, and you need to know which assertions have failed as we commonly assert more than one attribute in the tests.

The Soft Assertion approach will solve it.

What’s a Soft Assertion?

Soft Assertion is an approach to verifying numerous assertions (more than one), by storing them temporarily into an object and running the assertions internally, showing the possible test failures without halting the test execution. In short, the assertions will be done before showing its result.

The tools that support the Soft Assertions normally work like the following example in a pseudo-code:

SoftAssertion softAssertion = new SoftAssertion()

softAssertion.assertSomething...
softAssertion.assertAnotherThing...
softAssertion.assertTheLastThing...

softAssertion.assertThenAll();

The way of working is that the libraries will have a class to manage the assertions. This class will have as many assertions as you want. Think about this class as an array where each assertion is added to it.

The assertions will be performed as soon. as you call a method that tells the class: run them all!
This class will run and will record all the results, not halting the test execution if the expected result does not match.

Why should I use Soft Assertions?

You should use Soft Assertions when you have more than one assertion to apply to the same object.

Why? Because it’s better to know if all the assertions for that object are matching with the expected result rather than running the test multiple times to know what’s not matching.

I have a golden rule for myself: when I have more than one assertion to do, the soft assertion is in place.

Tools support for Soft Assertions

TestNG

If you are using TestNG as your testing library I have good news: it has the Soft Assertion approach implemented as a feature! If you use JUnit, please jump to the AssertJ 🙂

The TestNG has the SoftAssert class that does the same as the pseudocode you read previously: it groups the assertions and verifies them as soon as we call a specific method.

public class SoftAssertTestNGTest {

    @Test
    public void testNGSoftAssertion() {
        var person = Person.builder().name("John").phoneNumber(null).age(16).build();
        SoftAssert softAssert = new SoftAssert();

        softAssert.assertEquals(person.getName(), "John");
        softAssert.assertNotNull(person.getPhoneNumber(), "Phone number cannot be null");
        softAssert.assertEquals(person.getAge(), 25, "Age should be equal");

        softAssert.assertAll();
    }
}

The example below has the same requirement: the phoneNumber cannot be null and the age must be equal to 25.

• On line 6 you can see a necessary instance of the SoftAssertion class
• Lines 8 to 10 use the softAssertion before the assertion methods, telling the code that it belongs to the SoftAssertion class. You can use any supported assertion from TestNG, as its proxies assertions
• Line 12 calls the assertAll() method, which will run all the assertions associated with the softAssertion reference

We can see that an assertion error will be shown, and this will be the result:

java.lang.AssertionError: The following asserts failed:
	Phone number cannot be null
Expected :25
Actual   :16
<Click to see difference>

Instead of halting the test execution, TestNG ran all the assertions showing all the failures. You can see the AssertionError describing the failure in the phone number and the difference between the expected and actual result for the age attribute.

JUnit 5

JUnit 5 has the assertAll() method as the soft assertion approach. You can see it in action in their assertions example.

It does not have an external specific class to use, it’s already part of the Assertions class. All you need to do is import it statically.

Example:

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertAll;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;

class SoftAssertionJunit5Test {
    @Test
    void softAssertionUsingJUnit5() {
        var person = Person.builder().name("John").phoneNumber(null).age(16).build();

        assertAll("person",
                () -> assertNotNull(person.getName(), "Name must not be null"),
                () -> assertNotNull(person.getPhoneNumber(), "Phone number should not be null"),
                () -> assertEquals(18., person.getAge(), "Age must be 18")
        );
    }
}

• Line 12 shows the assertAll() method using two parameters
  • a heading, as String, to identify the assertions
  • a Stream of Executable commands, meaning the assertion methods
• Lines 13 to 15 show the usage of the Streams, each for any assertion we must apply

You will see the following exception in the console:

org.opentest4j.AssertionFailedError: Phone number should not be null ==> expected: not <null>
	at org.example.SoftAssertionJunit5Test.lambda$softAssertionUsingJUnit5$1(SoftAssertionJunit5Test.java:17)

org.opentest4j.AssertionFailedError: Age must be 18 ==> 
Expected :18.0
Actual   :16.0
	at org.example.SoftAssertionJunit5Test.lambda$softAssertionUsingJUnit5$2(SoftAssertionJunit5Test.java:18)

org.opentest4j.MultipleFailuresError: person (2 failures)
	org.opentest4j.AssertionFailedError: Phone number should not be null ==> expected: not <null>
	org.opentest4j.AssertionFailedError: Age must be 18 ==> expected: <18.0> but was: <16.0>

The exception is divided into 2 parts:

• The list of failures with the explanation and stack trace
• The summary of failures

AssertJ

The AssertJ assertion library provides different ways to apply Soft Assertions, with the possibility to create your own:

• Calling assertAll() (basic approach)
• Using a JUnit 4 rule that takes care of calling assertAll() after each test
• Using the provided JUnit 5 extension which injects a SoftAssertions or a BDDSoftAssertions parameter and calls assertAll() after each test
• Using a AutoCloseableSoftAssertions
• Using assertSoftly static method

You can use all these different ways to apply it and see all the examples on the AssertJ page. Here you see the assertSoftly static method in use, that’s the most convenient way.

import org.assertj.core.api.SoftAssertions;
import org.junit.jupiter.api.Test;

class SoftAssertionTest {

    @Test
    void softAssertionUsingAssertJ() {
        var person = Person.builder().name("John").phoneNumber(null).age(16).build();

        SoftAssertions.assertSoftly(softly -> {
            softly.assertThat(person.getName()).isNotBlank();
            softly.assertThat(person.getPhoneNumber()).isNotNull();
            softly.assertThat(person.getAge()).isGreaterThan(18);
        });
    }
}

• Line 12 shows the assertSoftly method using a single parameter: the Consumer of SoftAssertions, which we need to name it
• Lines 11 to 13 use the consumer reference to call the assertion method

The exception output will look like this:

java.lang.AssertionError: 
Expecting actual not to be null
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:16)

java.lang.AssertionError: 
Expecting actual:
  16
to be greater than:
  18
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:17)

org.assertj.core.error.AssertJMultipleFailuresError: 
Multiple Failures (2 failures)
-- failure 1 --
Expecting actual not to be null
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:16)
-- failure 2 --
Expecting actual:
  16
to be greater than:
  18
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:17)

• The list of AssertionErrors and its full stack trace (which was omitted in this example)
• The summary of failures with the link for the line of the class that has the assertion error

Which one should I use?

Well, it depends on your preferences. You will have a way to use the Soft Assertions either in JUnit or TestNG.

I would recommend using it from AssertJ. Why? Because it extends the assertions in a lot of different ways not limiting you. For example: did you notice that using AssertJ, instead of isEqualsTo() (or any equals variation) we are using isGreatherThan() method?

AssertJ will complement the way we can assert the expected results and it provides a lot of different ways to apply the Soft Assertions, even extending them and creating your own if you want.

Takeaways

• you learned what’s the difference between hard and soft assertions
• you learned which most used tools and libraries support it
• you learned how to apply the soft assertion in the supported tools and libraries