Soft Asserts - Why should you use them for Unit and Integration tests?

How a Service Virtualization approach can save your and others’ time playing with Mocks

January 2, 2023

JUnit 5 – When to use @ValueSource

April 4, 2023

Published by Elias on March 27, 2023

Introduction

Assertions are part of the day-by-day activities of any software engineer. There are numerous ways to apply it to support us in finding issues earlier in the development process. One of the best possible ways to apply it for unit and integration layers, even into another layer, is. the usage of the Soft Assertions approach.

Hard vs Soft Assertions

What’s a Hard Assertion

Hard Assertion is the normal assertion we know: when an expected result is not matching the test framework will halt the test execution and an assertion error is thrown. The main thing is that the test execution stops at the first failure, even though you have more assertions in the test.

Example:

	import org.assertj.core.api.SoftAssertions;
	import org.junit.jupiter.api.Test;

	class HardAssertionTest {

	@Test
	void hardAssertion() {
	var person = Person.builder().name("John").phoneNumber(null).age(16).build();

	assertThat(person.getName()).isNotBlank();
	assertThat(person.getPhoneNumber()).isNotNull();
	assertThat(person.getAge()).isGreaterThan(18);
	}
	}

view raw HardAssertionTest.java hosted with ❤ by GitHub

Let’s imagine a situation where the Person object cannot have null values and the age attribute must be equal to or greater than 18.

On line 8 you can see the Person object with the phoneNumber as null and the age as 16
The assertions on lines 11 and 12 will fail, but as it’s a hard assertion the test execution will be stopped before it reaches the assertion on line 11

This is the test output:

java.lang.AssertionError: 
Expecting actual not to be null

Hard Assertions make sense when it stops the execution as, as soon as the test has an error, the execution must stop. But there are cases in which you have an object, and you need to know which assertions have failed as we commonly assert more than one attribute in the tests.

The Soft Assertion approach will solve it.

What’s a Soft Assertion?

Soft Assertion is an approach to verifying numerous assertions (more than one), by storing them temporarily into an object and running the assertions internally, showing the possible test failures without halting the test execution. In short, the assertions will be done before showing its result.

The tools that support the Soft Assertions normally work like the following example in a pseudo-code:

SoftAssertion softAssertion = new SoftAssertion()

softAssertion.assertSomething...
softAssertion.assertAnotherThing...
softAssertion.assertTheLastThing...

softAssertion.assertThenAll();

The way of working is that the libraries will have a class to manage the assertions. This class will have as many assertions as you want. Think about this class as an array where each assertion is added to it.

The assertions will be performed as soon. as you call a method that tells the class: run them all!
This class will run and will record all the results, not halting the test execution if the expected result does not match.

Why should I use Soft Assertions?

You should use Soft Assertions when you have more than one assertion to apply to the same object.

Why? Because it’s better to know if all the assertions for that object are matching with the expected result rather than running the test multiple times to know what’s not matching.

I have a golden rule for myself: when I have more than one assertion to do, the soft assertion is in place.

Tools support for Soft Assertions

Tool	Type	Support	Notes
AssertJ	Assertion library	✅	The best choice!
Truth	Assertion library	❌	There’s an open issue, from 2021 with a proposal to have it
Hamcrest	Assertion library	❌	No support 🙁
JUnit 5	Testing framework	✅	Supported by the `assertAl()` method
TestNG	Testing framework	✅	Supported by the `SoftAssert` class

List of supported tools and libraries for Soft Assertion

TestNG

If you are using TestNG as your testing library I have good news: it has the Soft Assertion approach implemented as a feature! If you use JUnit, please jump to the AssertJ 🙂

The TestNG has the SoftAssert class that does the same as the pseudocode you read previously: it groups the assertions and verifies them as soon as we call a specific method.

	public class SoftAssertTestNGTest {

	@Test
	public void testNGSoftAssertion() {
	var person = Person.builder().name("John").phoneNumber(null).age(16).build();
	SoftAssert softAssert = new SoftAssert();

	softAssert.assertEquals(person.getName(), "John");
	softAssert.assertNotNull(person.getPhoneNumber(), "Phone number cannot be null");
	softAssert.assertEquals(person.getAge(), 25, "Age should be equal");

	softAssert.assertAll();
	}
	}

view raw SoftAssertTestNGTest.java hosted with ❤ by GitHub

In the example below w have the same requirement: the phoneNumber cannot be null and the age must be equal to 25.

On line 6 you can see a necessary instance of the SoftAssertion class
Lines 8 to 10 use the softAssertion before the assertion methods, telling the code that it belongs to the SoftAssertion class. You can use any supported assertion from TestNG, as its proxies assertions
Line 12 calls the assertAll() method, which will run all the assertions associated with the softAssertion reference

We can see that an assertion error will be shown, and this will be the result:

java.lang.AssertionError: The following asserts failed:
	Phone number cannot be null
Expected :25
Actual   :16
<Click to see difference>

Instead of halting the test execution, TestNG ran all the assertions showing all the failures. You can see the AssertionError describing the failure in the phone number and the difference between the expected and actual result for the age attribute.

JUnit 5

JUnit 5 has the assertAll() method as the soft assertion approach. You can see it in action in their assertions example.

It does not have an external specific class to use, it’s already part of the Assertions class. All you need to do is import it statically.

Example:

	import org.junit.jupiter.api.Test;

	import static org.junit.jupiter.api.Assertions.assertAll;
	import static org.junit.jupiter.api.Assertions.assertEquals;
	import static org.junit.jupiter.api.Assertions.assertNotNull;

	class SoftAssertionJunit5Test {
	@Test
	void softAssertionUsingJUnit5() {
	var person = Person.builder().name("John").phoneNumber(null).age(16).build();

	assertAll("person",
	() -> assertNotNull(person.getName(), "Name must not be null"),
	() -> assertNotNull(person.getPhoneNumber(), "Phone number should not be null"),
	() -> assertEquals(18., person.getAge(), "Age must be 18")
	);
	}
	}

view raw SoftAssertionJunit5Test.java hosted with ❤ by GitHub

Line 12 shows the assertAll() method using two parameters
- a heading, as String, to identify the assertions
- a Stream of Executable commands, meaning the assertion methods
Lines 13 to 15 show the usage of the Streams, each for any assertion we must apply

You will see the following exception in the console:

org.opentest4j.AssertionFailedError: Phone number should not be null ==> expected: not <null>
	at org.example.SoftAssertionJunit5Test.lambda$softAssertionUsingJUnit5$1(SoftAssertionJunit5Test.java:17)

org.opentest4j.AssertionFailedError: Age must be 18 ==> 
Expected :18.0
Actual   :16.0
	at org.example.SoftAssertionJunit5Test.lambda$softAssertionUsingJUnit5$2(SoftAssertionJunit5Test.java:18)

org.opentest4j.MultipleFailuresError: person (2 failures)
	org.opentest4j.AssertionFailedError: Phone number should not be null ==> expected: not <null>
	org.opentest4j.AssertionFailedError: Age must be 18 ==> expected: <18.0> but was: <16.0>

The exception is divided into 2 parts:

The list of failures with the explanation and stack trace
The summary of failures

AssertJ

The AssertJ assertion library provides different ways to apply Soft Assertions, with the possibility to create your own:

Calling assertAll() (basic approach)
Using a JUnit 4 rule that takes care of calling assertAll() after each test
Using the provided JUnit 5 extension which injects a SoftAssertions or a BDDSoftAssertions parameter and calls assertAll() after each test
Using a AutoCloseableSoftAssertions
Using assertSoftly static method

You can use all these different ways to apply it and see all the examples on the AssertJ page. Here you see the assertSoftly static method in use, that’s the most convenient way.

	import org.assertj.core.api.SoftAssertions;
	import org.junit.jupiter.api.Test;

	class SoftAssertionTest {

	@Test
	void softAssertionUsingAssertJ() {
	var person = Person.builder().name("John").phoneNumber(null).age(16).build();

	SoftAssertions.assertSoftly(softly -> {
	softly.assertThat(person.getName()).isNotBlank();
	softly.assertThat(person.getPhoneNumber()).isNotNull();
	softly.assertThat(person.getAge()).isGreaterThan(18);
	});
	}
	}

view raw SoftAssertionTest.java hosted with ❤ by GitHub

Line 12 shows the assertSoftly method using a single parameter: the Consumer of SoftAssertions, which we need to name it
Lines 11 to 13 use the consumer reference to call the assertion method

The exception output will look like this:

java.lang.AssertionError: 
Expecting actual not to be null
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:16)

java.lang.AssertionError: 
Expecting actual:
  16
to be greater than:
  18
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:17)

org.assertj.core.error.AssertJMultipleFailuresError: 
Multiple Failures (2 failures)
-- failure 1 --
Expecting actual not to be null
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:16)
-- failure 2 --
Expecting actual:
  16
to be greater than:
  18
at SoftAssertionTest.lambda$assertJSoftAssertion$0(SoftAssertionTest.java:17)

The list of AssertionErrors and its full stack trace (which was omitted in this example)
The summary of failures with the link for the line of the class that have the assertion error

Which one should I use?

Well, it depends on your preferences. You will have a way to use the Soft Assertions either in JUnit or TestNG.

I would recommend using it from AssertJ. Why? Because it extends the assertions in a lot of different ways not limiting you. For example: did you notice that using AssertJ, instead of isEqualsTo() (or any equals variation) we are using isGreatherThan() method?

AssertJ will complement the way we can assert the expected results and it provides a lot of different ways to apply the Soft Assertions, even extending them and creating your own if you want.

Takeaways

you learned what’s the difference between hard and soft assertions
you learned which most used tools and libraries support it
you learned how to apply the soft assertion in the supported tools and libraries

Elias

I help professional software engineers (backend, frontend, qa) to develop their quality mindset and deliver bug-free software so they become top-level developers and get hired for the best positions in the market.

1 Comment

deepali gautam says:

August 20, 2023 at 3:27 PM

Awesome Read!

Reply

Soft Asserts – Why should you use them for Unit and Integration tests?

How a Service Virtualization approach can save your and others’ time playing with Mocks

JUnit 5 – When to use @ValueSource

Elias

Leave a Reply Cancel reply

Soft Asserts – Why should you use them for Unit and Integration tests?

How a Service Virtualization approach can save your and others’ time playing with Mocks

JUnit 5 – When to use @ValueSource

How a Service Virtualization approach can save your and others’ time playing with Mocks

JUnit 5 – When to use @ValueSource

Introduction

Hard vs Soft Assertions

What’s a Hard Assertion

What’s a Soft Assertion?

Why should I use Soft Assertions?

Tools support for Soft Assertions

TestNG

JUnit 5

AssertJ

Which one should I use?

Takeaways

Elias

Related posts

JUnit 5 – When to use Internal MethodSource

Better Page Objects strategy using AjaxElementLocatorFactory with Selenium and Java

The best way to create browser instances using the Factory Pattern, Java, and Selenium WebDriver

1 Comment

Leave a Reply Cancel reply