What's the purpose of the JUnit 5 @Nested annotation

The @Nested annotation allows you to have an inner class that's essentially a test class, allowing you to group several test classes under the same parent (with the same initialization).

I just don't understand why we need to have nested test class in our test.

@Nested makes really sense to organize big test classes.

Typical use case

Very often, developer teams define a test class by class to test. That is a shared good practice but it also may make your test class very big and to count several hundred of lines. You can indeed have classes to test with multiple methods to test, multiple scenarios for each one and also some initialization steps required in the unit test methods to test the scenarios.
All of these will naturally increase the test class size.
Above a threshold (maybe 500 lines or about), it becomes legitimate to ask yourself whether a refactoring is needed.

A big class (test class or not), even well organized is harder to read, maintain than multiple classes grouping things with high cohesion/relationship between.
In the unit tests cases, it can be sometime still worse because you may not find a test scenario and write a new one while it existed but you didn't manage to find it because the test class is big.

@Nested : the solution

@Nested addresses this issue by giving the possibility to group multiple test methods inside multiple nested classes of a main(outer) test class.
The test methods of all nested classes defined in the main(outer) test class are handled as any test methods. So @BeforeEach, @AfterEach, @ExtendWith... are applied for all the them.
The single exception is @BeforeAll and @AfterAll :

Only non-static nested classes (i.e. inner classes) can serve as @Nested test classes. Nesting can be arbitrarily deep, and those inner classes are considered to be full members of the test class family with one exception: @BeforeAll and @AfterAll methods do not work by default. The reason is that Java does not allow static members in inner classes. However, this restriction can be circumvented by annotating a @Nested test class with @TestInstance(Lifecycle.PER_CLASS) (see Test Instance Lifecycle).

Using @Nested combined with @DisplayName that takes a String value becomes still finer as the display name will be used for test reporting in IDEs and build tools and may contain spaces, special characters, and even emoji.

Example

I have a FooService with multiple methods and multiple scenarios. I can groups scenarios of the same concern inside nested classes of the unit test class.
Here I choose the method to test to group them (so I group by scenario) but the discriminator could be another thing if it makes sense.

For example :

public class FooServiceTest {

    Foo foo;

    // invoked for ALL test methods
    @BeforeEach
    public void beforeEach() {
         Foo foo = new Foo(...);
    }

    @Nested
    @DisplayName("findWith methods")
    class FindMethods {
        @Test
        void findWith_when_X() throws Exception {
             //...
             foo.findWith(...);
             //...
        }
        @Test
        void findWith_when_Y() throws Exception {
             //...
             foo.findWith(...);
             //...

        }
        @Test
        void findWith_when_Z() throws Exception {
             //...
             foo.findWith(...);
             //...
        }           
    }

    @Nested
    @DisplayName("findAll methods")
    class FindAllMethods {
        @Test
        void findAll_when_X() throws Exception {
             //...
             foo.findAll(...);
             //...
        }
        @Test
        void findAll_when_Y() throws Exception {
             //...
             foo.findAll(...);
             //...

        }
        @Test
        void findAll_when_Z() throws Exception {
             //...
             foo.findAll(...);
             //...
        }   
    }   

    @Nested
    @DisplayName("computeBar methods")
    class ComputeBarMethods {   
         //...

    }

    @Nested
    @DisplayName("saveOrUpdate methods")
    class SaveOrUpdateMethods { 
         //...

    }
}

Sample renderings in the IDE

Child methods of Nesteds are folded by default :

In case or test failure or on demand you can unfold child methods of Nesteds :

All my tests need a database server running. Most of my tests also need a Users table in the database, to be able to log in. In addition to that, some tests need the Friends table, to be able to login and query friends.

Each resource has a setup and teardown. I have to start and stop server, create and delete tables.

With the @Nested annotation, I can group my tests in a hierarchy of nested classes so that every test gets the setup and teardown of all tests up the hierarchy.

This idea of nesting tests was popularized in Ruby. In Java is implemented for Junit 4 by the HierarchicalContextRunner. See the justification on its page https://github.com/bechte/junit-hierarchicalcontextrunner/wiki.