Should service layer accept a DTO or a custom request object from the controller?

I'm from C# background but the concept remains same here.

In a situation like this, where we have to pass the parameters/state from application layer to service layer and, then return result from service layer, I would tend to follow separation-of-concerns. The service layer does not need to know about the Request parameter of you application layer/ controller. Similarly, what you return from service layer should not be coupled with what you return from your controller. These are different layers, different requirements, separate concerns. We should avoid tight coupling.

For the above example, I would do something like this:

class Controller
{
     @Autowired
     private ItemService service;

     public ItemResponse createItem(CreateItemRequest request)
     {
        var creatItemDto = GetDTo(request);
        var itemDto = service.createItem(createItemDto);
        return GetItemResponse(itemDto);
    }
}

This may feel like more work since now you need to write addional code to convert the different objects. However, this gives you a great flexiblity and makes the code much easier to maintain. For example: CreateItemDto may have additional/ computational fields as compared to CreateItemRequest. In such cases, you do not need to expose those fields in your Request object. You only expose your Data Contract to the client and nothing more. Similarly, you only return the relevant fields to the client as against what you return from service layer.

If you want to avoid manual mapping between Dto and Request objects C# has libaries like AutoMapper. In java world, I'm sure there should be an equivalent. May be ModelMapper can help.

Conceptually speaking, you want to be able to reuse the service/application layer across presentation layers and through different access ports (e.g. console app talking to your app through a web socket). Furthermore, you do not want every single domain change to bubble up into the layers above the application layer.

The controller conceptually belongs to the presentation layer. Therefore, you wouldn't want the application layer to be coupled upon a contract defined in the same conceptual layer the controller is defined in. You also wouldn't want the controller to depend upon the domain or it may have to change when the domain changes.

You want a solution where the application layer method contracts (parameters & return type) are expressed in any Java native types or types defined in the service layer boundary.

If we take an IDDD sample from Vaughn Vernon, we can see that his application service method contracts are defined in Java native types. His application service command methods also do not yield any result given he used CQRS, but we can see query methods do return a DTO defined in the application/service layer package.

In the above listed 3 methods which ones are correct/wrong?

Both, #1 and #2 are very similar and could be right from a dependency standpoint, as long as ItemDto and CreateItemRequest are defined in the application layer package, but I would favor #2 since the input data type is named against the use case rather than simply the kind of entity it deals with: entity-naming-focus would fit better with CRUD and because of that you might find it difficult to find good names for input data types of other use case methods operating on the same kind of entity. #2 also have been popularized through CQRS (where commands are usually sent to a command bus), but is not exclusive to CQRS. Vaughn Vernon also uses this approach in the IDDD samples. Please note that what you call request is usually called command.

However, #3 would not be ideal given it couples the controller (presentation layer) with the domain.

For example, some methods receive 4 or 5 args. According to Eric Evans in Clean Code, such methods must be avoided.

That's a good guideline to follow and I'm not saying the samples couldn't be improved, but keep in mind that in DDD, the focus is put on naming things according to the Ubiquitous Language (UL) and following it as closely as possible. Therefore, forcing new concepts into the design just for the sake of grouping arguments together could potentially be harmful. Ironically, the process of attempting to do so may still offer some good insights and allow to discover overlooked & useful domain concepts that could enrich the UL.

PS: Robert C. Martin has written Clean Code, not Eric Evans which is famous for the blue book.