calling AddAutoMapper once per assembly instead of passing in multiple assemblies?

The accepted answer was correct at the time but it would appear that things have changed in the recent past.

The AutoMapper.Extensions.Microsoft.DependencyInjection package has been updated to allow the call to AddAutoMapper() multiple times.

See PR Use Microsoft.Extensions.Options to configure AutoMapper for details. You will need to update the package to version 8.0.0 or higher to use it.

The code seems to work fine but AddAutoMapper will be called twice. Once for the API assembly and once for the library assembly. Should I stick to the first approach because AutoMapper should only be added once or is it fine to separate it?

You should stick to the first approach, because AddAutoMappper does nothing when called for the second, third etc. time, thus profiles and other AM related types from the assemblies passed to these calls won't be registered.

It can be seen in the beginning of the implementation of the private method which is called by all public AddAutoMapper overloads:

private static IServiceCollection AddAutoMapperClasses(IServiceCollection services, Action<IServiceProvider, IMapperConfigurationExpression> configAction, 
    IEnumerable<Assembly> assembliesToScan, ServiceLifetime serviceLifetime = ServiceLifetime.Transient)
{
    // Just return if we've already added AutoMapper to avoid double-registration
    if (services.Any(sd => sd.ServiceType == typeof(IMapper)))
        return services;

followed by the actual registration code, which at the end registers IMapper.

Currently there is an open issue Allow usage of Options Pattern to configure AutoMapper #132 with exactly the same concerns as yours.