How to log from .NET Core Web API into Elasticsearch on own index

Elasticsearch is "just" a log browser. In order to browse your logs, you gonna have to generate those logs.

Configure you application to work with Serilog for instance (https://stackify.com/serilog-tutorial-net-logging/). It will generates the log files.

Then, configure a sink to Elasticsearch (https://github.com/serilog/serilog-sinks-elasticsearch). It will write your logs where elasticsearch can read it.

There is now also a stand alone logger provider that will write .NET Core logging direct to Elasticsearch, following the Elasticsearch Common Schema (ECS) field specifications, https://github.com/sgryphon/essential-logging/tree/master/src/Essential.LoggerProvider.Elasticsearch

Disclaimer: I am the author.

Add a reference to the Essential.LoggerProvider.Elasticsearch package:

dotnet add package Essential.LoggerProvider.Elasticsearch

Then, add the provider to the loggingBuilder during host construction, using the provided extension method.

using Essential.LoggerProvider;

// ...

    .ConfigureLogging((hostContext, loggingBuilder) =>
    {
        loggingBuilder.AddElasticsearch();
    })

You can then inject the ILogger into your controllers, etc, and write to it using the usual .NET logging, including scopes and semantic values (for a general introduction to logging see https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/):

using (_logger.BeginScope("{CustomerId}", customerId))
{
  _logger.LogWarning("End of processing reached at {EndTime}.", end);
}

The default configuration will write to a local Elasticsearch running at http://localhost:9200/.

There is an example project that includes a docker-compose file to set up a local instance of Elasticsearch and Kibana if you need to, https://github.com/sgryphon/essential-logging/tree/master/examples/HelloElasticsearch

The example project also shows best practice for high performance logging, using the Microsoft LoggerMessage helper.

Once you have sent some log events, open Kibana (e.g. http://localhost:5601/) and define an index pattern for "dotnet-*" with the time filter "@timestamp" (this is the default index pattern for the logger provider).

Note: To use the index logging-*, as per the question, you will need to also change a configuration setting and add the following to your appsettings.json file:

{
  "Logging": {
    "Elasticsearch": {
      "Index": "logging-{0:yyyy.MM.dd}"
    }
  }
}

You can then discover the log events for the index. Some useful columns to add are log.level, log.logger, event.code, message, tags, and process.thread.id.

If you are running multiple applications or on multiple servers, you might want to include service.type, service.version, and host.hostname.

Additional fields are defined below, and all individual message and scope values are logged as labels.* custom key/value pairs, e.g. labels.CustomerId.

One benefit of the ElasticsearchLoggerProvider is that it follows the Elasticsearch Common Schema (ECS) for fields, so is compatible with other applications that log to Elasticsearch (e.g. Beats).

Example output: Example - Elasticsearch output via Kibana

Thank you Skrface for your support. I will summarize my code for others who pass by for the same issue. (For CLI and Solution Folders look below.)

Implementing in .NET Core Web API

add NuGet packages:

• Serilog
• Serilog.AspNetCore
• Serilog.Sinks.Elasticsearch

add to appsettings.json:

"Serilog": {
"MinimumLevel": "Information",
"WriteTo": [
  {
    "Name": "RollingFile",
    "Args": {
      "pathFormat": "C:\\Temp\\log-{Date}.txt",
      "outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] {Message}{NewLine}{Exception}"
    }
  }
],
"Properties": {
  "Application": "DataPicker.Api"
}

}

modify the Startup.cs

    public IConfiguration Configuration { get; }
    public Startup(IHostingEnvironment hostingEnvironment)
    {
        var builder = new ConfigurationBuilder()
            .SetBasePath(hostingEnvironment.ContentRootPath)
            .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
            .AddJsonFile($"appsettings.{hostingEnvironment.EnvironmentName}.json", reloadOnChange: true, optional: true)
            .AddEnvironmentVariables();
        Configuration = builder.Build();
        var uri = Configuration["ConnectionStrings:ElasticSearchConnection"];
        Log.Logger = new LoggerConfiguration()
            .Enrich.FromLogContext()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(uri))
            {
                AutoRegisterTemplate = true,
            })
        .CreateLogger();
    }

add to Startup.cs Configure(..)

loggerFactory.AddSerilog();

modify the Controller:

public class MyController : Controller
{
    private readonly ILogger<MyController > logger;

    public MyController (ILogger<MyController> logger)
    {
        this.logger = logger;
    }

and use the logging in the POST / PUT / GET / ... -method like that:

logger.LogDebug("My message");
logger.LogError("Exception: " + ex.Message);

Implementing in .NET Core CLI

add NuGet package:

• Serilog.Sinks.Elasticsearch

add to Programm.cs into the Main(..)

Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Debug()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
    .Enrich.FromLogContext()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("myUri:myPort")) // e.g. "http://localhost:9200"
    {
        AutoRegisterTemplate = true,
    })
    .CreateLogger();

than use it like that:

Log.Debug("Start CLI !");
Log.Error("Can't create data base entry: " + ex.Message);

Implementing in .NET Core Solution Folder

Works just like in the CLI (see above), just use your constructor instead of the Main(..).