"..." cannot implement an interface member because it is not public

When implementing an interface member in the class, it should be public

See: Interfaces (C# Programming Guide)

To implement an interface member, the corresponding member of the implementing class must be public, non-static, and have the same name and signature as the interface member.

public class MyDbContext : DbContext, IDatabaseContext {

    public IDbSet<MyEntity1> Entities1 { get; set; }
}

Or as @Marc Gravell said in comment you can do Explicit interface implemenation, More could be found at this answer

I had the same issue only to discover that I forgot to make the method of the interface public. Make sure all the methods of the interface are public too.

However, this makes no sence since the interface obviously IS public. What could be the error here?

No, it isn't. Members on classes are private by default. This Entities1 is private:

public class MyDbContext : DbContext, IDatabaseContext {    
    IDbSet<MyEntity1> Entities1 { get; set; }    
}

Note that this is different to interfaces, where everything is public and access modifiers do not make sense. So: either make the member public:

public class MyDbContext : DbContext, IDatabaseContext {    
    public IDbSet<MyEntity1> Entities1 { get; set; }    
}

or do an explicit interface implementation:

public class MyDbContext : DbContext, IDatabaseContext {    
    IDbSet<MyEntity1> IDatabaseContext.Entities1 { get; set; }    
}