GitLab issuing temporary IP bans - 403 forbidden

We are running Gitlab EE and for us this issue was caused by a combination of using git lfs inside a build on a Gitlab CI runner, and having installed the rack-attack gem on the Gitlab server.

Background

In order to work around an issue with git-lfs 1.2.1 (where it insisted on requiring username and password despite cloning a public repository), the build contained this line:

git clone https://fakeuser:[email protected]/group/repo.git

On build, this resulted in every LFS request from the runner triggering a login attempt with fakeuser, which obviously failed every time. However, since no login was actually required by the server, the client could continue downloading the files using LFS, and the build passed.

Issue

The IP banning started when the package rack-attack was installed. By default, after 10 failed login attempts, rack-attack bans the origin IP for one hour. This resulted in all runners being completely blocked from Gitlab (even visiting the web page from the runner would return 403 Forbidden).

Workaround (insecure)

A short-term workaround, if the servers (Gitlab runners in our case) are trusted, is to add the servers' IPs to a whitelist in the rack-attack configuration. Adjusting the ban time, or allowing more failed attempts, is also possible.

Example configuration in /etc/gitlab/gitlab.rb:

gitlab_rails['rack_attack_git_basic_auth'] = {
  'enabled' => true,
  'ip_whitelist' => ["192.168.123.123", "192.168.123.124"],
  'maxretry' => 10,
  'findtime' => 60,
  'bantime' => 600
}

In this example, we are whitelisting the servers 192.168.123.123 and 192.168.123.124, and adjusting down the ban time from one hour to 10 minutes (600 seconds). maxretry = 10 allows a user to get the password wrong 10 times before ban, and findtime = 60 means that the failed attempts counter resets after 60 seconds.

Then, you should reconfigure gitlab before changes take effect: sudo gitlab-ctl reconfigure

More details, and for the YAML version of the config example, see gitlab.yml.example.

NOTE: whitelisting servers is insecure, as it fully disables blocking/throttling on the whitelisted IPs.

Solution

The solution to this problem should be to stop the failing login attempts, or possibly just reduce the ban time, as whitelisting leaves Gitlab vulnerable to password brute-force attacks from all whitelisted servers.

Follow next steps to remove ban for your IP

1. Find the IPs that have been blocked in the production log:

   grep "Rack_Attack" /var/log/gitlab/gitlab-rails/production.log

2. Since the blacklist is stored in Redis, you need to open up redis-cli:

   /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket

3. You can remove the block using the following syntax, replacing with the actual IP that is blacklisted:

   del cache:gitlab:rack::attack:allow2ban:ban:<ip>

Source on GitLab Docs: Remove blocked IPs from Rack Attack via Redis