Making mirror://mirrors.ubuntu.com highly available

Personally I think that the best way to select the best Ubuntu repositories mirrors is to use the GUI method:

download server

Now, to improve the situation described in the question, you need to set some rules somehow. These rules must to act on mirrors.ubuntu.com. I can suggest some rules as follow:

  • make a list of best/preferred mirrors; there are a lot of mirrors as you can see here, here or here
  • if you found a good mirror, add it to the list
  • if a mirror was down or broken sometimes, it means that is not a good mirror and you should remove it from the list
  • you can use netselect, apt-spy or apt-fast
  • and others, depending on your requirements.

Next, to see how you can work around, I can give you a method described step by step with three bash scripts examples. First script use the mirrors from the country you are in at the moment instead of mirrors.ubuntu.com/mirrors.txt (for each country there is a text file with mirrors asociated; see http://mirrors.ubuntu.com/):

  • In a terminal run mkdir -p bin - this command will make a bin directory in your home folder if you don't already have it.
  • After run gedit ~/bin/change_sources.sh - this will create the new file change_sources.sh in gedit.
  • Copy and paste one of the next scripts in the new created file:
#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print toupper($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb mirror://mirrors.ubuntu.com/$country.txt $release main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-security main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, something similar to what can be found at http://repogen.simplylinux.ch/:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print tolower($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Main Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe  multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe multiverse

## Ubuntu Update Repos for $ip
deb http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, a script using netselect (download from here, install instructions here) as izx explained verry nice in this answer:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

url=$(netselect \
    `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors \
        | grep -P -B8 "statusUP|statusSIX" \
        | grep -o -P "(f|ht)tp.*\"" \
        | tr '"\n' '  '` \
    | awk '{print $2}')
release=$(lsb_release -sc)

if [ "$url" == "" ] ; then
    exit 0
fi

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

cp -f $file $old_file

printf "## Ubuntu Best Repos
deb http://extras.ubuntu.com/ubuntu $release main
deb-src http://extras.ubuntu.com/ubuntu $release main
deb $url $release main universe restricted multiverse
deb http://security.ubuntu.com/ubuntu/ $release-security restricted universe main multiverse
deb $url $release-updates restricted universe main multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0
  • Save the file and close it.
  • Go back into terminal and run: chmod +x ~/bin/change_sources.sh - to grant execute access for the script.
  • Just for test, to run your new script, type in terminal ~/bin/change_sources.sh. It will give you an error, because you don't have the right to edit /etc/apt/sources.list. So, use sudo ~/bin/change_sources.sh
  • Edit the root user's crontab file using sudo crontab -e command and add the following line:
@hourly /home/$USER/bin/change_sources.sh  
#change $USER with your user name
  • I have set the cron job for every hour, but you can change as you wish or as you think is better. See http://en.wikipedia.org/wiki/Cron in this sense.
  • Save the file and check the new crontab entry with sudo crontab -l.

NOTE: To revert the changes made by this script, delete the cron job and follow the indications from the picture above or use next command in terminal: 

cp -f /etc/apt/sources.list.bak /etc/apt/sources.list

From now, the file will be dynamically changed after it finds a change of the IP address.

It may not be the best solution, but, in my opinion, a good solution can be given in this way like in the scripts above.


I appreciate all the input on this question, but since no one came up with a simple solution which fit our circumstances, I decided to fix the problem myself.

I created a tool (specifically for Ubuntu) which I call apt-spy2.

The primary objective of this tool is to find a working mirror fast. Working is defined by that the mirror server is available and (hopefully :) up to date.

I make no assumptions about if the selected server is necessarily the closest and the fastest. I'm not doing any pings or GEO DNS tricks — but so far this works when something breaks.

How it works — in a nutshell:

  1. I use http://mirrors.ubuntu.com or launchpad's list of mirrors to retrieve servers.
  2. I do a simple check on each (for HTTP Response Status Code).
  3. LBNL, I update /etc/apt/sources.list.

Please note: This assumes that people play nice and put additional mirrors (e.g. 3rd party repositories into /etc/apt/sources.list.d. But I guess that means there's room for improvement.

You can obtain this tool like so:

$ [sudo] gem install apt-spy2

The cli comes with list, check, fix and help (with extended information on how to use it).

I tried to document as much as possible in the project's README.

The current version is a very conservative 0.5.0.

The code is open source and the license is liberal. And I take all contributions.


There was this command at Debian 6.0.4 :

apt-spy

This did the job to find the next closest available server automatic and to generate a new sources.list

In Ubuntu this command seems not to exist ?

It still exists in Debian 7.0 wheezy :

https://launchpad.net/debian/wheezy/+source/apt-spy/+copyright

You can download your *.deb package here :

http://packages.debian.org/sid/apt-spy

... still searching for the sources ...

You obviously need Debian-7.0-Installation for to get the source-code after editing sources-list with entry :

deb-src http://http.debian.net/debian wheezy main

Then after sudo apt-get update you would simply suck the code with :

sudo apt-get source apt-spy

Tags:

Apt

Mirrors

Recent Posts