Using PREMIRRORS in Bitbake configuration
Note: These results are based on experimentation with Yocto 2.3, but probably apply to 2.5 as well.
A simple example
Suppose that your recipe file contains this target URL:
SRC_URI = "http://download.example.com:8080/foo/bar/baz-1.0.tar.gz"
Then in your local.conf
, you can define your custom download location as:
PREMIRRORS_prepend = "http://download\.example\.com:8080/.* http://my-mirror.example.com/copies/\n"
In this default case (with no special placeholders) Bitbake does not include the additional /foo/bar
path elements, and instead tries to download just the filename from http://my-mirror.example.com/copies/baz-1.0.tar.gz
Advanced examples
These samples use special predefined placeholders, which are detailed in the next section.
HTTP/HTTPS with same file structure
- Recipe:
SRC_URI = "https://example.com:1234/foo/bar.zip"
- Setting:
PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/PATH\n"
- Attempts:
http://mirror.local/foo/bar.zip
HTTP/HTTPS with flat structure
- Recipe:
SRC_URI = "https://example.com:1234/foo/bar.zip"
- Setting:
PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/MIRRORNAME\n"
- Attempts:
http://mirror.local/example.com.1234.foo.bar.zip
Just switch the hostname
- Recipe:
SRC_URI = "ftp://example.com:1234/foo/bar.zip"
- Setting:
PREMIRRORS_prepend = "(\w+)://example\.com(:\d+)?/.* TYPE://mirror.local/PATH\n"
- Attempts:
ftp://mirrors.local/foo/bar.zip
Placeholders in replacement URI
PREMIRRORS
parses all matched URIs and provides five special placeholder values in the target URI. Supposing the matched URI is http://host.example.com:1234/foo/bar/baz.txt
:
TYPE https
HOST host.example.com%3A1234
PATH foo/bar/baz.txt
BASENAME baz.txt
MIRRORNAME host.example.com.1234.foo.bar.baz.txt
Altering the PREMIRRORS
variable
The PREMIRRORS
variable consists of series of lines (separated by \n
) each with a regular expression to match a URI, and then a replacement string, with both portions separated by a space.
Bitbake tries them in order of appearance, and you generally want your private mirrors to take priority, so prepend onto PREMIRRORS
, ex:
PREMIRRORS_prepend = "http://original/location/.* http://alternate/location/\n"
What file should I edit?
You can add entries to PREMIRRORS
inside your bitbake recipes, but it is not recommended, since a major use of PREMIRRORS
is for people reusing your recipe in some other context or location.
Instead, you can put it inside your local.conf
file in an existing build directory. Alternately, edit the source template which Poky script use when creating a new local.conf
in a fresh build-directory.
Other questions
What about SOURCE_MIRROR_URL
?
The SOURCE_MIRROR_URL
is a quick way to add a series of PREMIRROR
entries for all supported protocols. For example, this setting:
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "TYPE://mirror.local/PATH"
is the same as writing:
PREMIRRORS_prepend = "\
cvs://.*/.* TYPE://mirror.local/PATH \
svn://.*/.* TYPE://mirror.local/PATH \
git://.*/.* TYPE://mirror.local/PATH \
gitsm://.*/.* TYPE://mirror.local/PATH \
hg://.*/.* TYPE://mirror.local/PATH \
bzr://.*/.* TYPE://mirror.local/PATH \
p4://.*/.* TYPE://mirror.local/PATH \
osc://.*/.* TYPE://mirror.local/PATH \
https?$://.*/.* TYPE://mirror.local/PATH \
ftp://.*/.* TYPE://mirror.local/PATH \
npm://.*/?.* TYPE://mirror.local/PATH \
"
It seems the INHERIT
+SOURCE_MIRROR_URL
directives will still work if used in your local.conf
(as opposed to a particular recipe.) However, Bitbake will emit warnings, so it may not be the intended use-case. Ex:
WARNING: Invalid protocol in PREMIRRORS: ('cvs://.*/.*', 'TYPE://mirror.local/PATH')
How can I check and debug my settings?
The -D
debug flag will cause bitbake
to emit information about what URLs it attempts to download from. You can also use -C do_fetch
, which will force it to try the fetch step and re-download anything needed for the given recipe.
bitbake -D -C do_fetch software-recipe-name-here
Here's some example debug output, showing the PREMIRROR
URL it attempts to access:
DEBUG: some-software-1.0 do_fetch: Trying PREMIRRORS
DEBUG: some-software-1.0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /home/user/build_foo/DL_DIR 'http://mirror.local/path/to/the/filename.ext
If you need to experiment and run bitbake
many times, it will be faster to temporarily put your new PREMIRRORS_prepend
directive into a particular test-recipe, as opposed to modifying the local.conf
. This is because Bitbake won't need to re-parse all the other recipes whenever you change it.
What if I want to isolate a port-number, e.g. http://host:123/foo
?
Apparently there's no easy way to get the 123
on its own. While PREMIRRORS
allows you to match with regular expressions, it does not seem to support using captured text from the match inside the replacement URI.
The port number is present inside HOST
and MIRRORNAME
, but there's no standard mechanism to split those values apart.