Parallelization of recursive jobs in GNU make
SUBDIRS = a b c
default: all
$(SUBDIRS)::
$(MAKE) -C $@ $(MAKECMDGOALS)
all clean : $(SUBDIRS)
This probably will not answer your question directly, but besides what the other answers suggest, I would recommend to look into non-recursive make techniques. That is truly an elegant way to parallelize build, although, depending on what are the existing Makefiles, can require significant effort. Non-recursive make has advantages not limited to easy parallelization: it sees a complete dependency graph and so it does not need to build too little or too much, meaning faster (sometimes much faster) build times.
Some resources:
- Discussion on SO
- The classic text about trouble with recursive make
- Design of non-recursive makefile
- Implementation
- Another implementation
- Benchmarks (quite outdated though)
Are dir-1, dir-2 and dir-3 interdependent or independent?
I have a similar structure but dependence between the subdirectories so with that I preferred to just use parallel builds within each of the subdirectories. You'd get that via
## default to four parallel runs
MAKEFLAGS += -j 4
all:
@for dir in $(SUBDIRS); do (cd $$dir; $(MAKE) ); done
But another trick is to read up on SUBDIRS in the make manual -- you do not need the for loop as make can unroll this for you. Try something like
## default to four parallel runs
MAKEFLAGS += -j 4
SUBDIRS = dir-1 dir-2 dir-3
$(SUBDIRS): #whatever your depends here
$(MAKE) -C $@
I found that when using the :: (double-colon rule), I could not find a way to force order if there were any dependencies among app, lib, and doc. After reading much of the GNU Make manual, I came up with the following rules to enforce the dependencies at the same time as having a generic recursive make. Notice that the .PHONY rule is there to force make to enter the directory even though the directory already exists.
SUBDIRS = app lib doc
default: all
app: lib
.PHONY: $(SUBDIRS)
$(SUBDIRS):
$(MAKE) -C $@ $(MAKECMDGOALS)
all clean install: $(SUBDIRS)