Using otool (recursively) to find shared libraries needed by an app
Here's my take on the topic. My script is intended to start with the app main executable and traverse recursively all the frameworks. My use is around verifying if the app referenced frameworks match the embedded ones by Xcode. The key assumptions I made to focus on non-system frameworks were:
- import path must begin with
@rpath - must be a framework of
X.framework/Xformat weakframeworks are ignored
If any of these is not needed the awk regex /weak\)$/ { next }; match($1, /^@rpath.*(.framework)/) { ... } may be modified.
First I wrote a shell script:
#/bin/sh
recursiveFrameworksParseStep() {
#fail on 1st otool error
set -e
set -o pipefail #not really POSIX compliant but good enough in MacOS where sh is emulated by bash
otool -L $1|awk -v pwd=${PWD} '/weak\)$/ { next }; match($1, /^@rpath.*(.framework)/) { gsub("@rpath",pwd"/MyApp.app/Frameworks",$1); print $1 }'| while read line; do
if [ $1 != $line ]; then #safety check for otool -L output not to self reference resulting in infinite loop
recursiveFrameworksParseStep $line
fi
done
}
recursiveFrameworksParseStep MyApp.app/MyApp
It will fail on first referenced framework not found on the filesystem.
That's all great, but the drawback is there is no track of visited frameworks, and there might be a lot of duplicate checks. Shell isn't particularly suited for global dictionary like structure to keep track of that.
That's why I rewrote this script using a python3 wrapper:
#!/usr/bin/python3
import subprocess
import os.path
from sys import exit
visitedFrameworks = set()
def fn(executableToProcess):
try:
otoolOut = subprocess.check_output(['otool','-L',executableToProcess])
except subprocess.CalledProcessError:
exit(-1)
pipeOutput = subprocess.Popen(['awk', '-v', os.path.expandvars('pwd=$PWD'),'/weak\)$/ { next };match($1, /@rpath.*(.framework)/) { gsub(\"@rpath\",pwd\"/MyApp.app/MyApp\",$1); print $1 }'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate(otoolOut)
lines = pipeOutput[0].decode('utf-8').split('\n')
for outputLine in lines[1:-1]:
if executableToProcess != outputLine:
if outputLine not in visitedFrameworks:
visitedFrameworks.add(outputLine)
fn(outputLine)
fn("MyApp.app/MyApp")
Conceptually the only difference is keeping track of visited frameworks which leads to dramatic elapsed time reduction (in my case from 7-8s to less than a second).
Finally this can be made an Xcode shell script in the Target build process (shell interpreter likewise set to /usr/bin/python3).
import subprocess
import os.path
from sys import exit
visitedFrameworks = set()
missingFrameworksCandidates = set()
numberOfMissingFrameworks = 0
def fn(executableToProcess):
global numberOfMissingFrameworks
if os.path.exists(executableToProcess):
otoolOut = subprocess.check_output(['otool','-L',executableToProcess])
else:
missingFrameworksCandidates.add(executableToProcess)
return
pipeOutput = subprocess.Popen(['awk', '-v', os.path.expandvars('frameworkPath=$TARGET_BUILD_DIR/$FRAMEWORKS_FOLDER_PATH'),'/weak\)$/ { next };match($1, /@rpath.*(.framework)/) { gsub(\"@rpath\",frameworkPath,$1); print $1 }'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE).communicate(otoolOut)
lines = pipeOutput[0].decode('utf-8').split('\n')
linesWithoutSubFrameworks = []
for outputLine in lines[1:-1]:
frameworkPathToCheck = os.path.dirname(outputLine) + "/Frameworks/"
#check for Frameworks within Frameworks case
if os.path.exists(frameworkPathToCheck) == False:
linesWithoutSubFrameworks.append(outputLine)
continue
#else it's framework within framework which is processed first
if executableToProcess != outputLine:
if outputLine not in visitedFrameworks:
visitedFrameworks.add(outputLine)
fn(outputLine)
for outputLine in linesWithoutSubFrameworks:
if executableToProcess != outputLine:
if outputLine not in visitedFrameworks:
visitedFrameworks.add(outputLine)
fn(outputLine)
# in SPM dependencies the frameworks may end up not in BrandName.app/Frameworks but in BrandName.app/Frameworks/SomeLib.framework/Frameworks
# this check accounts for that
for missingFrameworksCandidate in missingFrameworksCandidates:
if os.path.basename(missingFrameworksCandidate) in visitedFrameworks == False:
print(missingFrameworksCandidate," is missing")
numberOfMissingFrameworks += 1
fn(os.path.expandvars('$TARGET_BUILD_DIR/$EXECUTABLE_PATH'))
exit(numberOfMissingFrameworks)
No, you'll have to run otool repeatedly, or incorporate its parsing code (here). Don't forget about handling @executable_path.
Here it is in Python (without @executable_path, canonicalization, or filenames-with-spaces supported), since this was easier than trying to debug pseudocode:
import subprocess
def otool(s):
o = subprocess.Popen(['/usr/bin/otool', '-L', s], stdout=subprocess.PIPE)
for l in o.stdout:
if l[0] == '\t':
yield l.split(' ', 1)[0][1:]
need = set(['/Applications/iTunes.app/Contents/MacOS/iTunes'])
done = set()
while need:
needed = set(need)
need = set()
for f in needed:
need.update(otool(f))
done.update(needed)
need.difference_update(done)
for f in sorted(done):
print f
Here's my solution that I use to fix macdeployqt's output when using Homebrew-installed libraries. What I've found is that macdeployqt does a good job of putting the dylibs in the Framework folder, but it fails to fix the paths.
https://github.com/jveitchmichaelis/deeplabel/blob/master/fix_paths_mac.py
I've modified Nicholas' script to be a bit more usable - it corrects for @executable_path, @rpath and @loader_path. This isn't exactly production code, but it has let me run apps on other Macs without any dependencies already installed.
Run with: python fix_paths_mac.py ./path/to/your.app/Contents/MacOS/your_exe. i.e. point it to the binary inside an app package and it'll figure out the rest.
I've assumed that most of the problems come from stuff linked to /usr/local. So if the code detects that there's a dependency that points to a file in /usr/local, it'll fix the paths appropriately. You could change the pass statement to copy in a file if it's not in the Frameworks folder, but I've not encountered a situation where there's a missing dylib, it's just linked wrong.
import subprocess
import os
import sys
from shutil import copyfile
executable = sys.argv[1]
app_folder = os.path.join(*executable.split('/')[:-3])
content_folder = os.path.join(app_folder, "Contents")
framework_path = os.path.join(content_folder, "Frameworks")
print(executable)
print("Working in {} ".format(app_folder))
def file_in_folder(file, folder):
return os.path.exists(os.path.join(folder, file))
def otool(s):
o = subprocess.Popen(['/usr/bin/otool', '-L', s], stdout=subprocess.PIPE)
for l in o.stdout:
l = l.decode()
if l[0] == '\t':
path = l.split(' ', 1)[0][1:]
if "@executable_path" in path:
path = path.replace("@executable_path", "")
# fudge here to strip /../ from the start of the path.
path = os.path.join(content_folder, path[4:])
if "@loader_path" in path:
path = path.replace("@loader_path", framework_path)
if "@rpath" in path:
path = path.replace("@rpath", framework_path)
dependency_dylib_name = os.path.split(path)[-1]
if "usr/local" in path:
if app_folder in s:
print("Warning: {} depends on {}".format(s, path))
if file_in_folder(dependency_dylib_name, framework_path):
print("Dependent library {} is already in framework folder".format(dependency_dylib_name))
print("Running install name tool to fix {}.".format(s))
if dependency_dylib_name == os.path.split(s)[-1]:
_ = subprocess.Popen(['install_name_tool', '-id', os.path.join("@loader_path", dependency_dylib_name), s], stdout=subprocess.PIPE)
_ = subprocess.Popen(['install_name_tool', '-change', path, os.path.join("@loader_path", dependency_dylib_name), s], stdout=subprocess.PIPE)
else:
# Potentially you could copy in the offending dylib here.
pass
yield path
need = set([executable])
done = set()
while need:
needed = set(need)
need = set()
for f in needed:
need.update(otool(f))
done.update(needed)
need.difference_update(done)