Failure digitally signing a Mac app outside Xcode

Sorry for answering my own question, but I see no other way as editing the original question would lead to spaghetti text.

I finally solved my problem. First the credit: (i) The answer to my other stackoverflow question was very useful and (ii) I got very good (paid) advice from an official Apple developer, by filing a so-called Technical Support Incident (TSI).

On the basis of all this I am now able to give here a very concise recipe of how getting your Mac app successfully treated by GateKeeper. After detailing the recipe I will show what my original mistake was.

Goal: After having developed a Mac app outside Xcode to have GateKeeper issuing the warning "Downloaded from the Internet ..." with three buttons, one of which is "open".

Failure: When GateKeeper issues a warning with either the text ".. unidentified developer.." or the text ".. unconfirmed developer .. " with - in both cases - a messagebox with a single OK button.

Getting your app GateKeeper-ready involves three steps:

• Make your app standalone with no unacceptable external dependencies. The only acceptable external dependences are system libraries. All other dependencies should have been copied to your MyApp.app folder. GateKeeper rejects any app that has non-system external dependencies
• Binaries should not be located at illegal positions inside the MyApp.app folder. Libraries go into MyApp/Contents/Frameworks and the executable goes into MyApp/Contents/MacOS
• All binaries inside MyApp should be digitally signed. Then the MyApp.app folder should be signed. For this signing an Apple "Developer ID Application ..." certificate is necessary

Our recipe is automatic. All the work is done by one script. In case of Qt Creator we use a qmake script where we access the system shell through the $$system command. When using either of the (Xcode) system commands codesign, spctl or check-signature we assume you have redirected stderr to stdout as outlined in answer to question . Otherwise you will not be able to catch the system response when running these utilities. In the following we will not explicitly show this redirection.

HERE IS OUR RECIPE

A. Making the app stand-alone:

1. copy (with a script) all the needed binaries to the MyApp.folder
2. run (with a script) install_name_tool -change and install_name_tool -id such that all dependences inside the app are of the relative type @executable_path/../MacOS.. or @executable_path/../Frameworks
3. run (with a script) otool -L on all binaries inside the MyApp.app folder and flag any illegal dependence, like "@rpath..." or absolute file paths not being system paths. Note that otool -L is not guaranteed to find all dependencies. Plugins are often beyond the horizon of otool. That is why you need the next check.
4. start a terminal at the location "MyApp.app/Contents/MacOS". Run export DYLD_PRINT_LIBRARIES=1. Then run inside the same terminal window ./MyApp. Your terminal will fill up with over hundred loaded libraries. Check this list again for forbidden libraries (libraries present on your computer, but not on the computer of your customers).
5. proof of the pudding is in the eating. We use the MacInCloud virtual machines and check whether or not our app runs there. Alternative solution could be the Mac of a relative who is not a developer. Or you could also create a new user ("test") on your own Mac and copy the app to its Download (or Desktop folder, or ...). In the latter case you must temporarily rename the root folder of your IDE as otherwise the user "test" will find the missing binaries there.

B Signing the app

6. Signing: With our script we run codesign --force --verify --verbose --sign \"Developer ID Application: ....\" \"/path/to/binary\" on all the binaries in the app and then on the app folder itself. In each case the system response is caught. It should contain in each case the string "signed Mach-O thin".
7. Verification: Run (with a script) command codesign --verify --verbose \"/path/to/binary\" on each binary in your app and on the app itself and catch the system response. It should in each case contain the strings "valid on disk" and "satisfies its Designated Requirement".
8. GateKeeper check: Run (with a script) spctl -a -t exec -vv /path/to/binary\" on each binary and on the app folder itself. The system response is caught. It should contain in all cases the string "accepted source".
9. check-signature: Run (with a script) check-signature \"/path/to/banary\" on each binary and on the app folder itself. The system response is caught. It should contain the string "YES" in each case.

C External check

10. zip your app into a single zip file. Upload to one of your cloud servers

11. GateKeepers keeps a long list (typically hundreds of items) of exceptions on its general gate-keeper role. Your app must not be in that list if you want to test GateKeeper. Rather than editing this list a much simpler trick is creating a new user on your Mac. Log in to that user and download the zip file from the Internet cloud server. Finder will automatically uncompress it. Click on it. If GateKeeper tells you that it can open the application but it warns you at the same time that it is downloaded from the Internet, it is time to grab a (white) beer.

Here the desired GateKeeper warning:

My mistake

I did much of the installing and signing without explicitly checking the result for each binary. After that I would use otool -L on a number of binaries but not on all. I missed the fact that upgrading to Qt 5.5 from an earlier Qt version the binary libqminimal.dylib has acquired an extra dependency, viz.: QtDBus. I had not noticed it, but GateKeeper did.

Qt developers might wonder why we not just use macdeployqt for deploying Qt application on a Mac. In the first place we do not like not to use ill-documented black-box utilities. On Internet fora there are quite a number of people reporting issues with macdeployqt. In addition the Qt libraries can have different install locations (as reported by otool-L) when comparing different Qt versions. When we have a new Qt version our script will immediately start to yell about forbidden dependencies. In this way we get information about what has changed in this new version.

adlag's question and self-answer was invaluable in helping me overcome the same issue. However, as good as his recipe is, some statements are not quite right, so I'd like to offer a few additional points.

• It's not necessary to replace @rpath entries in your binaries and dynamic libraries with @executable_path statements. @rpath statements are fine so long as the actual rpath entries embedded in the binaries are not absolute. You can find plenty of valid Qt app bundles that use rpath. You can make it work doing what adlag said, but you may be making work for yourself.
• See jil's comment above for how to use otool -l $file | grep -A2 LC_RPATH and install_name_tool -delete_rpath $path $file to inspect and remove the embedded paths in your binaries and libraries
• See https://developer.apple.com/library/content/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG207 for a clear explanation of why GateKeeper complains about paths in your binaries, and how you can see the specific complaint in syslog.
• If you have a problem with absolute paths, you should first try to fix your build, rather than use install_name_tool after the fact.
• If you're using cmake, this is likely helpful: https://cmake.org/Wiki/CMake_RPATH_handling#Mac_OS_X_and_the_RPATH
• Don't run spctl -a -t exec -vv /path/to/binary on dylib files. You will get errors about the resource envelope. This is expected, and not a problem.
• In my experience, macdeployqt works fine. I solved the problem by changing my build, such that the absolute paths did not get into the offending dylib file (libquazip). I still used install_name_tool to remove the absolute paths to the Qt installation. I then used macdeployqt to create the bundle, sign the bundle and create the DMG file.

My two bits:

1. To really verify codesigning, I had to either upload my DMG to a server and download it using a browser or set the quarantine attribute manually:

   APP_PATH="Any.app"
   xattr -w com.apple.quarantine '0081;5a37dc6a;Google Chrome;F15F7E1C-F894-4B7D-91B4-E110D11C4858' "$APP_PATH"
   xattr -l "$APP_PATH" # You should see the quarantine attribute here
   open "$APP_PATH"
   

   If your app is correctly signed, you should see a system dialog with an "Open" button.

   I found the value of the quarantine attribute by looking at another .app downloaded from the internet. I don't know what the value means.

   I don't really understand why the spctl command says "accepted" even if the Gatekeeper service denies opening the app.

2. I had the "unidentified developer" message box because my Qt frameworks were referenced as "@rpath/QtCore.framework". Changing it to "@application_path/../Frameworks/QtCore.framework" using the install_name_tool fixed the issue in my app.