in what case does bindService return false?

The implementation of binderService is in android.app.ConntextImpl:

1412    public boolean bindService(Intent service, ServiceConnection conn, int flags, int userHandle) {
1413        IServiceConnection sd;
1414        if (conn == null) {
1415            throw new IllegalArgumentException("connection is null");
1416        }
1417        if (mPackageInfo != null) {
1418            sd = mPackageInfo.getServiceDispatcher(conn, getOuterContext(),
1419                    mMainThread.getHandler(), flags);
1420        } else {
1421            throw new RuntimeException("Not supported in system context");
1422        }
1423        try {
1424            IBinder token = getActivityToken();
1425            if (token == null && (flags&BIND_AUTO_CREATE) == 0 && mPackageInfo != null
1426                    && mPackageInfo.getApplicationInfo().targetSdkVersion
1427                    < android.os.Build.VERSION_CODES.ICE_CREAM_SANDWICH) {
1428                flags |= BIND_WAIVE_PRIORITY;
1429            }
1430            service.setAllowFds(false);
1431            int res = ActivityManagerNative.getDefault().bindService(
1432                mMainThread.getApplicationThread(), getActivityToken(),
1433                service, service.resolveTypeIfNeeded(getContentResolver()),
1434                sd, flags, userHandle);
1435            if (res < 0) {
1436                throw new SecurityException(
1437                        "Not allowed to bind to service " + service);
1438            }
1439            return res != 0;
1440        } catch (RemoteException e) {
1441            return false;
1442        }
1443    }

From line 1431, you can see it calls ActivityManagerNative's bindService.
This implementation is in com.android.server.am.ActivityManagerService:

11070    public int bindService(IApplicationThread caller, IBinder token,
11071        Intent service, String resolvedType,
11072        IServiceConnection connection, int flags, int userId) {
11073        enforceNotIsolatedCaller("bindService");
11074        // Refuse possible leaked file descriptors
11075        if (service != null && service.hasFileDescriptors() == true) {
11076            throw new IllegalArgumentException("File descriptors passed in Intent");
11077        }
11078
11079        synchronized(this) {
11080            return mServices.bindServiceLocked(caller, token, service, resolvedType,
11081                connection, flags, userId);
11082        }
11083    }

so it finally calls the bindeServiceLocked of com.android.server.am.ActiveServices.

Update:

By reading the code of binderServiceLocked():

472         ServiceLookupResult res =
473             retrieveServiceLocked(service, resolvedType,
474                     Binder.getCallingPid(), Binder.getCallingUid(), userId, true);
475         if (res == null) {
476             return 0;
477         }
478         if (res.record == null) {
479             return -1;
480         }

Found that if result of retrieveServiceLocked() is null, it will return false.
SAfter checking the code of retrieveServiceLocked()

721                 ResolveInfo rInfo =
722                     AppGlobals.getPackageManager().resolveService(
723                                 service, resolvedType,
724                                 ActivityManagerService.STOCK_PM_FLAGS, userId);
725                 ServiceInfo sInfo =
726                     rInfo != null ? rInfo.serviceInfo : null;
727                 if (sInfo == null) {
728                     Slog.w(TAG, "Unable to start service " + service + " U=" + userId +
729                           ": not found");
730                     return null;
731                 }

Found that If can't get the ResolveInfo of Service, it will false.

So I tried to remove the Service declaration in AndroidManifest.xml and find that the bindService() return false.

Also it happens when you try to call BindService of stopped activity (onStop called already)

One very common case in which bindService returns false is if the service was not declared in the Manifest. In that case you should add code like seen below to your manifest file

<manifest ... >
  ...
  <application ... >
      <service android:name=".ExampleService" />
      ...
  </application>
</manifest>