Why can I call an on-VPC aws lambda from an off-VPC aws lambda but not vice versa?

In order to trigger a Lambda function, all that's required is for the caller to have outbound access to the Invoke AWS API. It is not necessary for the invoked Lambda function to have any open inbound ports, or any public Internet access.

So a public (non-VPC, has Internet access) Lambda function can call the Invoke API to trigger the private Lambda function, but the private VPC (no Internet access) Lambda function cannot access the Invoke API to trigger any Lambda function.

Inside VPC you need an AWS VPC interface endpoint (https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint). It allows access to AWS Lambda service without going through the Internet.