Is HTTPS required for local network server to server communication
The answer to your question comes down to threat modeling. Using cryptographic protocols like HTTPS is a security mechanism to protect against certain threats. If those threats are relevant for you, must be analyzed:
- Are there potential threat actors in your internal network? Based on your question you seem to assume that the internal network can be fully trusted. This is often a misconception, because there are several ways your internal network can be compromised (e.g. valid users with access to this network are turning malicious, a systems in this network gets compromised, a misconfiguration opens up the network segment, etc.).
- Will the architecture be subject to change? It is likely that the system will change over time and prior security assumptions (e.g. my internal network is trusted) no longer hold. If that's a reasonable scenario, it might be a good idea to build the necessary security mechanism in in advance. That's what security best-practices are for. Providing security in an area of uncertainty.
- Is there a regulatory, legal or compliance requirement that must be fulfilled? You said that your customer considers HTTPS to be state-of-the-art / modern best-practice. The source of this friendly worded statement might actually be an externally driven requirement, that must be fulfilled. Non-compliance is a threat that should also be covered in a threat analysis.
Those are important topics worth analyzing. When I design system architectures and I am in doubt, I prefer to err on the side of security. In this case the best-practice approach is indeed using HTTPS for communication, no matter the circumstances, as long as there are isn't a considerable impact on the application (e.g. performance impact).
Difficulty to maintain server certificates shouldn't be a problem nowadays, as this is common practice. This should be part of normal scheduled operations activity.
Having said all this, there is of course additional effort required to use HTTPS instead of HTTP and it is your right to charge the customer for this additional effort. I suggest you calculate what this will cost during development and over time during operation and let the customer decide if the cost is worth the benefit.
Mixing and matching HTTP and HTTPS is not a good idea - you will constantly be juggling configurations.
Usually adding a component into a system should only be done if there is a very specific reason for it - just because someone thought it a good idea is not a specific reason.
I'm not saying that HTTPS is a bad idea - quite the opposite - but you have a lot of learning to do. The model you propose undermines the trust relationship that is the primary reason for using TLS in the first place. You also don't seem to have thought about how to plan your PKI.
servers broken someday in the future because some server has certificate expired and no one know
If you are providing the service, then you should be configuring monitoring for the service, including certificate expiry.
It sounds like you are looking for reasons to argue with the approach of rolling out certificates. Reading between the lines here, it seems you are currently lacking the skills and planning you need to implement this.
Yes, it's a lot of work, but that's the business model - you assess the amount of work, the skills you need to acquire and those you can buy in and you charge the customer for that. (Serge highlights the cost of the certificates - but that is the smallest cost in this whole exercise).
Internal networks are not secure
In general, internal networks are more secure than public-facing systems, but they should not be considered as completely secure. A significant portion of attacks come from the inside - spearphishing, social engineering and insider attacks are all popular vectors which start with a foothold inside of your network.
So there's no good reason for unencrypted traffic of secret or private information even over your internal networks. You don't necessarily need public names or CA hierarchy - if you have well-defined bilateral communications channels, then it may be simpler to have an explicit trust relationship where your load-balancers are configured to trust a particular self-signed certificate of your backend servers and nothing else.