IP Spoofing: How secure is to control access by user’s public IP address?

Short Answer

Assuming a TCP connection, it is nearly impossible to spoof a source IP address without control of the network.

Longer Answer

Assuming you are not using any proxies (which can cause issues if you're getting their IP address from a X-FORWARDED-FOR header), and running a service on TCP, it's extremely difficult to spoof a source IP address

To initialize a TCP connection multiple packets have to be sent back and forth between the server and the attacker. If the source address for the initial request is spoofed, then the attacker would be unable to finish opening the connection because the spoofed address is not their address. So when the server sends a packet 'back' to them, it would instead be directed to the real owner of the address and not the attacker.

I would make a diagram on Visio for this, but I am in class, so hopefully a sketch is sufficient.

Spoofing the source address is fairly easy, there are still many ISPs that don't implement source address filtering.

Receiving the replies to those spoofed packets is harder. The attacker would need to either get on the network path between client and server or modify routing to change the network path. This is harder but certainly not impossible.

Modern TCP implementations use randomised sequence numbers which make the probability of successfully opening a TCP connection without receiving the reply to the SYN packet very low. Older systems and UDP based protocols may be more vulnerable to such attacks. Use of "SYN cookies" by the server also increases the probability of such an attack suceeding (though it's still a very low probability).

I would consider source IP filtering to be a useful extra line of defense but I would not want to rely on it as the sole means of protection.