Sharepoint - Is there a way to limit SharePoint Online usage to users (or a group of users) to the offfice premises ( Physical location ) only?

Open up the Admin Centre for SharePoint Online.

In the left navigation, click device access.

In the section for Control access based on network location, select Only allow access from specific IP address locations

Enter the IP address or address range/s for your corporate networks.

This will prevent users outside of these ranges from accessing SharePoint online.

Open Admin Center of your SharePoint Online.

In the left navigation, click on "Access Control".

In the section for Control access based on network location, select Only allow access from specific IP address locations

Enter the IP address or address range/s for your corporate networks.

Please find the below Microsoft article for your reference :

https://support.office.com/en-gb/article/control-access-to-sharepoint-online-and-onedrive-data-based-on-network-location-b5a5f1f1-1174-4c6b-91d0-9273a6b6971f

If you have the applicable enterprise licence agreement you can do this using Azure AD Conditional Access in the Azure Portal.

https://docs.microsoft.com/en-us/azure/active-directory/active-directory-conditional-access-azure-portal

This approach might be preferable because you can provide more granular access instead of a block at IP address level. In future when your pointy haired boss learns the true value of SharePoint Online for external access you may be able to lift the policy gently if you've implemented conditional access.