Is it possible to test list size from a VisualForce page?

This is just a syntax issue, your condition will work if you make a few small changes

Change

 <!-- doesn't work, this is apex, not a formula as required -->
    <apex:pageBlockTable ... rendered="contacts.size()>0">

To

 <apex:pageBlockTable ... rendered="{!contacts.size>0}">

You can do this as:

<apex:pageBlockTable ... rendered="{!IF(contacts.size>0,'true','false')">

On visualforce page we can check conditions by:{!IF(condition,InTrueCondition,IfConditionIsFalse)}

Same as List size can be checked as:

<apex:pageBlockTable value="{!contacts}" var="ct" rendered="{!IF(contacts.size>0,'true','false')">
          <apex:column value="{!ct.name}"/>
          <apex:column value="{!ct.phone}"/>
      </apex:pageBlockTable>
      <apex:pageMessage summary="Account has no contacts" severity="info" rendered="{!IF(contacts.size>0,'false','true')" />

So by {!IF()} you can check various conditions and display results according to it. Also on VF page {!contacts.size>0} will result to true when rendered. So rendered="{!contacts.size>0}" can be used in your case.

Reference: read here

Edit: Also instead of using rendered="{!contacts.size>0}" or using {!IF()}, function {!listInstance.empty} can be used.

So in case for a empty list {!contacts.empty} will render true and {!!contacts.empty} will render false