Drupal - Which hook to implement to limit node creation?

Alter the access callback for the menu callback:

function custom_module_menu_alter(&$items){
    $items['node/add/page']['access callback'] = 'my_custom_access_callback';
    // Next line needed for Drupal 7
    unset($items['node/add/page']['access arguments']);
}

Add a function as your custom access callback:

function my_custom_access_callback() {
   global $user;
   $allowed_limit = variable_get('allowed_limit', 10);

   // If Drupal 6.
   $node_counts = db_result(db_query("SELECT count(*) FROM {node} WHERE uid = %d;", $user->uid));
   // End If

   // If Drupal 7.
   $node_counts = db_query("SELECT count(*) FROM {node} WHERE uid = :uid", array(':uid' => $user->uid))->fetchField();
   // End If

   if ($node_counts < $allowed_limit && user_access("create page")) {
     return TRUE;
   }
   else{
     drupal_set_message(t('Print message about exceeded limit and/or user permissions.'));
     return FALSE;
   }
}

This kind of logic will restrict a user to exceed the limit.

The Node Limit project now has a Drupal 7 port as an Alpha release.

The Node Limit module allows administrators to restrict the number of nodes of a specific type that roles or users may create. For example, if a site has an "Advertiser" role that can create "advertisement" nodes, then the node limit administrator can restrict all users in that role to a specific number of nodes. He may also restrict users on a per-user basis.

There is a plan to include Rules support in the near future as well. The documentation could use some work but if you poke through the module it will likely accomplish what you need.

Hook node validate may be more appropriate here. http://api.drupal.org/api/drupal/modules--node--node.api.php/function/hook_node_validate/7

That way if you use Services the solution will work, while the one proposed by Shoaib will only work if a user accesses the actual node creation page. If a remote client calls node.save via Services the menu access hack will fail.