Drupal - How do I hide nodes that shouldn't be directly accessed from users and search engines?

The Rabbit Hole module provides this feature.

Rabbit Hole is a module that adds the ability to control what should happen when an entity is being viewed at its own page.

Perhaps you have a content type that never should be displayed on its own page, like an image content type that's displayed in a carousel. Rabbit Hole can prevent this node from being accessible on its own page, through node/xxx.

This sounds to me like a good use case for the Panels module because you can create panels that will override node pages, and can set the context for the panel in a way should guarantee users see the page you want them to see it as well as access rules for panel pages if you need them. For further details, see this post on d.o.

One option may be to never publish the node but have the consumer page ignore their publication status. So in your example, your "About" view may just ignore the publication status of your "About-xxx" nodes. Since the nodes are not published, un-authorized user cannot access them and they shouldn't be indexed by the search engine.

Another solution is to use an hook_nodeapi('view')/hook_node_view() implementation to issue a drupal_goto() or drupal_access_denied() when the node page is visited by an un-authorized user. Be aware that hook_nodeapi()/hook_node_view() are used in many cases and not only when viewing a node page.

hook_menu_alter() can also be used to alter the access callback of the node pages to deny access to the hidden nodes.

The best is probably not to base the filtering on hard-coded node ID but to use either a custom field on the node (using CCK/Field API or a custom table) or a list of hidden nodes stored in a variable.