Drupal - file_scan_directory() takes about 10 seconds to execute

It sounds like you're affected by a known issue in Drupal 7.

Most probably, you are hitting Avoid re-scanning module directory when multiple modules are missing. It happens if you have some missing modules in your installation. Try to check your system table:

SELECT name, filename FROM system WHERE type = 'module' AND status = 1 ORDER BY filename

And clean-up any modules that are still enabled but missing from the filesystem.

Overall, Drupal 7 is way more resource friendly and scalable then Drupal 6, other than some unfortunate regressions like this one.

Looking at those functions, it does look like it's missing a module or maybe a single file of a module. Have a look at drupal_get_filename(), it does call the drupal_system_listing(), which calls this function if it can't find the requested file. Add a dpm(func_get_args()) right before it calls drupal_system_listing(), that should tell you which file it isn't finding.

There are several reasons why this issue may arise, and to my great dismay, I now find myself somewhat knowledgeable about those reasons. Frustratingly, if you've just noticed this problem after upgrading Drupal core to 7.33+, this could be a typo in any module, even if you haven't upgraded that module.

Modules removed from code base

You may wish to first check for the known bug that @Berdir mentions, especially if you've recently been removing "unused" modules from the code base. To find out if you have modules that are enabled but have been removed from the file system you can run a script such as the one mentioned here - or use mine, written for a multi-site install on a system with drush, to be run from the Drupal base directory:

find sites -maxdepth 1 -iname '*.*' -type d | sed -rne 's:sites/(.+):echo \1; drush @\1 sqlq "select filename from system where status = 1" | grep "/" | sed -rne "s_(.+)_test -f \\1 || echo \\1_p" | bash:p' | bash

or the following:

while read -r file; do [ -f "$file" ] || echo "$file is missing."; done < <(drush sqlq "SELECT filename FROM system WHERE status = 1")

If you find a module that has been removed from the code base, follow the directions in the issues that @Berdir mentioned.

Coding errors

If it's not that, your situation is likely caused by a coding error, such as a file that has been removed but is still being added by a drupal_add_js call (from comment 19 in issue #1082892) or an unfortunate typo in a module or theme, e.g. imagecache_actions (see https://drupal.org/node/2381357).

In any case, to figure out exactly why this is happening, you need to know exactly which file Drupal can't find. So, as per Berdir's comment, you can temporarily hack drupal_get_filename in bootstrap.inc by adding a log or message call just before the call to drupal_system_listing(). If you have Devel module installed then dpm will work; if not, you can use drupal_set_message or syslog. Examples:

dpm(func_get_args());
drupal_set_message(implode(', ', func_get_args()));
syslog(LOG_WARNING, implode(', ', func_get_args()));

Once you know what Drupal is looking for, it's a good bet that you'll be able to figure out where to go from there. My problem was caused by a call to include a file from the non-existent module imagcache_actions (note the typo). So, I searched for imagecache_actions in my codebase (e.g. grep -r imagcache_actions .), and found that version 1.4 of imagecache_canvasactions.module uses module_load_include outside of any function call, in the file scope, with a typo. Again, this error was only exposed after updating to Drupal 7.33+. I found that an issue had already been created for imagecache_actions, applied the patch, and was back in business.

I had a very similar issue - file_scan_directory() was killing the site. Turns out a huuge node_modules folder embedded within my custom theme for gulp was being scanned each cache flush. Moving these files out of the theme folder (and updating some paths in my gulpfile) seemed to fix it for me. Alternatively: I think you can hack file.inc:

'nomask' => '/(\.\.?|CVS|node_modules)$/', // https://www.drupal.org/node/2329453#comment-9360519