Drupal - How to completely turn off caching?

You can replace the default cache backend to one which does not actually do anything. Simply add this to your settings.php file:

if (!class_exists('DrupalFakeCache')) {
  $conf['cache_backends'][] = 'includes/cache-install.inc';
}
// Default to throwing away cache data.
$conf['cache_default_class'] = 'DrupalFakeCache';
// Rely on the DB cache for form caching - otherwise forms fail.
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

Just to be sure, set the cache expiration time to none on the Performance page.

I actually use this in a settings.local.php file which, if it exists, gets included at the bottom of my settings.php file so I can sync settings.php from the live or staging environments to my dev environment without having to worry about using the wrong database settings, accidentally pushing dev settings to live, etc.

settings.php:

if (file_exists(__DIR__ . '/settings.local.php')) {
  require_once __DIR__ . '/settings.local.php';
}

Theme functions are handled by the theme registry: rebuilding (or worse, disabling) all your site's caches to pick up a new theme function is unnecessary and overkill.

What you can do is use the Devel module to rebuild your theme's registry on every page reload. Once installed, go to Configuration → Development → Devel settings. There, check "Rebuild the theme registry on every page load" and press the "Save configuration" button. Remember to uncheck it (or better yet, disable Devel entirely) when you're done with development.

You can't turn of all caching completely. And if you actually did, it would slow down your progress so much more. Consider the time it takes it actually clear the cache, that's what every single page load would be like.

There are so many hooks running for building information about the available entities, fields, forms, implemented hooks etc that only need to execute once currently, you don't want to prevent that.

As you get used to Drupal, you will soon learn what things that require a cache clear, and probably get friendly with either the admin menus shortcut or drush cc all, and this will no longer be an issue.