How can I create a simple index.html file which lists all files/directories?

You can either: Write a server-side script page like PHP, JSP, ASP.net etc to generate this HTML dynamically

or

Setup the web-server that you are using (e.g. Apache) to do exactly that automatically for directories that doesn't contain welcome-page (e.g. index.html)

Specifically in apache read more here: Edit the httpd.conf: http://justlinux.com/forum/showthread.php?s=&postid=502789#post502789 (updated link: https://forums.justlinux.com/showthread.php?94230-Make-apache-list-directory-contents&highlight=502789)

or add the autoindex mod: http://httpd.apache.org/docs/current/mod/mod_autoindex.html

There are enough valid reasons to explicitly disable automatic directory indexes in apache or other web servers. Or, for example, you might want to only include certain file types in the index. In such cases you might still want to have a statically generated index.html file for specific folders.

tree

tree is a minimalistic utility that is available on most unix-like systems (ubuntu/debian: sudo apt install tree, mac: brew install tree, windows: zip). tree can generate plain text, XML, JSON or HTML output.

Generate an HTML directory index one level deep:

tree -H '.' -L 1 --noreport --charset utf-8 -o index.html

Only include specific file types that match a glob pattern, e.g. *.zip files:

tree -H '.' -L 1 --noreport --charset utf-8 -P "*.zip" -o index.html

^{The argument to -H is what will be used as a base href, so you can pass either a relative path such as . or an absolute path from the web root, such as /files. -L 1 limits the listing to the current directory only.}

tree does not expose a flag to disable the credits in the HTML footer, but you can cut out that block by piping through sed:

# delete 7 lines starting with the line matching <hr>
tree -H '.' -L 1 --noreport --charset utf-8 | sed -e '/<hr>/,+7d' > index.html

For all supported options see tree --help or man tree in a shell.

Generator script with recursive traversal

I needed an index generator which I could style the way I want, and which would also include the file sizes, so ended up writing this script (python 3) which in addition to having customisable styling can also recursively generate an index.html file in all the nested subdirectories (with the --recursive or -r flag). The styling borrows heavily from caddyserver's file-server module. It includes last modified time and is responsive in mobile viewports.

For me PHP is the easiest way to do it:

<?php
echo "Here are our files";
$path = ".";
$dh = opendir($path);
$i=1;
while (($file = readdir($dh)) !== false) {
    if($file != "." && $file != ".." && $file != "index.php" && $file != ".htaccess" && $file != "error_log" && $file != "cgi-bin") {
        echo "<a href='$path/$file'>$file</a><br /><br />";
        $i++;
    }
}
closedir($dh);
?> 

Place this in your directory and set where you want it to search on the $path. The first if statement will hide your php file and .htaccess and the error log. It will then display the output with a link. This is very simple code and easy to edit.