How do i use Composer to install a package without a version (only master)

It's simple! You should type on console:

composer require blablabla@thepackage dev-master

You have to specify the version name in the composer.json file. Just find any package which are installable with Composer at : https://packagist.org/

Search the package name there, you will find the version name. And here is the link for the package you wanted : https://packagist.org/packages/neitanod/forceutf8

In composer.json:

{
    "name": "example/example-app",
    "repositories": [
      {
        "type": "vcs",
        "url": "https://github.com/neitanod/forceutf8"
      }
    ],
    "require": 
      {
        "neitanod/forceutf8": "dev-master",
        "raven/raven": "0.7.1",
        "monolog/monolog": "1.7.*",
        "smarty/smarty": "3.1.16"
      }
}

First of all, you have the wrong name: forceutf8/forceutf8 Correct name is: neitanod/forceutf8

Composer adds a prefix of "dev-" to every branch name that is not looking like a version number, and a suffix "-dev" to branch names that look like version numbers.

Example: Branch "master" is called "dev-master", branch "feature" is called "dev-feature". Branch "1.0.x" is called "1.0.x-dev".

So this part is wrong:

"require": {
    "raven/raven": "0.7.1",
    "monolog/monolog": "1.7.*",
    "smarty/smarty": "3.1.16",
    "forceutf8/forceutf8": "master"
}

Correct version would be:

"require": {
    "raven/raven": "0.7.1",
    "monolog/monolog": "1.7.*",
    "smarty/smarty": "3.1.16",
    "neitanod/forceutf8": "dev-master"
}

Now requiring branches without tagged versions is not the best thing because these info is quite unstable - any new commit can break things, and it is not that easy to point exactly at the commit you wanted to use. To protect you agains this, Composer by default does not load these development branch, but will only load stable versions. You have to enable loading development versions:

"require": {
    "raven/raven": "0.7.1",
    "monolog/monolog": "1.7.*",
    "smarty/smarty": "3.1.16",
    "neitanod/forceutf8": "dev-master@dev"
}

The "@dev" flag allows to load development versions of the version mentioned (which in this case is a branch, but "1.0.0@dev" would work the same way, allowing all (including dev) versions of 1.0.0, like "1.0.0-alpha", but also the stable "1.0.0").

Note that you could also allow development version for ALL your dependencies by using "minimum-stability", but this is not recommended, because it will load development versions for EVERYTHING according to the version requirement. In your case, you would grab the latest dev version of monolog's 1.7 branch only, but this might be enough to bring a once stable software into a broken state.