How to optimize gradle build performance regarding build duration and RAM usage?

You need to give more memory to the Gradle JVM, not to the compile task/JVM. One way to do so is via the GRADLE_OPTS environment variable (GRADLE_OPTS=-Xmx512m).

If using the Gradle Wrapper you can set DEFAULT_JVM_OPTS in gradlew like this:

DEFAULT_JVM_OPTS="-Xmx512m"

Set it in a similar fashion in gradlew.bat if you're on Windows:

set DEFAULT_JVM_OPTS=-Xmx512m

The Gradle Wrapper task can also be modified to include this automatically. This is how the Gradle developers have solved it:

wrapper {
    gradleVersion = '1.8'

    def jvmOpts = "-Xmx512m"
    inputs.property("jvmOpts", jvmOpts)
    doLast {
        def optsEnvVar = "DEFAULT_JVM_OPTS"
        scriptFile.write scriptFile.text.replace("$optsEnvVar=\"\"", "$optsEnvVar=\"$jvmOpts\"")
        batchScript.write batchScript.text.replace("set $optsEnvVar=", "set $optsEnvVar=$jvmOpts")
    }
}

I just found a very nice way to handle this problem. No need for custom gradle wrapper or GRADLE_OPTIONS.

compileJava {
    options.fork = true  // Fork your compilation into a child process
    options.forkOptions.setMemoryMaximumSize("4g") // Set maximum memory to 4g
}

Run Gradle with the --info option to see where it's going to use your parameter for max memory size.

gradle build --info

I am using following version of gradle.properties to make Gradle performance better in Android projects

# The Gradle daemon aims to improve the startup and execution time of Gradle.
# When set to true the Gradle daemon is to run the build.
org.gradle.daemon=true

# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
# Default value: -Xmx10248m -XX:MaxPermSize=256m
org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8

# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
org.gradle.parallel=true

# Enables new incubating mode that makes Gradle selective when configuring projects.
# Only relevant projects are configured which results in faster builds for large multi-projects.
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:configuration_on_demand
org.gradle.configureondemand=true