Can FFmpeg be used as a library, instead of a standalone program?

If you just wanted to make a call to ffmpeg as function rather than a system call, you can do that pretty easily.

in ffmpeg.c, change:

int main(int argc, char **argv) to int ffmpeg((int argc, char **argv)

Then in your call the ffmpeg function and pass in an array that mimics the command line. To make it even easier use a function to create the argc, argv variables.

static int setargs(char *args, char **argv)
{
    int count = 0;

    while (isspace(*args)) ++args;
    while (*args) {
        if (argv) argv[count] = args;
        while (*args && !isspace(*args)) ++args;
        if (argv && *args) *args++ = '\0';
        while (isspace(*args)) ++args;
        count++;
    }
}

char **parsedargs(char *args, int *argc)
{
    char **argv = NULL;
    int    argn = 0;

    if (args && *args
        && (args = strdup(args))
        && (argn = setargs(args,NULL))
        && (argv = malloc((argn+1) * sizeof(char *)))) {
          *argv++ = args;
          argn = setargs(args,argv);
    }

    if (args && !argv) free(args);

    *argc = argn;
    return argv;
}

void freeparsedargs(char **argv)
{
    if (argv) {
        free(argv[-1]);
        free(argv-1);
    }
}
    return count;
}

int main()
{
    char **argv;
    char *cmd;
    int argc;

    cmd = "ffmpeg -i infile outfile";
    argv = parsedargs(cmd,&argc);
    ffmpeg(argc, argv);
}

You need libavcodec and libavformat. The FAQ tells you:

4.1 Are there examples illustrating how to use the FFmpeg libraries, particularly libavcodec and libavformat?

Yes. Read the Developers Guide of the FFmpeg documentation. Alternatively, examine the source code for one of the many open source projects that already incorporate FFmpeg at (projects.html).

The FFmpeg documentation guide can be found at ffmpeg.org/documentation.html, including the Developer's guide. I suggest looking at libavformat/output-example.c or perhaps the source of the ffmpeg command line utility itself.