Linux CreateProcess?

posix_spawn is probably the preferred solution these days.

Before that fork() and then execXX() was the way to do this (where execXX is one of the exec family of functions, including execl, execlp, execle, execv, execvp, and execvpe). In the GNU C library currently, at least for Linux, posix_spawn is implemented via fork/exec anyway; Linux doesn't have a posix_spawn system call.

You would use fork() (or vfork()) to launch a separate process, which will be a clone of the parent. In both the child and parent process, execution continues, but fork returns a different value in either case allowing you to differentiate. You can then use one of the execXX() functions from within the child process.

Note, however, this problem - text borrowed from one of my blog posts (http://davmac.wordpress.com/2008/11/25/forkexec-is-forked-up/):

There doesn’t seem to be any simple standards-conformant way (or even a generally portable way) to execute another process in parallel and be certain that the exec() call was successful. The problem is, once you’ve fork()d and then successfully exec()d you can’t communicate with the parent process to inform that the exec() was successful. If the exec() fails then you can communicate with the parent (via a signal for instance) but you can’t inform of success – the only way the parent can be sure of exec() success is to wait() for the child process to finish (and check that there is no failure indication) and that of course is not a parallel execution.

i.e. if execXX() succeeds, you no longer have control so can't signal success to the original (parent) process.

A potential solution to this problem, in case it is an issue in your case:

[...] use pipe() to create a pipe, set the output end to be close-on-exec, then fork() (or vfork()), exec(), and write something (perhaps errno) to the pipe if the exec() fails (before calling _exit()). The parent process can read from the pipe and will get an immediate end-of-input if the exec() succeeds, or some data if the exec() failed.

(Note that this solution through is prone to causing priority inversion if the child process runs at a lower priority than the parent, and the parent waits for output from it).

There is also posix_spawn as mentioned above and in other answers, but it doesn't resolve the issue of detecting failure to execute the child executable, since it is often implemented in terms of fork/exec anyway and can return success before the exec() stage fails.

The fork/exec combination was already mentioned, but there is also the posix_spawn family of functions that can be used as a replacement for fork + exec and is a more direct equivalent to CreateProcess. Here is an example for both possibilities:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <unistd.h>
#include <spawn.h>
#include <sys/wait.h>

extern char **environ;

void test_fork_exec(void);
void test_posix_spawn(void);

int main(void) {
  test_fork_exec();
  test_posix_spawn();
  return EXIT_SUCCESS;
}

void test_fork_exec(void) {
  pid_t pid;
  int status;
  puts("Testing fork/exec");
  fflush(NULL);
  pid = fork();
  switch (pid) {
  case -1:
    perror("fork");
    break;
  case 0:
    execl("/bin/ls", "ls", (char *) 0);
    perror("exec");
    break;
  default:
    printf("Child id: %i\n", pid);
    fflush(NULL);
    if (waitpid(pid, &status, 0) != -1) {
      printf("Child exited with status %i\n", status);
    } else {
      perror("waitpid");
    }
    break;
  }
}

void test_posix_spawn(void) {
  pid_t pid;
  char *argv[] = {"ls", (char *) 0};
  int status;
  puts("Testing posix_spawn");
  fflush(NULL);
  status = posix_spawn(&pid, "/bin/ls", NULL, NULL, argv, environ);
  if (status == 0) {
    printf("Child id: %i\n", pid);
    fflush(NULL);
    if (waitpid(pid, &status, 0) != -1) {
      printf("Child exited with status %i\n", status);
    } else {
      perror("waitpid");
    }
  } else {
    printf("posix_spawn: %s\n", strerror(status));
  }
}

You wrote:

I want to create a new proccess in my library without replacing the current executing image. system() blocks the current process, it is not good. I want to continue current process.

Just add an ampersand after the command call. Example: system("/bin/my_prog_name &");

Your process will not be blocked!