How to create an X window which won't close by any way other than a terminal command

If switching to xterm is an option, you could use the hack below. There are a few caveats though. Once you address most of them, the solution ends up quite complicated, see the final script at the end.

xterm -e 'trap "" HUP; your-application'

Upon receiving the instruction to close from the window manager, xterm will send a SIGHUP to the process group of your-application, and only exit itself when the process returns.

That assumes your-application doesn't reset the handler for SIGHUP and could have unwanted side effects for the children of your-application.

Both of which seem to be a problem if your-application is tmux.

To work around those, you could do:

xterm -e sh -c 'bash -mc tmux <&1 & trap "" HUP; wait'

That way, tmux would be started in a different process group, so only the sh would receive the SIGHUP (and ignore it).

Now, that doesn't apply to tmux which resets the handler for those signals anyway, but in the general case, depending on your implementation of sh, the SIGINT, SIGQUIT signals and generally both will be ignored for your-application as that bash is started as an asynchronous command from a non-interactive sh. That means you couldn't interrupt your-application with Ctrl+C or Ctrl+\.

That's a POSIX requirement. Some shells like mksh don't honour it (at least not the current versions), or only in part like dash that does it for for SIGINT but not SIGQUIT. So, if mksh is available, you could do:

xterm -e mksh -c 'bash -mc your-application <&1 & trap "" HUP; wait'

(though that may not work in future versions of mksh if they decide to fix that non-conformance).

Or if you can't guarantee that mksh or bash will be available or would rather not rely on behaviour that may change in the future, you can do their work by hand with perl and for instance write an unclosable-xterm wrapper script like:

#! /bin/sh -
[ "$#" -gt 0 ] || set -- "${SHELL:-/bin/sh}"
exec xterm -e perl -MPOSIX -e '
  $pid = fork;
  if ($pid == 0) {
    setpgrp or die "setpgrp: $!";
    tcsetpgrp(0,getpid) or die "tcsetpgrp: $!";
    exec @ARGV;
    die "exec: $!";
  }
  die "fork: $!" if $pid < 0;
  $SIG{HUP} = "IGNORE";
  waitpid(-1,WUNTRACED)' "$@"

(to be called as unclosable-xterm your-application and its args).

Now, another side effect is that that new process group we're creating and putting in foreground (with bash -m or setpgrp+tcsetpgrp above) is no longer the session leader process group, so no longer an orphaned process group (there's a parent supposedly caring for it now (sh or perl)).

What that means is that upon pressing Ctrl+Z, that process will be suspended. Here, our careless parent will just exit, which means the process group will get a SIGHUP (and hopefully die).

To avoid it, we could just ignore the SIGTSTP in the child process, but then if your-application is an interactive shell, for some implementations like mksh, yash or rc, Ctrl-Z won't work either for the jobs they run.

Or we could implement a more careful parent that resumes the child each time it's stopped, like:

#! /bin/sh -
[ "$#" -gt 0 ] || set -- "${SHELL:-/bin/sh}"
exec xterm -e perl -MPOSIX -e '
  $pid = fork;
  if ($pid == 0) {
    setpgrp or die "setpgrp: $!";
    tcsetpgrp(0,getpid) or die "tcsetpgrp: $!";
    exec @ARGV;
    die "exec: $!";
  }
  die "fork: $!" if $pid < 0;
  $SIG{HUP} = "IGNORE";
  while (waitpid(-1,WUNTRACED) > 0 && WIFSTOPPED(${^CHILD_ERROR_NATIVE})) {
    kill "CONT", -$pid;
  }' "$@"

Another issue is that if xterm is gone for another reason than the close from the window manager, for example if xterm is killed or loses the connection to the X server (because of xkill, the destroy action of you Window manager, or the X server crashes for instance), then those processes won't die as SIGHUP would also be used in those cases to terminate them. To work around that, you could use poll() on the terminal device (which would be torn down when xterm goes):

#! /bin/sh -
[ "$#" -gt 0 ] || set -- "${SHELL:-/bin/sh}"
exec xterm -e perl -w -MPOSIX -MIO::Poll -e '
  $pid = fork; # start the command in a child process
  if ($pid == 0) {
    setpgrp or die "setpgrp: $!"; # new process group
    tcsetpgrp(0,getpid) or die "tcsetpgrp: $!"; # in foreground
    exec @ARGV;
    die "exec: $!";
  }
  die "fork: $!" if $pid < 0;
  $SIG{HUP} = "IGNORE"; # ignore SIGHUP in the parent
  $SIG{CHLD} = sub {
    if (waitpid(-1,WUNTRACED) == $pid) {
      if (WIFSTOPPED(${^CHILD_ERROR_NATIVE})) {
        # resume the process when stopped
        # we may want to do that only for SIGTSTP though
        kill "CONT", -$pid;
      } else {
        # exit when the process dies
        exit;
      }
    }
  };

  # watch for terminal hang-up
  $p = IO::Poll->new;
  $p->mask(STDIN, POLLERR);
  while ($p->poll <= 0 || $p->events(STDIN) & POLLHUP == 0) {};
  kill "HUP", -$pid;
  ' "$@"

Your question has a somewhat misleading title, leading your research in the wrong direction. When you close a window in X, the server doesn't kill your process. Instead, it sends a WM_DELETE_WINDOW message to the process, which in turn terminates. This fact dooms your first approach: even if you set the SIGNAL_UNKILLABLE flag, you'll still be able to close it.

Your second approach seems usable, all you have to do is to discriminate a single terminal window against all others. One way to achieve that is to start urxvt with a particular title, for example urxvt -name urxvt-iddqd, and create a rule for its name:

<keybind key="A-F4">
 <action name="If">
  <application title="urxvt-iddqd"
    title="tmux-session">
  </application>
  <then><!-- Do nothing for urxvt-tmux-session --></then>
  <else>
   <action name="Close" />
  </else>
 </action>
</keybind>