How to make longtable respect \nopagebreak in the beginning?

Use the needspace package and put \Needspace{5\baselineskip} before the section. (The exact value depends on the class and the spaces the section inserts.)

EDIT: as pointed out in a comment below, this solution sometimes produces spurious headers; I do not know how (nor have time) to fix it, sorry.

There are two problems: first, \longtable does the calculations of whether it has enough space on the current page before starting to read the material of the table. Secondly, it inserts a penalty of \z@ (read "zero"), in other words, a place where TeX is allowed to do a page break. We need to inhibit both of those, but in case there is more than one longtable in your book, the change must not be permanent.

Putting the change within a group does not seem possible (because \longtable redefines the output routine locally, I think), so I added a private switch to control whether a penalty and a \break are inserted or not. Put \normalLTfalse before and \normalLTtrue after the longtable (not within) to inhibit the page-break and penalty for that table. (Note that commenting our the line \normalLTfalse will lead to a page break there.)

It may be possible to use the "value" of \if@nobreak instead of explicitly having to use a switch, but I am not sure when \if@nobreak is \iftrue or \iffalse.

\documentclass{article}
\usepackage{longtable}
\usepackage{etoolbox}

\newif\ifnormalLT
\normalLTtrue

\makeatletter
\patchcmd {\LT@start}
          {\vfil \break}
          {\ifnormalLT \vfil \break \fi}
          {\typeout{Patching longtable succeeded!}}
          {\typeout{Patching longtable failed!}\ERROR}
\patchcmd {\LT@start}
          {\penalty \z@}
          {\ifnormalLT \penalty \z@ \fi}
          {\typeout{Patching longtable succeeded!}}
          {\typeout{Patching longtable failed!}\ERROR}
\makeatother

% "lipsum" is just used to have some dummy text.
% The vspace is there so that without patch
% we would have a bad page break. Remove all that.
\usepackage{lipsum}
\begin{document}
\lipsum[1-2]
\vspace{2.5cm}
\lipsum[3-4]

\section{XY}
\normalLTfalse
\begin{longtable}{c}
A \\
C \\
\end{longtable}
\normalLTtrue

\end{document}