How to calculate GOP size of a file H264

Well, just parsing the bitstream to find the each I-frame is a bit tricky; among other things the encode order might be (or not) different from the display-order. One solution is to use http://www.ffmpeg.org/ffprobe.html from the ffmpeg-suite.

Example:

ffprobe -show_frames input.bin | grep key_frame
key_frame=1
key_frame=0
key_frame=0
key_frame=0
key_frame=0
key_frame=0
...

from the output you can easily calculate the GOP-length

Another solution is to patch the reference implementation found at http://iphome.hhi.de/suehring/tml/

Let me know if you need help with this part :-)

I personally prefer filtering by pict_type:

ffprobe -show_frames input.h264 | grep pict_type

This will show you the frame structure:

pict_type=I
pict_type=P
pict_type=P
pict_type=P
pict_type=P
pict_type=P
...