Why is cv::resize so slow?

Resizing an image is slow, and you are doing it twice for each processed frame. There are several ways to somehow improve your solution but you have to provide more details about the problem you are trying to solve.

To begin with, resizing an image before detecting edges will result in feeding the edge detection with less information so it will result in less edges being detected - or at least it will make it harder to detect them.

Also the resizing algorithm used affects its speed, CV_INTER_LINEAR is the fastest for cv::resize if my memory does not fail - and you are using CV_INTER_CUBIC for the first resize.

One alternative to resize an image is to instead process a smaller region of the original image. To that you should take a look at opencv image ROI's (region of interest). It is quite easy to do, you have lots of questions in this site regarding those. The downside is that you will be only detecting edges in a region and not for the whole image, that might be fine, depending on the problem.

If you really want to resize the images, opencv developers usually use the pyrDown and pyrUp functions when they want to process smaller images, instead of resize. I think it is because it is faster, but you can test it to be sure. More information about pyrDown and pyrUp in this link.

About cv::resize algorithms, here is the list:

INTER_NEAREST - a nearest-neighbor interpolation
INTER_LINEAR - a bilinear interpolation (used by default)
INTER_AREA - resampling using pixel area relation. It may be a preferred method for image decimation, as it gives moire’-free results. But when the image is zoomed, it is similar to the INTER_NEAREST method.
INTER_CUBIC - a bicubic interpolation over 4x4 pixel neighborhood
INTER_LANCZOS4 - a Lanczos interpolation over 8x8 pixel neighborhood

Can't say for sure if INTER_LINEAR is the fastest of them all but it is for sure faster than INTER_CUBIC.

INTER_NEAREST is the fastest and has the worst quality results. In the downscale, for each pixel, it just uses the pixel nearest to the hypothetical place.

INTER_LINEAR is a good compromise of performance and quality, but it is slower than INTER_NEAREST.

INTER_CUBIC is slower than INTER_LINEAR because it uses more interpolation.

INTER_LANCZOS4 is the algorithm with the best quality result, but it is slower than the others.

Here you can find a good comparison article: http://tanbakuchi.com/posts/comparison-of-openv-interpolation-algorithms/