Better to give a poor/unfinished talk at a conference or cancel it?

When this happens to me, I just describe it as "work in progress" or a "research attempt" and present what I can with what I've got. What's wrong with that? That's how research works.

"...hasn't panned out like I hoped it would when I accepted the talk." You can say this during your talk, and explain why it didn't pan out, and ask the audience if they can help. Maybe someone in the audience (your next co-author perhaps?) knows how to proceed, and has the expertise you're lacking. If nobody can help, maybe that's an indication that there is an obstacle (which is important to know).

You've said you're a PhD student, without expertise in the side-topic. Feel free to mention it in your talk "I'm a PhD student in [some subject] at [some university] under [someone]."

You said a post-doc invited you to give a talk (which I expect is not in the sense of an "invited speaker"), who presumably knows you're an early PhD student, and presumably knows what early PhD student talks are sometimes like. My suspicion is that this question is more about inexperience and lack of confidence than an inability to give a reasonable talk due to the work being unfinished.

Perhaps you could try option 3: send your colleague the slide deck of your presentation, or a detailed summary, and ask him whether he thinks it would be "good enough."

Obviously, you wouldn't just drop this on your colleagues lap; you would explain your situation, much like what you have explained here, and see what he thinks. I have no idea whether this is a good idea in your neck of the woods or not; however, I certainly wouldn't hesitate to ask for an opinion before canceling outright or not holding up my end of a perceived bargain.

I would encourage you to reconsider your premise. In many cases it is possible to give an excellent, polished talk on unfinished research. This would be appropriate and welcomed by the audiences at many (not all) conferences.

As others have suggested, I think it is worth contacting the session organizer. If you do proceed, I would spend most of your time talking about the problem you're working on, why it's interesting, and give an example or two. You could then spend a limited amount of time discussing the progress you've made.

Circumstances may vary, but when I've gone to conferences I've often enjoyed talks like this, and gotten a positive impression of the speaker.