To stay or leave PhD after having completed 4 years
Firstly, no apology needed, your question is thorough and easy to read and understand. It sounds like you are in quite an unpleasant situation.
Don't take any advice I may give as gospel, but in answer to your questions:
You are most certainly not foolish to wait until now to stand up to your advisor, you have had several leads and have given many chances for the project to kick into gear. You have every right and reason to feel skeptical about the current promises and project direction.
Ultimately, how you proceed is up to you (you're probably understandably sick of hearing that), but look at the following considerations (no doubt many other members will add to this):
- The new direction could well be a winner, leading to papers, conferences and most of all, fulfillment. It could also be a good one as now, you have made your feelings clear to your supervisor.
- Could this be just another academic 'false positive'?
Perhaps outline a couple of potential papers and present them to your advisor (this is something I do). This could be an ongoing thing, alongside your research - outline potential papers.
As for the timeframe, 17 months - I would not be too worried about that - I have been able to get three papers published in less than ten months, with a 4th on the way and the 5th planned (I finish my Ph.D. at the end of the year).
I hope this helps, and I hope it all gets sorted out for you.
One thing that you need to consider is the way quitting is going to look in your CV. If I was a prospective employer and I saw that you spent 4 years in a PhD program without getting a PhD (or even publications, for the matter), that would be a huge red flag. I'd wonder if you spent those years doodling on facebook and hanging out in cafes. I could even reason that perhaps you are just not as bright as you claim to be. Either way, that's not the kind of person I would want in my company. If you quit now, you should really find a way to preempt this kind of concerns.
With respect to time, I can tell you from experience that 17 months is more than enough provided that (i) you have a clearly defined dissertation topic; (ii) you work hard (and here we are talking about 60-to-70 hours/week; one of the guys in my cohort wrote his entire dissertation in 12 months and his girlfriend complained that, during those 12 months it was almost like she didn't have a boyfriend at all); and (iii) you have a good supporting network of peers and mentors to keep you going in the right direction. I'd say that, at this stage, (i) and (iii) are the most important points. If you can produce a proof-of-concept paper within the next couple of months and a couple of more experienced people agree that it is a worthy project, then you've overcome the largest obstacle.
I just wanted to share my experience with you as I am going through almost the exact same situation.
After a promising two year start and killing my physics classwork and getting my Masters in physics, I picked an adviser and took over a project that a graduate student, who was graduating as I was joining the group, had been working on. Like you, I spent almost all my time coding (a good deal of it CUDA programming) or dealing with certain mathematical problems. I've spent approximately 5-10% of my time on physics and feel that I've done more reproduction of others research, albeit in a more innovative and optimized way, than answering new questions.
I chose to use my time guiding these different projects to learn job-market relevant skills. I use my status as a student to take advantage of school-specific career fairs and professional development. What I've found is that there is quite a demand for physicists (and even more so computational applied mathematicians) out there. It also opened my mind to the types of skills the job market is looking for.
As a result, I've found my anxiety concerning lack of research results has dropped off dramatically! The burnt out feeling I had dissipated considerably as I started seeing that the skills I was learning directly contributed to my future success. I'd highly recommend you start the job search now and try and pick a project that you enjoy that would make you even more attractive to an employer you'd enjoy working for. Also, as for restructuring your relationship, I agree with you that you should indeed take more of a lead in your own research topics. Find projects that force you to learn modern, in-demand techniques and methods, especially those YOU find interesting. It'll help you from feeling burnt out.
Don't worry about quitting the Ph.D. Contrary to other answers, none of the employers I interviewed with cared about me quitting the Ph.D. In fact, they were specifically trying to hire Masters or below. I guess if you have your heart set on academia, then quitting the Ph.D. is an issue. In summary, I'd just say start your job search now and tailor your studies towards employment : it'll help your motivation stay high, produce solid results and allow you to seamlessly hit the job market when you finish!
Another possibility : get an internship. It'll help you get your foot in the door somewhere, give you some much needed professional experience on your resume and a much needed change of scenery. I find that when I take a break and come back to a project, I can hit it all the harder and get over some of the bad humps.