Can a transformer work if the core is not circular?
The core doesn't need to be circular, but it must be closed, otherwise the linked flux will be very low.
Moreover, the fact that the pipe is empty doesn't improve the situation, since the flux is concentrated where there is higher permeability, i.e. in the core, but the net section of the core in your case is small. In fact most of the section of the coil is filled with air, which has poor permeability.
You cannot close the core with a simple piece of iron wire. It won't be effective, since the flux will be constrained in the smaller section of the wire. Keep in mind that flux obeys a sort of "Ohm's law for magnetic circuits", called Hopkinson's law.
The role of resistance is taken by a quantity known as reluctance, which is proportional to the net section of the core where the flux flows. The flux is analogous to current. Therefore a tiny section will limit the flux greatly. Since the role of the voltage is taken by the magnetomotive force (MMF) which depends on the current in the coil, you can understand that with the same current in the primary and a high reluctance due to a flux constrained in a little section of wire, the flux will be small, and hence the induced current in the secondary will be small.
If you try to pump more current in the primary, the result will be that the core will saturate (a strongly non-linear effect), with the consequence that its permeability will drop drastically, voiding your attempt.
To have enough coupling between the two coils you need a closed magnetic circuit with substantially low reluctance. Therefore you need a closed path made of ferromagnetic material with a more or less constant section, since any narrowing in the section will increase the reluctance.
EDIT (prompted by a useful comment by @Asmyldof)
Although, I explained above why your setup is not efficient for a power transformer, and the explanation still stands, there are a couple of issues to be aware of when dealing with transformer operation. This interesting article on transformers has nice pictures and delves into the subject in more detail. I'll point out briefly two key aspects below.
As I said, to be able to have high coupling between primary and secondary winding you need low reluctance and a closed core. This calls for a solid core with a closed magnetic path. Relative to your setup, this will improve the situation, but be aware that using a ferromagnetic core which is also electrically conducting, as iron is, has its drawbacks.
First (and really important for a power transformer) there are core power losses. If the core is made of a good conducting material, eddy currents will be induced in its cross section and this will cause power loss by Joule heating (as in a resistor). This is not the only source of core losses, but for conductive cores it's the most relevant usually. Therefore using a solid iron bar as transformer core you risk to lose much power heating the core itself (that's why cores made of iron are not solid, they are still "filled", but laminated, i.e. made by many layers of insulated material).
The second key aspect is saturation. If you increase the primary current over a certain limit the core will saturate and the permeability will drop, hence the reluctance will rise. Having a not-completely-closed-loop core is, in this case, beneficial. In fact sometimes cores are built with a small air-gap, i.e. the core forms an almost-closed loop, but not quite. The small air gap has much higher reluctance than the rest of the core, hence it increases the overall reluctance of core+gap, which seems bad, but the advantage is that the gap helps linearizing the core, i.e. limits the effect of saturation. Moreover, the gap is very small (say about the thickness a sheet of paper) and this prevents the flux from dispersing in the space around the core, hence it doesn't worsen the overall coupling too much.
Other interesting links about transformers:
Transformers Basics
Transformers Construction
It will "work" in a sense, just as any other transformer, but as the flux circuit is only closed by magnetic field leakage from one end of the core to the other, its reluctance will be enormous, and therefore it'll be much less efficient than you would like. This is normally modelled as a "leakage inductance".
Measure the primary inductance with the secondary open-circuit. This is called the primary inductance. Re-measure with the secondary short-circuit and you should see the primary inductance reduce slightly, as you have placed the "leakage inductance" in parallel with it. Calculating the leakage inductance will allow you to calculate the loss it your transformer.
In a good transformer the leakage inductance is 1% or less of the primary inductance : in yours it is probably 10x the primary inductance or more.
Actually if you look at the ferrite rod antenna in an AM radio, you'll see several windings; it acts both as antenna, tuned circuit, and transformer. The smallest winding transfers a small proportion of the energy from the tuned circuit into the RF amplifier and mixer.
But it's not an effective transformer for power conversion.
You can improve it by bending the rod into an "U" or better, round into a ring with a gap, then the flux simply has to jump the gap, giving lower reluctance. As you reduce the gap width the reluctance decreases, and so does the leakage inductance, increasing the efficiency of the transformer.
Best of all is to close the gap entirely
However, sometimes a small gap is left (set by the thickness of a piece of paper!) deliberately, to keep the flux density down to avoid saturating the core. This is usually done in signal transformers, where distortion from saturation is a problem, not in power conversion transformers.
No, the magnetic material doesn't need to form a closed loop, but that will allow you to make a smaller transformer for the same amount of power. The magnetic field lines will always be in a loop, the only question is whether you provide nice material for them to easily follow or not.
However, the problem in your case is that you are using a conductive core. The metal pipe acts as a short-circuit secondary, giving your secondary winding little chance to pick up much of anything. You've built a induction heater, not a transformer.
Also, you are putting AC into the primary, right? Transformers work only on AC. It's the change of the magnetic field that induces a voltage across the secondary.