QGIS, interpolate along an irregular shape or line to create a DEM

GRASS can do what you're asking.

Here's how I would approach the mssion:

First get or digitize the banks of the river as a (long, winding) polygon. Then import your xyz data as a point vector (v.in.ascii), and your river polygon in the appropriate GRASS location/mapset with v.in.ogr.

Now you'll have to think about the region settings with g.region . You need to set a resolution that relects how accurate you want the bathymetric layer to be. And be sure that the region extents cover the whole stretch of the river that you are working with.

Now convert your river polygon to a raster layer (v.to.rast), and use r.mask to make it into a mask layer so that the next interolation step works only on the river area.

And now choose which of the several interpolation modules suits your needs. The basic v.surf.idw implements "Inverse Distance Weighting" which usually has the resulting surface going exactly thru the original points. On the other hand v.surf.rst (Regularized Spline with Tension) does smoothing of the measured points, and is often better for terrain interpolation.

You might like to check out one of the presentations at FOSS4GIS 2014 titled " Open Source Work-flow for Surface Interpolation with Curvilinear Anisotropy — Michele Tobias, University of California Davis."

http://vimeo.com/106235881