Can you update the web.config file for an Azure web app without redeploying?

You can use the portal, there is a tool called "App Service Editor" in preview that lets you edit any of the files you've deployed. I do wonder why you want to do this though, it's not considered good practice to modify source files on the fly like this! Config and app settings are exposed via the portal as well and can be modified without dropping to the app service editor tool. (under Settings/Application Settings in portal). Updating these does not update the web.config but will override web.config settings.

As Russell Young said, on Azure portal, we could use App Service Editor that provides an in-browser editing experience for our App code. And we could specify connection string in App settings section to override existing settings.

Besides, we could also to access and update Web.config file (under D:\home\site\wwwroot folder) via Kudu Debug console.