How to expose and serve WOFF file from Azure without web.config?

I don't have a config in my deployment

You can just add a web.config file at the root folder of your app with the following contents.

<configuration>
  <system.webServer>
    <staticContent>
      <mimeMap fileExtension="woff" mimeType="application/font-woff" />
      <mimeMap fileExtension="woff2" mimeType="application/font-woff2" /> 
    </staticContent>
  </system.webServer>
</configuration>

You can install an extension for your Azure App Service:

https://github.com/johnnyqian/enable-font-awesome-site-extension