NGINX read body from proxy_pass response

I assume you are software developer and your have full control over your application so there is no need to force square peg in a round hole here.

Different kinds of reverse proxies support ESI(Edge Side Includes) technology which allow developer to replace different parts of responce body with content of static files or with response bodies from upstream servers.

Nginx has such technology as well. It is called SSI (Server Side Includes).

location /file {
    ssi on;
    proxy_pass http://go.example.com;
}

Your upstream server can produce body with content <!--# include file="/path-to-static-files/some-static-file.ext" --> and nginx will replace this in-body directive with content of the file.

But you mentioned streaming...

It means that files will be of arbitrary sizes and building response with SSI would certainly eat precious RAM resources so we need a Plan #B.

There is "good enough" method to feed big files to the clients without showing static location of the file to the client. You can use nginx's error handler to server static files based on information supplied by upstream server. Upstream server for example can send back redirect 302 with Location header field containing real file path to the file. This response does not reach the client and is feed into error handler.

Here is an example of config:

location /file {
    error_page 302 = @service_static_file;
    proxy_intercept_errors on;
    proxy_set_header Host            $host;
    proxy_pass http://go.example.com;
}

location @service_static_file {
    root /hidden-files;
    try_files $upstream_http_location 404.html;
}

With this method you will be able to serve files without over-loading your system while having control over whom do you give the file.

For this to work your upstream server should respond with status 302 and with typical "Location:" field and nginx will use location content to find the file in the "new" root for static files.

The reason for this method to be of "good enough" type (instead of perfect) because it does not support partial requests (i.e. Range: bytes ...)

Looks like you are wanting to make an api call for data to run decision and logic against. That's not quite what proxying is about.

The core proxy ability of nginx is not designed for what you are looking to do.

Possible workaround: extending nginx...

---

Nginx + PHP

Your php code would do the leg work.
Serve as a client to connect to the Golang server and apply additional logic to the response.

<?php
    $response = file_get_contents('https://go.example.com/getpath?file_id='.$_GET["id"]);
    preg_match_all("/filePath: \"(.*?)\"/", $response, $filePath);
    readfile($filePath[1][0]);
?>

    location /getpath {
        try_files /getpath.php;
    }

This is just the pseudo-code example to get it rolling.

Some miscellaneous observations / comments:

• The Golang response doesn't look like valid json, replace preg_match_all with json_decode if so.
• readfile is not super efficient. Consider being creative with a 302 response.

---

Nginx + Lua

sites-enabled:

lua_package_path "/etc/nginx/conf.d/lib/?.lua;;";

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location /getfile {
        root /var/www/html;
        resolver 8.8.8.8;
        set $filepath "/index.html";
        access_by_lua_file /etc/nginx/conf.d/getfile.lua;
        try_files $filepath =404;
    }
}

Test if lua is behaving as expected:

getfile.lua (v1)

  ngx.var.filepath = "/static/...";

Simplify the Golang response body to just return a bland path then use it to set filepath:

getfile.lua (v2)

local http = require "resty.http"
local httpc = http.new()
local query_string = ngx.req.get_uri_args()
local res, err = httpc:request_uri('https://go.example.com/getpath?file_id=' .. query_string["id"], {
    method = "GET",
    keepalive_timeout = 60,
    keepalive_pool = 10
})

if res and res.status == ngx.HTTP_OK then
    body = string.gsub(res.body, '[\r\n%z]', '')
    ngx.var.filepath = body;
    ngx.log(ngx.ERR, "[" .. body .. "]");
else
    ngx.log(ngx.ERR, "missing response");
    ngx.exit(504);
end

resty.http

mkdir -p /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http_headers.lua" -P /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http.lua" -P /etc/nginx/conf.d/lib/resty