Exploring File Download Reverse Proxy With Nginx + Lua + Redis

Medium Link: Exploring File Download Reverse Proxy with Nginx + Lua + Redis
If you have a medium.com membership, I would appreciate it if you read this article on medium.com instead to support me~ Thank You! 🚀
images/intro.png
Photo by Author

Intro

This article is an exploration of how to reverse proxy an HTTP request to a dynamic URL stored in Redis. I will be using download as the base for my exploration where my dynamic URL is a S3 Presigned URL. So read on if you are interested to find out more!

Before we start, check out my previous article “Building a S3 File Download Service using Spring Boot and Minio” if you have yet to :)

File Download Series

Background Context

I have a download service for downloading files from my S3 cloud storage service (Minio). The client can download files using the Presigned URL after the download service has validated/authenticated the client’s request.

What is a Presigned URL

images/download_flow_diagram.png
Photo by Author - Flow for Downloading with Presigned Url

Presigned URLs are a commonly used method to share files securely while maintaining control over access. In general, a user with access to the file generates a Presigned URL and any unauthorized user can download the file using the Presigned URL.

Note: Presigned URL can also be used for uploading and it is only valid for a specified duration.

However, instead of providing the unauthorized user with the Presigned URL, I want to explore using a custom dynamic download URL.

What is the Download URL

Download URL is a custom URL generated by the download service where a typical download flow is made up of 2 parts (similar to Presigned URL).

  1. The API call to get a download URL
  2. The API call to download the file using the download URL
images/download_flow_diagram.png
Photo by Author - Flow for Downloading with dynamic download URL

The diagram above illustrates how the download flow works with a custom download URL. An example of a download request is as such:

  1. The client requests to download a file.
  2. Nginx proxy request to download service.
  3. Download service process the download request (Eg. authenticating/validating / etc…) → generates a downloadId (Eg 123456) → request for a Presigned URL from S3 (Eg. http://s3_presigned_url) → stores the presigned URL in Redis where key = downloadId → return a response with the download URL (Eg. http://example/download/123456).
  4. The client requests to download using the download URL — http://example/download/123456.
  5. Nginx gets the Presigned URL from Redis using downloadId (123456) and proxies the download request from http://example/download/123456 to http://s3_presigned_url. Note: The downloadId key is deleted from Redis after retrieval.

Implementing the Download Reverse Proxy

This exploration is an expansion from the previous article, so please refer to that on how to create a File Download Service.

Step 1 - Generating Download Url

Instead of returning a Presigned URL, we want to return a custom download URL as our response. So firstly, using the service below, we can connect to S3 and generate the Presigned URL.

@Service
class StorageService(
    private val minioClient: MinioClient,
    @Value("\${object-storage.bucket}") private val bucket: String
) {
    fun generatePresignedUrl(fileId: String): String? {
        val args = GetPresignedObjectUrlArgs.builder()
            .bucket(bucket)
            .`object`(fileId)
            .method(Method.GET)
            .expiry(5, TimeUnit.MINUTES)
            .build()
        return minioClient.getPresignedObjectUrl(args)
    }
}

Next, we will create a unique download ID to generate a download URL and store the download ID to the Presigned URL mapping in Redis.

@Service
class UrlService(
    private val redisTemplate: StringRedisTemplate,
    private val uniqueIdGenerator: UniqueIdGenerator,
    @Value("\${download.url}") private val downloadUrl: String
) {
    companion object {
        private val logger = LoggerFactory.getLogger(UrlService::class.java)
    }

    fun generateDownloadUrl(presignedUrl: String): String {
        val downloadId = uniqueIdGenerator.generateUniqueId() ?: throw Exception("Unable to generate unique download Id")
        if (!redisTemplate.opsForValue().get(downloadId).isNullOrEmpty()) {
            throw Exception("Download Id already exists!")
        }

        logger.info("Storing to Redis :  $downloadId = $presignedUrl")
        redisTemplate.opsForValue()[downloadId] = presignedUrl
        return "$downloadUrl/$downloadId"
    }
}

Note that the download.url value refers to the Nginx URL, which is our file download gateway.

Also If you are curious, the uniqueIdGenerator is a just simple service that creates a md5 hash of a timestamp combined with some random string value.

@RestController
@RequestMapping("/api/v1/download")
class DownloadController(
    private val storageService: StorageService,
    private val urlService: UrlService
) {
    companion object {
        private val logger = LoggerFactory.getLogger(DownloadController::class.java)
    }

    @GetMapping("/url")
    fun getFileDownloadUrl(@RequestParam fileId: String): ResponseEntity<String> {
        logger.info("Received request to generate download url for file - $fileId")
        val presignedUrl = storageService.generatePresignedUrl(fileId) ?: return ResponseEntity.notFound().build()
        val downloadUrl = urlService.generateDownloadUrl(presignedUrl)
        return ResponseEntity.ok(downloadUrl)
    }
}

Lastly, the download URL is then returned as a response to the client as shown in the controller above. With that, we completed the first part of the download flow!

images/download_flow_diagram_cut1.png
Photo by Author - Recap of the 1st part (generating a download URL)

Whenever a download request is made, the download service will process the request, generate and store the Presigned URL and download ID in Redis, and return a download URL to the client.

Now that we have the download URL, let’s look at how Nginx is configured to route download requests to the download service and the S3.

Step 2- Configure Nginx Reverse Proxy

We are using openresty docker image as our Nginx gateway. It is a superset of Nginx, bundled with LuaJIT interpreter as well as various Lua libraries which enables us to run embeddable scripts in Nginx. Here’s what we will be configuring in the nginx.conf!

# nginx-lua.conf
env DOWNLOAD_SERVICE_URL;

http {
  lua_package_path "/path/to/lua-resty-redis/lib/?.lua;;";
  include /usr/local/openresty/nginx/conf/mime.types;
  default_type application/octet-stream;

  resolver 127.0.0.11 ipv6=off;

  server {
    listen 8000;

    # Download Service
    location /download-service {
      set_by_lua $upstream_server 'return os.getenv("DOWNLOAD_SERVICE_URL")';
      rewrite ^/download-service/(.*) /$1 break;
      proxy_pass $upstream_server;
      proxy_set_header Host $host;
    }
  }
}

# docker-compose.yaml
version: '3.9'

services:
  minio:
    ...
  redis:
    ...
  download_service:  # <-- note: service's name = hostname
    ...
  nginx:
    container_name: nginx_gateway
    image: openresty/openresty:1.25.3.1-3-alpine
    ports:
      - "8000:8000"
    volumes:
      - ./nginx-lua.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro
    environment:
      - "DOWNLOAD_SERVICE_URL=http://download_service:8080"

resolver 127.0.0.11 ipv6=off → this is necessary to instruct NGINX to use Docker’s internal DNS resolver which allows containers within the same bridge network to communicate with each other using the service names as hostnames (for docker-compose).

Firstly, in the nginx.conf above, we added a path proxy that will route all requests with prefix /download-service to the download service as shown above. Notice, that we are using environment variables to set the upstream_server at runtime.

Next, we will add a path proxy in the nginx.conf to process the download URL.

# nginx-lua.conf
env REDIS_PASSWORD;
env REDIS_HOSTNAME;
env REDIS_PORT;
env S3_HOSTNAME;
env S3_PORT;

http {
    resolver 127.0.0.11 ipv6=off;

    init_worker_by_lua_block {
        redis = require "resty.redis"
    }

    server {
        listen 8000;

        # Allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # Disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location /download/ {
            set $target '';
            access_by_lua_block {
                local uri = ngx.var.uri
                local downloadId = string.match(uri, "^/download/(%w+)$")
                if not downloadId then
                    ngx.say("Invalid Download URL")
                    return ngx.exit(400)
                end

                local red = redis:new()
                red:set_timeout(1000) -- 1 second

                local redis_password = os.getenv("REDIS_PASSWORD")
                if not redis_password then
                    ngx.log(ngx.ERR, "REDIS_PASSWORD environment variable is not set")
                    return ngx.exit(500)
                end

                local redis_hostname = os.getenv("REDIS_HOSTNAME")
                local redis_port = os.getenv("REDIS_PORT")
                local ok, err = red:connect(redis_hostname, redis_port)
                if not ok then
                    ngx.log(ngx.ERR, "failed to connect to redis: ", err)
                    return ngx.exit(500)
                end

                local res, err = red:auth(redis_password)
                if not res then
                    ngx.log(ngx.ERR, "failed to authenticate: ", err)
                    return ngx.exit(500)
                end

                local presignedUrl, err = red:get(downloadId)
                if not res then
                    ngx.log(ngx.ERR, "failed to get key: ", err)
                    return ngx.exit(500)
                end

                if presignedUrl == ngx.null then
                    ngx.say("Download URL no longer valid!")
                    return ngx.exit(400)
                else
                    local res, err = red:del(downloadId)
                    if not res then
                        ngx.log(ngx.ERR, "failed to delete key: ", err)
                        return ngx.exit(500)
                    end
                end

                ngx.var.target = presignedUrl
            }

            set_by_lua $s3_hostname 'return os.getenv("S3_HOSTNAME")';
            set_by_lua $s3_port 'return os.getenv("S3_PORT")';

            proxy_set_header Host $s3_hostname:$s3_port;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            proxy_http_version 1.1;                         # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass $target;
        }
    }
}

# docker-compose.yaml
version: '3.9'

services:
  minio:
    ...
  redis:
    ...
  download_service:
    ...
  nginx:
    ...
    environment:
      - "REDIS_PASSWORD=APHGGBHQkHdgD49pjuBfkyKCKZAsrr"
      - "REDIS_HOSTNAME=redis"
      - "REDIS_PORT=6379"
      - "S3_HOSTNAME=minio"
      - "S3_PORT=9000"

proxy_set_header Host $s3_hostname:$s3_port; → this is important for resolving SignatureDoesNotMatch error. Refer below for more details.

In the nginx.conf above, to connect to Redis, we will make use of lua-resty-redis - a lua redis client driver. The script above basically extracts the download Id from the download URL, lookup Redis using the download ID (key) for the Presigned URL, and proxies the request from the download URL to the Presigned URL.

images/download_flow_diagram_cut1.png
Photo by Author - Recap of the 2nd part (downloading file using download URL)

With that, our Nginx is now configured to proxy our custom download URL to the Presigned URL that we stored in Redis. Awesome!

Issues you might face…

SignatureDoesNotMatch Error

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <Key>test.txt</Key>
  <BucketName>storage</BucketName>
  <Resource>/storage/test.txt</Resource>
  <RequestId>17C8ACB0FC2FA7B7</RequestId>
  <HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>

Referencing this issue, it is noted that the proxy_set_header Host value must match the Presigned URL. When using docker-compose, my s3 endpoint was http://minio:9000 as shown below.

version: '3.9'

services:
  minio:
    ...
  download_service:
    ...
    environment:
     - "OBJECT_STORAGE_ENDPOINT=http://minio:9000"
    depends_on:
      - minio

When a download request is made, the download URL will look like http://localhost:8000/download/<downloadId> and a Presigned URL will look like this http://minio:9000/presigned_params…. This will result in a SignatureDoesNotMatch error if we do not set the appropriate Host in our reverse proxy configuration.

Hence, in other to resolve the SignatureDoesNotMatch error, we just have to set proxy_set_header Host minio:9000.

Summary

That’s it! This was an interesting topic for exploration. I am not sure whether this is the right approach for dynamically proxying URLs, but it was definitely fun to get this to work. If you have the time, do give this a try as I think it challenges your knowledge of Lua Scripting, Nginx, and Docker Networking.

Refer to the GitHub repo below for the codes.

Thank you for reading till the end! ☕
If you enjoyed this article and would like to support my work, feel free to buy me a coffee on Ko-fi. Your support helps me keep creating, and I truly appreciate it! 🙏

mediumsoftware-engineeringspring-boots3minionginxredisluaref