• Oded Levy

Streaming files between services in Node and Express

Updated: Jan 12, 2020

Often I find myself developing an app that requires to upload and download files from some kind of storage down to a client app. The client can be a swagger or Postman app exposing your set of api's. But more often it will be some html page or more complex web client built using Angular or React in which you will upload the file to the server and save it in a storage for future consumption.

When developing the server side I like the microservice architecture since it provides a better way for maintaining, encapsulating and reusing code.

Here is a sample of a very basic architecture. On the left we have various web apps that consume the services. They can connect to the server from a single exposed port as to not expose all other services to the outside world and to allow a central place for managing cross cutting features such as authentication, authorization auditing etc.

When dealing with upload of files from the client to the server it brings us to the need of way to stream the files from the client into the Storage Service with two things in mind:

  1. The Gateway service should not store the file locally before passing it off to the Storage service since it is not required.

  2. The Gateway service will not load the entire file into memory prior to sending it to the Storage service since it will cause memory consumption of the service to rise drastically and possibly crash the service.

In order to achieve this we would like our Gateway service to expose a route that will be able to receive chunks of data and stream them as they arrive down to the storage service one chunk at a time.

The idea is simple and after some struggling the solution itself was eventually simple as well — but it took me time to get there.

In the past I would use npm package named formidable. With approx 2.5 million downloads a week I figured that this would be a good place to start.

Later on multer npm package started to appear as the preferred package to use in such a scenario. I found out that multer had 2 major options:

  1. Use multer as middleware in order to save the file to the file system before reaching the code to my route (clean and simple for the file server).

  2. Do the same yet use the in memory option which I thought would be great for the Gateway service.

So I went and wrote the code that does just that and it actually worked. Or so I thought at the time. In order to validate that both major points are fulfilled (file not saved to disk and not entirely loaded to memory) I passed a very large file to the service and monitored the node services memory. To my disappointment I found out that multer will load the entire file to memory and only then pass it on to my route. This meant a huge memory issue when dealing with large files or with a service that it used extensively since eventually the app will grow to a point where it will crash.

The solution

And so I returned to seek for a better solution until I came up with this.

In the gateway service we expose a route that will pipe the request down to the Storage service. For this sample I am using request npm package for making the call from the Gateway to the Storage.

And this is how my route code inside the Gateway service looks like

sendFile = (req, res, next) => {  
  let url = "http://localhost:3001/files/sendFile"

So basically what we are doing is piping our request containing the multipart files directly to the FileServer service (using request npm package) and then piping the result back into the response. Its that simple.

The File server code using multer will look as following:

private createRoutes() {  
  var storage = multer.diskStorage({    
    destination: "./uploads",
    filename: function (req, file, cb) {
       cb(null, file.originalname )
var upload = multer({ storage: storage })"/sendFile",upload.any(), this.sendFile);

You can try out the full source code on github

To test that the memory consumption remains steady even when passing large files you can use Postman’s form data for sending a very large file to the server using route http://localhost:3000/files/sendFile. Just make sure to change the key from ‘text’ to ‘file’ so that you can upload the file in postman.

I hope the post will save you some time if find yourself with similar needs.

110 views0 comments

Recent Posts

See All