How To Upload Files In Elixir Phoenix JSON Api — Tentamen Software Testing Blog

Gigalixir Node Memory Consumption Of 25MB File Upload As Base64 encoded string.

TL;DR

In my Elixir Phoenix application, I have a JSON API for mobile applications. One endpoint accepts file uploads. In this post, I explain my architectural decision why to implement file upload as multi-part form data.

JSON API Upload As Base64 Data Stream

Google returned this excellent DockYard blog post: Building An Image Upload API With Phoenix. I implemented image and video upload as a Base64 encoded string in JSON. The above Screenshot depicts memory consumption for file upload of 25MB. So for 40 concurrent file uploads of that size, I will need 1GB Gigalixir instance. What about 400 concurrent uploads?

MultiPart Form Data

I switched to multipart form data. Your controller is straightforward:

@spec add_note(Plug.Conn.t(), any) :: Plug.Conn.t()
def add_note(conn, params) do

When clients send multi-part Form Data with a file, the Phoenix framework does all the heavy work before your controller is in control. So if you sent a file under the form data attribute named screenshot, then Phoenix will put in params["screenshot"] Plug.Upload structure.

Note that you still can send back a JSON response. My initial guess was that I must use JSON on input for JSON API. Wrong, we can also use multipart form data. For sending additional attributes, just send them as additional form data parameters.

The Gain

The same file upload of 25MB left the following memory footprint on my Gigalixir instance:

Memory Consumption For Multipart upload Of 25MB file

Check two mini spikes at the end. 3MB memory consumption.

Under The Hood

Plug.Upload is A server (a GenServer specifically) that manages uploaded files. Uploaded files are stored in a temporary directory and removed from that directory after the requested file dies. This temporary location is stored in :path an attribute. But the most important is this:

All options supported by Plug.Conn.read_body/2 are also supported here. They are repeated here for convenience:

:length — sets the maximum number of bytes to read from the request, defaults to 8_000_000 bytes

:read_length — sets the amount of bytes to read at one time from the underlying socket to fill the chunk, defaults to 1_000_000 bytes

Socket stream is read in 1Mb and is written to tmp file location using Elixir File.Stream. When 1Mbytes is ready, they are written to file. Making memory impact minimal.

Remember

Use multipart file import in Phoenix applications because Plug.Upload does all the magic in the background.

Originally published at https://blog.tentamen.eu on November 16, 2020.