I Found a BETTER Way to Do File Uploads

The way you learned to do file uploads has some SERIOUS problems. Let's see how to used signed upload URLs which removes the need for you to handle file uploads on your own server!
*Newsletter*
Newsletter 🗞 - www.jamesqquick.com/newsletter
*RESOURCES*
Xata Upload URLS - Xata Docs - xata.io/docs/sdk/file-attachm...
Theos Tweet on File Uploads -x.com/t3dotgg/status/17502606...
Vercel Serverless Size and Runtime Limits - vercel.com/docs/functions/ser...
*DISCORD*
Join the Learn Build Teach Discord Server 💬 - / discord
Follow me on Twitter 🐦 - / jamesqquick
Check out the Podcast - compressed.fm/
Courses - jamesqquick.com/courses
*QUESTIONS ABOUT MY SETUP*
Check out my Uses page for my VS Code setup, what recording equipment I use, etc. www.jamesqquick.com/uses
*TIMESTAMPS*
00:00 - Intro
00:30 - Basic HTML File Uploads
01:45 - File Uploads with JavaScript
02:30 - Handling File Uploads on Your Server with Multer
03:50 - Why You Should Handle File Uploads on Your Own Server
04:45 - How Signed Upload URLs Work
05:50 - Hands-on Demo with Next.js and Xata
07:50 - Wrap Up

[@DanielAWhite27]

A big thing here that is missing is the transactional safety of this. I treat all my endpoints/requests as a transactional boundary. If using a second request, the onus is on the client to do more work that still can fail, and your data source wouldn’t be any wiser. It looks like that db has the capability to make sure things are “complete” but not out of the box using s3. It also isn’t free. It’s shifting from one cost to another

[@thewildweb]

Hey can you briefly help me to explain via blog or example that will be very helpful

[@davefox2257]

We had done something similar, but you have to add the extra step of upload confirmation. If there was an error on upload, the entry still exists on the db. This caused name already exists errors. It ends up being two to three steps depending on if all goes well or not.

[@naterpotatoers]

Interesting idea I’ve not heard before. Now I’m curious if you have any ideas on live video streaming and your approach to how to handle that - thanks

[@buddy.abc123]

Good tips, thanks for sharing

[@sergi0YT]

This is what I have implemented a while to upload files from a mobile device. Lambda creates an url for the file, sign it for write access and then you can just POST the file from the phone. All network load is on S3 now.

[@JustinSBarrett]

Really informative video, but there was one piece I was hoping for but didn't see: what's the process of creating a signed upload URL? Is that something that Xata (and other storage systems) do for you, or is there more code needed for that part?

[@DanielAWhite27]

AWS S3 has an api for your backend to create these

[@stephenpaulhassall]

That is super cool and very impressive. 👍

[@Rapid-eraser]

This does not look as a generally BETTER Way, more like an other way I would say :) I dont see any gain to small projects ... If you are build a YT clone you should have already have the resources required. Also in EU you are probable would have lots of problems regarding GDPR, you may have to sign a GDPR agreement with the third party service handling the storage ... looks like hell already.

[@naterpotatoers]

From my understanding this seems more performant in that you aren’t passing all of your data through your middleware essentially ? Dunno anything about gdpr tho

[@ardianhotii]

Is this the same logic using cloudinary ?

[@tomich20]

never heard of signed urls. Question.. if i rquest to upload a 2kb mp4 video, get the URL, and then continue to upload my 4GB full hd pirate movie to that URL, would that work? seems risky to give the power to the client.

[@jesper.ordrup]

would like to know too

[@racquad]

But what if I need to post process the file? Let's say from a video I need to create lighter versions of it (360p, 480p etc.). Or if it's an image I then need to save the master for later and then create a thumb and a lighter 1024px version of it? I need to go to my blob repository anyway, fetch the master file and process it, right?

[@gajrajsingh51]

Send a request to server to tell them the upload is complete

[@racquad]

@@gajrajsingh51 That's for sure. But my point is that server will need to download the full file anyway to run the post processes anyway. So you can either upload directly to S3 if you just need to save the file or you can upload to server and then server will produce derivative low quality versions and upload afterwards.

[@ErikMeliska]

Good question, there was a commented code about transcribing the video, I wonder, how this would be done, without having physical data on the server.

[@skapator]

Feel this is smart and has a couple of use cases that can be applied. But really not a new idea at all. At the end you are switching costs and introduce error creeps. An enterpize app will have storage on their servers. The main issue is that this simplified example of just a video upload with a filename/path is never the case in real production. From video manipulation, validation transactional integrity, queueing, background tasks, this is going to be a nightmare to split and handle unhappy paths. Makes sense if you are renting every bit of a small app and have no infra at all but not really an issue in real prod. This "renting all" mentality of the past few years is actually costing more on small apps where all of these concerns seem like premature opt. If we look at actuall full stack frameworks, these cost/issues are non existent.

[@kamill34]

I've tried serverless function, and I think the limitiation is 4 mb

[@nightshade427]

Does this leave you open to security issues? You are telling the client where all your videos live and how to access them directly. If the backend is s3 for example and policies aren't setup perfect could it lead to issue?

[@DanielAWhite27]

Usually the upload url is opaque

[@johnaroj]

Nice approach. I was doing something like that but in a different order. I send everything in one call and immediately return the name and a screenshot using ffmpeg. which will return a image and status pending. in the background will process the files without waiting for the server to respond. And with this process you can choose to request the progress in the background or just refresh the page manually to see if data is ready

[@polkulotznad3987]

how

[@arupde6320]

github link ??

[@9903918500]

why can't we do fs.unlinkSync to the filepath after we finished uploading the file to our blob storage ? this can stop the need to store any data within the server itself.