Getting 100TB of Data to the Cloud with AWS Snowball Edge Storage Optimized Devices

We recently completed a computation of Pi to 100 trillion digits that left us with a single 100TB output file. We wanted to hang on to the data, the cloud is an obvious fit for this need. So we reached out to Amazon to snag a few of their AWS Snowball Edge Storage Optimized devices with 80TB of capacity each.The goal was to get our data from the lab in Cincinnati to an S3 bucket in AWS us-east-2 region as fast as possible.
Full Snowball Edge Article -
www.storagerev...
AWS Snow Podcast -
www.storagerev...
100T Pi Computation Details -
www.storagerev...
Pi Video -
• We Computed 100 Trilli...
Discord Server -
/ discord

[@Nimitz_oceo]

I worked at AWS 4 years ago, my dream is still to own one of these

[@AdaptiveSystems]

do you know what motherboard it uses? supermicro X10SDV-CIBF-AM041 by any chance?

[@jfkastner]

"Snowball fight" is when you get an error message ;-) Well done, very interesting!

[@StorageReview]

Too heavy to throw them far. Need the Snowcone for that!

[@jtkilroy]

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway"

[@jtkilroy]

Love that 5.25 HDD in the box to the left. Ah, the memories.

[@jonsmith7718]

Are you going to the archive to datalake? Amazon S3 Glacier or Redshift ( what S3 storage costs are you considering) there is also a gotcha in S3 where you need to verify the moving of the objects sometimes they might appear to be moved only and you incur a cost down the road. or so ive heard ..lol also why so many pie digits? i think string theory has come up with the solution to pie again so ive heard ..haha

[@mcpr5971]

I tried one of these about 4 years ago, but despite having a 10gbe adapter (and 10gbe link), it choked on a large number of small files. It has to do with their software running in java, probably spawning a new process with hundreds of mb of RAM for the JRE for each file. We got around this problem by creating large files that were formatted as an ext4 filesystem, copying all the small files into the filesystem, and copying that file to the snowball. This is similar to a zip file, but compression was not necessary. Anyway, thanks for the video, hopefully AWS have solved this small-files issue on the latest versions.

[@StorageReview]

Thanks for the comment. How did you end up moving your pile of files?

[@mcpr5971]

@@StorageReview I said in the comment. copying them into large files that I formatted as if they were hdds, using mkfs.ext4. I got near linespeed transfer rates transferring large files.

[@StorageReview]

@@mcpr5971 oh nice. Almost the reverse of us that we had to chop large files into small ones.

[@qmriis]

... just use tar?

[@twei__]

SnowMobile test when?

[@StorageReview]

We’ve considered asking for it lol. Trying to get the GPU Snowball next though. Baby steps.

[@dcviper985]

Based on the shipping ZIP that went to an actual Amazon fulfillment center and not directly to an AWS data center.

[@LandonHalley]

Uploaded 3 minutes ago 😳😳😳😳😳

[@StorageReview]

Thanks.

[@ohiitppy]

Can I check with you whether is it possible for LTO tapes to transfer data to Snowball via NetBackup? Please let me know?

[@pubdoart8052]

Fire tablet is the display. 😂

[@ssamer88]

How did you split up the big file? I have a 24TB VMDK file that I need to put on the snowball edge.. any ideas would be greatly appreciated

[@StorageReview]

We used a built in utility to split the file, the alternative was using Linux. that vmdk is going to have to use something to try to split the file in Linux, recombine and compare hash. Nothing is straightforward for it.

[@franknowak9305]

wow this cool as shit

[@Logan800]

no ssds?😟

[@StorageReview]

Snowball has a tiny bit of flash.

[@flatlineprod]

Yo

[@relaxingnature2617]

I wish you would explain this in a more dumbed down way .... I'm pretty tec savy but i didnt know some of the terms and works you used ..you need to explain it in such a way that mediumly smart people can understand

[@StorageReview]

We have the written piece that may be easier to follow. Let us know if that helps. We also have a Discord server to live chat about this or any tech topic.

[@crisgallardo280]

I think you should learn some more AWS because everything mentioned was pretty rudimentary.

[@StorageReview]

@@crisgallardo280 our mission is not advanced AWS content. But let us know if you have ideas and maybe we can split the difference.