ProSoundWeb Community

Sound Reinforcement - Forums for Live Sound Professionals - Your Displayed Name Must Be Your Real Full Name To Post In The Live Sound Forums => LAB: The Classic Live Audio Board => Topic started by: Brad Harris on June 09, 2021, 10:59:09 AM

Title: Multitrack archiving
Post by: Brad Harris on June 09, 2021, 10:59:09 AM
Who is multi tracking shows (Dante/USB/AVB/etc) to DAW, and how are you archiving after the gig?


Assortment of hard-drives, servers, cloud?


What about software? Any filing/naming conventions, archival database?


Any differences in workflow for tour/multi day events and single one offs?




Brad
Title: Re: Multitrack archiving
Post by: Corey Scogin on June 09, 2021, 11:42:28 AM
I've been doing this for over a decade but only for a handful shows a year average.

I convert the uncompressed audio tracks to FLAC and archive on a local disk in my home desktop, organized in folders by date with the show title included.
If I do a mixdown, those get saved in FLAC format also with song titles if I have them in a "mixed" folder underneath the project folder.

ALAC is another option for archive format if that works better with your workflow. I use Reaper as a DAW which can natively use FLAC for track sources and output.
**caveat: FLAC and ALAC do not store any timecode as far as I know so pulling an archive to use alongside other media sources will require manual sync.

All that gets backed up to another local disk and to Crashplan (https://www.crashplan.com).

After the data grows much larger than what I have now, a different solution may be needed. For now, disk space technology is keeping up with my requirements without breaking the bank.
Other options for online archive are AWS S3 Glacier and Wasabi (https://wasabi.com). Those can get pricey depending on the usage: total data stored, data in, data out, etc.
Title: Re: Multitrack archiving
Post by: Frank Koenig on June 09, 2021, 11:55:15 AM
Other options for online archive are AWS S3 Glacier and Wasabi (https://wasabi.com). Those can get pricey depending on the usage: total data stored, data in, data out, etc.

A plug for Amazon Web Services Simple Storage Service (AWS S3). They charge by the MB and it may be less than you think. I don't have huge audio files but all my pictures and other stuff I care about are backed up there and it runs around $2.50 / month. I love the command line interface. --Frank
Title: Re: Multitrack archiving
Post by: Corey Scogin on June 09, 2021, 12:24:48 PM
A plug for Amazon Web Services Simple Storage Service (AWS S3). They charge by the MB and it may be less than you think. I don't have huge audio files but all my pictures and other stuff I care about are backed up there and it runs around $2.50 / month. I love the command line interface. --Frank

Yes...but caveats abound when working with the various storage tiers. 10TB at S3 Standard would be ~$230/mo but at "Deep Archive" it's ~$9.90.
"Deep archive" and Glacier require time to retrieve. Some tiers have data retrieval costs. All tiers incur data transfer costs coming back out of S3.
To get that entire 10TB back out would be ~$810 unless you pulled no more than 1TB/mo in which case it would be free.

You just have to be careful. You won't be warned before incurring charges.

Title: Re: Multitrack archiving
Post by: Justice C. Bigler on June 09, 2021, 02:13:28 PM
I can't imagine how long it would take me to up load 10TB over my satellite internet connection. I doubt that my connection would even stay stable long enough to finish the upload.

I store my stuff on hard drives.
Title: Re: Multitrack archiving
Post by: Dave Garoutte on June 09, 2021, 02:46:57 PM
Samsung T5 SSDs comes in 1 and 2 TB and are fairly cheap.
Title: Re: Multitrack archiving
Post by: John L Nobile on June 09, 2021, 03:00:19 PM
The method you should use depends on how much storage you'll need. I see no reason to use cloud based subscriptions when storage is so cheap. Since it will be archived, there's no need to keep it online and expose it to ransomware.

I've used a USB dock and just bought normal HDD's. If it's really important, I'll make 2 copies. Just find a dry secure place to store the HDD's. Having 2 copies means that you can keep them in 2 locations for even more safety. And don't use huge HDD's as you'll lose more data in case of failure.

Maybe I'm paranoid but I really don't trust my data with a service when I can do it myself.
Title: Re: Multitrack archiving
Post by: Corey Scogin on June 09, 2021, 03:02:45 PM
Samsung T5 SSDs comes in 1 and 2 TB and are fairly cheap.

I don't have a link to the research on hand but I recall reading that SSD retention isn't as good as HDD retention when "cold" (not powered on).
If using your own disks, John's suggestion of making two copies is a must. Buy new drives and refresh on a schedule.
Title: Re: Multitrack archiving
Post by: Frank Koenig on June 09, 2021, 03:27:31 PM
To get that entire 10TB back out would be ~$810 unless you pulled no more than 1TB/mo in which case it would be free.

You just have to be careful. You won't be warned before incurring charges.

Yikes! And thanks, Corey. Words to the wise. I always figured S3 was a gateway drug -- I'd better watch my usage. -F
Title: Re: Multitrack archiving
Post by: Brian Jojade on June 09, 2021, 04:22:08 PM
Local storage is most likely going to be less expensive than cloud storage solutions over time. (there are some exceptions under very specific use scenarios)

If you store on your own hard drives, it's imperative that you make sure you have an appropriate backup strategy.  Follow the 3:2:1 rule if you care about keeping your data.

Rotational hard drives are by far the most economical for archival storage today.  Realistically, expect 5 years of life out of a drive at max. At that point, you should migrate to new hardware.  Old drives are great for 3rd or 4th emergency backups though, so they don't completely go to waste.

The hardest part is deciding what you actually do want to keep and for how long, then determining how much storage space you'll end up needing.  One large volume makes things easier, but when something goes wrong, you risk the entire collection.  Smaller subsets of data on smaller drives is more to manage, but a failure of a single drive becomes less of a hassle to put back together.
Title: Re: Multitrack archiving
Post by: Mark Oakley on June 09, 2021, 04:22:45 PM
Just did a live Internet broadcast gig on Monday. I generally record all inputs to their own track plus a stereo board mix for quick reference. I usually record a stereo audience pair, but no audience on this one. My recording setup is direct off my iLive from a Waves M2 card to a Dell Alienware 15 R4 Laptop. For live recording I use Waves Tracks Live and mix in Cubase 11.

To archive I'll make two copies: one to an external 8Tb backup drive and one to a Blu-Ray disk. I bought this Blu-Ray burner a year ago and use the 100 Gb M disks: https://www.owcdigital.com/products/mercury-pro-optical

-Mark
Title: Re: Multitrack archiving
Post by: Erik Jerde on June 09, 2021, 04:48:20 PM
Fortunately audio isn’t data intensive (compared to video) so you can archive a lot for cheap.  Bit rot a offline device failure is real.  If you care about the data it needs to be on-line and checked for bit rot on a schedule (I check weekly).  If you want to go truly offline then the only time proven medium is tape.  LTO is expensive compared to disk but you’re buying reliability.  You can buy a nas unit and then use a cloud backup service for your second off-site copy.  Some nas devices can backup directly to a cloud provider which is great.

Just remember data in one place - even if that place is your “backup drive” - isn’t backed up.  I backup to multiple devices in multiple physical locations.  I’m not geographically diverse but I don’t anticipate caring if that becomes an issue.

There are media archiving software platforms out there.  I’ve looked into them in years past but they were always way to expensive.  They provide database front end that facilitates things like workgroup project sharing and searchable metadata.  Some will even do voice recognition transcripts that are then searchable.
Title: Re: Multitrack archiving
Post by: Tim McCulloch on June 09, 2021, 06:34:14 PM
Yikes! And thanks, Corey. Words to the wise. I always figured S3 was a gateway drug -- I'd better watch my usage. -F

The $300 credit AWS gives you when you sign up can be pissed away in a few days if you leave a 'machine' running or have to pay for data transfer between to AWS zones because you set up a 'machine' in the wrong zone.
Title: Re: Multitrack archiving
Post by: Tim McCulloch on June 09, 2021, 06:44:12 PM
I give all client recordings to the client, and the safety copy I keep... until the client verifies their original is intact.  After that the thumb drive, HD, SSD, whatever... goes into the pile for erasure and reuse.  The only audio files I keep 'forever' are the recordings I make for myself from purchased sources.

I'll take some bit rot over magnetic tape oxide shedding...
Title: Re: Multitrack archiving
Post by: Ned Ward on June 09, 2021, 07:14:56 PM
Backups on rotational drives that are always plugged in. If you have hard disk drives, don't make the mistake of leaving them unplugged - over time, the drive bearings can seize, and then when powering up your drive can be toast.

I use Backblaze for backups - it's a pay one price regardless of how much you have to back up or how many attached drives. Restore can either be over the internet or they'll mail you a drive for $139. Restore your files, return the drive and your payment is returned.

Check it out and if this looks like it works for you, this code gets you a month free - and in return gets me a month free. If you don't want to click on the link, no worries, but it's a great backup system that Pro Tools, Photoshop and Video editors also swear by - I found out about it in 2017 from a Photoshop editor doing all the packaging files for a small toy company...

https://secure.backblaze.com/r/01a5ln
Title: Re: Multitrack archiving
Post by: Russell Ault on June 09, 2021, 08:10:14 PM
Local storage is most likely going to be less expensive than cloud storage solutions over time.
{...}
Rotational hard drives are by far the most economical for archival storage today.  Realistically, expect 5 years of life out of a drive at max. At that point, you should migrate to new hardware.
{...}

Less expensive, perhaps, but I'd much rather pay someone to worry about that kind of maintenance.

For what it's worth, I use OVH for cloud storage (they're the largest European hosting provider), and their prices (at least in Canada) are quite a bit cheaper than AWS (I can't believe how much S3 charges for data transfers!).

-Russ
Title: Re: Multitrack archiving
Post by: Brian Bolly on June 10, 2021, 11:03:09 AM
Check it out and if this looks like it works for you, this code gets you a month free - and in return gets me a month free. If you don't want to click on the link, no worries, but it's a great backup system that Pro Tools, Photoshop and Video editors also swear by - I found out about it in 2017 from a Photoshop editor doing all the packaging files for a small toy company...

https://secure.backblaze.com/r/01a5ln

This thread is timely.  I just lost a 2yr old SSD in a RAID1 setup this morning.  Took me a bit to figure out why my MBP kept giving me the grey screen of death, and then I discovered my stack o' drives was hot enough to fry an egg on. 

I have a feeling the other drive may be headed south as well, so I'm dumping them to the cloud asap. Thanks for the link!
Title: Re: Multitrack archiving
Post by: Corey Scogin on June 10, 2021, 11:54:41 AM
I use Backblaze for backups - it's a pay one price regardless of how much you have to back up or how many attached drives. Restore can either be over the internet or they'll mail you a drive for $139. Restore your files, return the drive and your payment is returned.

https://secure.backblaze.com/r/01a5ln

Backblaze also has a more generic storage service "B2" which is S3 protocol compatible. It's useful if you don't want to keep everything locally.

I tried several systems a few years back and I found that the personal backup Backblaze service is good but there are some caveats: You don't have granular control over what gets backed up. Restores are only available through a web browser, not through the desktop app so downloading large restores will be as reliable as your web browser and connection. Other systems (like Crashplan) use their desktop apps so it can do multi-connection segmented downloads and more intelligently restart incomplete downloads without manual intervention. This may have changed since I looked at Backblaze years ago.
Title: Re: Multitrack archiving
Post by: Ned Ward on June 10, 2021, 01:38:58 PM
Backblaze also has a more generic storage service "B2" which is S3 protocol compatible. It's useful if you don't want to keep everything locally.

I tried several systems a few years back and I found that the personal backup Backblaze service is good but there are some caveats: You don't have granular control over what gets backed up. Restores are only available through a web browser, not through the desktop app so downloading large restores will be as reliable as your web browser and connection. Other systems (like Crashplan) use their desktop apps so it can do multi-connection segmented downloads and more intelligently restart incomplete downloads without manual intervention. This may have changed since I looked at Backblaze years ago.

Corey - it's still a web-based system for backup, but in terms of preferences, you can specify which drives are backed up, or exclude drives/folders/files if you want. You can also schedule when to back up instead of continuous, as well as choosing faster network vs. faster backups.

For single files or docs, the web system works, but for big Pro Tools sessions or when I had to replace a 4TB drive, I just paid the $139 to get a 4TB with my data on it, transferred it, sent the drive back and got back my $139. I haven't used other systems so I can't compare what's better, but I've been happy with it. May be worth revisiting to see if it works for you.
Title: Re: Multitrack archiving
Post by: Tim Weaver on June 11, 2021, 03:32:31 PM
We (the church where I work) have Google services so I have an unlimited google drive where all the multitracks go. I didn’t pick it, but it mostly works. Its annoying that it zips files when you download so it creates a mess out of your tracks, but its pretty easy to clean up.

We store our multitraks in a folder for each sermon. Labeled by date with a dash 1 or dash 2 to denote which service.

So todays date would be 210611-1. Year/month/day-service. Doing it this way, year and mnth first helps keep your folders neat and organized. I start a new folder every year and just dump the folders containing the tracks and session in the year folder.
Title: Re: Multitrack archiving
Post by: Alex Cheng on June 12, 2021, 03:26:46 AM
I use Backblaze for backups - it's a pay one price regardless of how much you have to back up or how many attached drives. Restore can either be over the internet or they'll mail you a drive for $139. Restore your files, return the drive and your payment is returned.


One more vote for Backblaze. Good pricing, fantastic service, and Amazon S3 compatibility. They even go above and beyond and publish extended metrics on drive failures per manufacturer every year - an invaluable reference when purchasing disks for my personal use.
Title: Re: Multitrack archiving
Post by: Dan Richardson on June 12, 2021, 09:02:13 AM
Who is multi tracking shows (Dante/USB/AVB/etc) to DAW, and how are you archiving after the gig?

I've multitracked nearly everything for years, directly into Reaper projects so all tracks are named on the fly. Adding video to many in the last two years and going forward. I have a QNAP RAID 6 file server with 8 6TB drives in it, and another at a lower RAID with 8 4TB drives in an out building as a backup. The near one is live 24/7. The remote boots up, runs drive tests, and shuts down every night.
Title: Re: Multitrack archiving
Post by: Eric Eskam on June 15, 2021, 04:26:08 PM
I have a QNAP RAID 6 file server with 8 6TB drives in it, and another at a lower RAID with 8 4TB drives in an out building as a backup. The near one is live 24/7. The remote boots up, runs drive tests, and shuts down every night.

I just set a new QNAP TS-h973AX for a friend to consolidate a server onto (as a VM) and serve as file storage too for them.  Very impressive.  Their new QTS Hero version of their software is based on ZFS - and one thing that ZFS is awesome at is bitrot.  As long as you are mirroring or RAIDing disks, you get automatic and *continuous* bitrot detection.  If issues are detected, the parity information from your mirror or raid config is used to fix it on the fly.

Biggest reason to use ZFS, IMNSHO.

The downside to ZFS is it can be pretty resource intensive.  Keeping that in mind, to obtain max performance I bought a slightly bigger box so I could put in a pair of SSDs for cache and a second pair of SSDs for the System storage pool (system pool is automatically assigned to the first storage pool that gets created when you first set your box up).  If all you are doing is file sharing - and especially if you are just archiving files, then this level is not required.  Having said that, just having a pair of inexpensive 250GB SSDs dedicated to cache can make a substantial perf difference even if you put the system stuff on regular hard drives.  If you aren't interested in running virtual machines, but just doing file sharing/backups then they have ARM based units that are not quite half the cost of the one I referenced.  I use those at remote locations all the time.  And most of their units can take external expansion boxes so you can keep growing over time fairly economically without having to start over.

It’s a steeper hardware cost for the QTS Hero edition, but the data protection ZFS brings is second to none - and with bigger hard drives they are packing more data into the same space; always makes me nervous. 

A final tip for your own cloud equivalent backup - if you have a willing friend/relative on the other side of the country, dropping a box at their house gives you great geographic diversity.  Letting them back their stuff up to it too can sweeten the pot :)  The built in Qnap stuff for networking two boxes across the Internet had some security issues earlier this year - I connect mine over my own VPNs (piVPN - great solution).  If they can be trusted, the built in Qnap cloud assisted utilities really are slick and plug ’n play able for mere mortals.  If they fixed the issues from earlier this year :p
Title: Re: Multitrack archiving
Post by: Scott Holtzman on June 17, 2021, 03:57:30 AM
A plug for Amazon Web Services Simple Storage Service (AWS S3). They charge by the MB and it may be less than you think. I don't have huge audio files but all my pictures and other stuff I care about are backed up there and it runs around $2.50 / month. I love the command line interface. --Frank


I think we have talked about this before, you can push to Glacier, which you have to wait to "thaw" out.  It's really cheap.



Title: Re: Multitrack archiving
Post by: drew gandy on June 18, 2021, 11:17:33 PM
I prefer the idea of "mixed media" meaning that I want my data on at least two different kinds of media for long term storage. DVD is doable up to a certain size but for multitrack audio backup, Bluray is looking pretty good. Of course, for the longest term storage, transcribing it all to written music on acid free paper, stored in a long term controlled environment is one of the tried and true methods.

Whatever you do, don't give your archives to Universal Studios to store.  https://www.nytimes.com/2019/06/11/magazine/universal-fire-master-recordings.html (https://www.nytimes.com/2019/06/11/magazine/universal-fire-master-recordings.html)

(I've heard that saving digital data to full page QR style codes on paper is actually a fairly viable and low cost archive method. One advantage it has is that it simply needs an optical imaging device to "retrieve" as opposed to computer hardware of a certain vintage. I filed this in 'ideas to explore further' and hope to get back to it sometime this decade.)
Title: Re: Multitrack archiving
Post by: Ned Ward on June 19, 2021, 12:05:22 AM
I prefer the idea of "mixed media" meaning that I want my data on at least two different kinds of media for long term storage. DVD is doable up to a certain size but for multitrack audio backup, Bluray is looking pretty good. Of course, for the longest term storage, transcribing it all to written music on acid free paper, stored in a long term controlled environment is one of the tried and true methods.

Whatever you do, don't give your archives to Universal Studios to store.  https://www.nytimes.com/2019/06/11/magazine/universal-fire-master-recordings.html (https://www.nytimes.com/2019/06/11/magazine/universal-fire-master-recordings.html)

(I've heard that saving digital data to full page QR style codes on paper is actually a fairly viable and low cost archive method. One advantage it has is that it simply needs an optical imaging device to "retrieve" as opposed to computer hardware of a certain vintage. I filed this in 'ideas to explore further' and hope to get back to it sometime this decade.)

I can personally attest that DVD is a very temporary storage option - with prosumer burners, the dye fades after a while and you lose all your data... I know from experience from losing 10 DVDs of Pro Tools files, Opcode Studio Vision Pro Files, home movies, etc...
Title: Re: Multitrack archiving
Post by: Mark Oakley on June 19, 2021, 09:19:46 AM
I can personally attest that DVD is a very temporary storage option - with prosumer burners, the dye fades after a while and you lose all your data... I know from experience from losing 10 DVDs of Pro Tools files, Opcode Studio Vision Pro Files, home movies, etc...

That's why I'm using "M" disks-apparently they're good for 1000 years: 

https://en.wikipedia.org/wiki/M-DISC#:~:text=M-DISC%20%28Millennial%20Disc%29%20is%20a%20write-once%20optical%20disc,Inc.%20and%20available%20as%20DVD%20and%20Blu-ray%20discs.

-Mark
Title: Re: Multitrack archiving
Post by: Dan Richardson on June 19, 2021, 10:15:19 AM
apparently they're good for 1000 years:

We'll know if they were right in 3009.
Title: Re: Multitrack archiving
Post by: drew gandy on June 19, 2021, 11:47:57 AM
I can personally attest that DVD is a very temporary storage option - with prosumer burners, the dye fades after a while and you lose all your data... I know from experience from losing 10 DVDs of Pro Tools files, Opcode Studio Vision Pro Files, home movies, etc...

Were all of your losses from the same batch of discs? Same manufacturer? The same burner? I've heard many anecdotes about this kind of failure but haven't heard many details.

I've pulled some stuff off of 10yr+ old DVD archives and had no issues but I've only attempted a tiny fraction of what I have on DVD. Back in the studio days I tried to save everything on DVD-R as well as DVD+R discs in an attempt to diversify. In the next few years I will hopefully go through and transfer the most important stuff to different media in order to avoid the bit rot. If you consider it a 10 year archive form, and then think about how much tech has (or will) advance in that 10 years, it buys you time to find a better archive method and perhaps cull some of what needs to be saved.

Again, I think of DVD as only one form of media with which to attempt your longer term archive.
Title: Re: Multitrack archiving
Post by: Noah D Mitchell on June 19, 2021, 12:49:16 PM
That's why I'm using "M" disks-apparently they're good for 1000 years: 

https://en.wikipedia.org/wiki/M-DISC#:~:text=M-DISC%20%28Millennial%20Disc%29%20is%20a%20write-once%20optical%20disc,Inc.%20and%20available%20as%20DVD%20and%20Blu-ray%20discs (https://en.wikipedia.org/wiki/M-DISC#:~:text=M-DISC%20%28Millennial%20Disc%29%20is%20a%20write-once%20optical%20disc,Inc.%20and%20available%20as%20DVD%20and%20Blu-ray%20discs).

-Mark


I don't know... 'millennial disks' - aren't they (according to the news) most likely to go out and spend your data on avocado toast or get offended by something you stored on them?


I kid, I kid.


*dons flame suit just in case*



Title: Re: Multitrack archiving
Post by: Ned Ward on June 19, 2021, 02:19:51 PM
Were all of your losses from the same batch of discs? Same manufacturer? The same burner? I've heard many anecdotes about this kind of failure but haven't heard many details.

I've pulled some stuff off of 10yr+ old DVD archives and had no issues but I've only attempted a tiny fraction of what I have on DVD. Back in the studio days I tried to save everything on DVD-R as well as DVD+R discs in an attempt to diversify. In the next few years I will hopefully go through and transfer the most important stuff to different media in order to avoid the bit rot. If you consider it a 10 year archive form, and then think about how much tech has (or will) advance in that 10 years, it buys you time to find a better archive method and perhaps cull some of what needs to be saved.

Again, I think of DVD as only one form of media with which to attempt your longer term archive.

Multiple media vendors, Apple Superdrive in a 2009 iMac.
Title: Re: Multitrack archiving
Post by: Russell Ault on June 19, 2021, 02:22:53 PM
I don't know... 'millennial disks' - aren't they (according to the news) most likely to go out and spend your data on avocado toast or get offended by something you stored on them?

I kid, I kid.

*dons flame suit just in case*

Eh, I'll just leave this here (https://forums.prosoundweb.com/index.php/topic,175436.msg1619202.html#msg1619202).

-Russ
Title: Re: Multitrack archiving
Post by: Tim McCulloch on June 19, 2021, 03:02:41 PM
I propose a toast ( dodges flying, charred bread )...
Title: Re: Multitrack archiving
Post by: Dave Bednarski on June 21, 2021, 08:55:51 PM
I use Amazon Glacier for archival of multi-tracks, video footage, and personal backups.  I offload to local drives and annually when its slow push it up to AWS.

Yes, the prices and configuration can be a little head-scratching the first time but it otherwise works as advertised.  "Thawing" a file/folder out is relatively straight forward - just has a time period to wait.

Previously I used their AWS Snowball appliance - they send a large gray yeti cooler with some I/O ports and a Kindle screen that doubles as a UPS label.  Mostly recently I've used AWS Snowcone device which was a smaller 8TB rugged drive, $60 for 5 day "rental" and no data transfer fees in.  They've improved the software and user experience so no more command line interaction is required.

https://aws.amazon.com/snowball
https://aws.amazon.com/snowcone

You can also pre-config the settings on the bucket the inbound data is coming into so the files immediately transitions into deep Glacier storage.  I found AWS Support to be very helpful in ensuring I had it setup properly to minimize my bill.