Cloud Backup; Backblaze in the Real World
Most of you know by now I’m a big fan of cloud backup, and specifically Backblaze — and in full disclosure I do not take any advertising or promotional revenue from them, however I do often post my affiliate link. The only bill I’m paying with that is maybe dinner once a month, so I’m not making a living suggesting you use this service. I’m suggesting it because it rocks.
That said, a lot of users say “sounds good but I don’t have the bandwidth to use it for my photo files”. Hogwash. And here’s why, proven by simple math.
File transfer speeds
Let’s say you have a pretty standard high-speed internet connection giving you 2 Mbs upload speed. That seems to be pretty common from what I’ve seen.
Using the Download Time Calculator, a 1 GB file will transfer at 2.048 Mbps in just over 1 hour. So let’s just call it a gig per hour if you’re getting full bandwidth and you’re not using your connection for anything else.
Initial upload
Yes, it’s true, your first upload could take weeks or even months to finish. Let’s say your internal 1 TB HD is ½ full, so 500 GB. 1 GB per hour, that’s 500 hours ÷ 24 hours = 20.8 days. OK so that’s at full-tilt, and obviously you will be using your internet connection for other things, so let’s be generous and add a 50% overhead… that’s still only 30 days. One month and 500 GB is backed up.
Yeah but you have Terabytes of data! OK, fine. 2 TB of data = 2,000 GB = 2,000 hours ÷ 24 = 83 days = 2.7 months, plus 50% and we’re still looking at just over four months for the initial 2 TB backup.
How long have you had your computer? How long have you been putting off backing up? More than four months? Sure you have. You could have been backed up completely by now, probably several times over.
Subsequent backups
On Monday I was on a small shoot and brought home 15 GB of data. I have a 10 Mbps upload speed (admittedly faster than many), and Backblaze reports about 8 Mbps average. Check out this screenshot:
Backblaze statistics on a 10 Mbps connection; 8 Mbps and 84 GB/day average
84 GB per day. That’s a lot of data. So my 15 GB shoot…
15 GB data, backed up overnight…was backed up in one evening
All backed up
According to my Backblaze statistics, I have over 3 TB of data on their servers. And I keep adding more.
According to Backblaze, I have over 3 TB stored with them
The cost of security
Backblaze is cheap. $5 per month (per computer) if you pay monthly, or under $4 per month if you pay two years in advance. That’s for unlimited storage, and they allow backing up of external drives. One caveat is that if a drive isn’t plugged in once per 30 days, then the backups will be removed from the server, however for your normal every-day attached drives, then the sky is the limit. If you’re going on African safari for two months you may want to leave your computer on at home or have someone boot it up every few weeks, but other than that, it’s set it and forget it.
Day to day use, and a note on performance
For most users, you can set it to fully automatic, and every so often a Backblaze daemon fires up and scans your drive for new material to backup, then starts pushing new content to the servers. With the auto-throttle turned on, you are unlikely to ever notice that it’s happening. If you happen to be uploading a massive file and want full bandwidth for that, you can always pause a backup that may be slowing you down.
For really intense users such as myself, it’s possible that you’ll find sporadic slowdowns while working (most noticeably in Aperture, unfortunately), when the Backblaze scanner kicks in. It’s not the upload that slows you down, but the scanning of the hard drive, looking for new files to back up. For this reason I’ve switched my backup routine to only kick in when I click “Backup Now”. The engineers at Backblaze recently added a “silent mode” feature where if you’re in the “Only when I click <Backup Now>” mode, then the scanner does NOT run throughout the day.
Backblaze set to backup only when I click “Backup Now”
If you’re on a scheduled mode, either “Continuously” or “Once Per Day”, the scanner will continue to run on its own schedule sporadically throughout the day. Yes it’s a little less convenient to have to manually click the Backup button, however it’s worth it for me to not have the occasional performance hit.
I’d suggest that you don’t need to do this unless you are seeing a noticeable performance hit. It’s easy to know if Backblaze is causing a slowdown; just open the Activity Monitor and search for “bzfilelist”. If it’s cranking away at the same time you’re experience slowdowns, that’s the cause. It may be time to switch to manual.
The process “bzfilelist” is actively scanning my drive now
Stop procrastinating and just get it
The peace of mind of a true offsite backup can’t be overstated. You should still have a local backup — after all it’s better to have a local copy of your 2 TB drive to recover from than to have to download 2 TB of data in the event of — but knowing that if your house burns down, someone steals everything in your office, or an earthquake or tornado takes out your entire city (including your house and the location where you currently keep your “offsite” backup), knowing all your data is safe elsewhere is priceless.
Get started using this affiliate link, if you don’t mind. You won’t regret it.
Comments
on May 6, 2012 - 12:53am
Joseph do you have a script that exports all 5star images to a folders that would exist on internal drive. One that runs or looks for new 5star images when ever I make them. My images are on the drobo but this would make a copy back on the internal HD for Backblaze. I could later add 4star images to the scrip, thus allowing Backblaze to get the most important first and keeping it automated. thanks dbm
davidbmoore@mac.com
Twitter= @davidbmoore
Scottsdale AZ
on May 6, 2012 - 1:16am
Two notes on Backblaze:
1. They also have a “Find My Stolen or Lost Computer” feature you can enable at no extra cost. (kinda like Find my iPhone)
2. Your paragraph on The Cost of Security is a little misleading. The 30 day limit for disconnected drives can easily be overcome. If you’re going on that African Safari for several months, just make sure your external drive is connected “before” you shut down your computer. Backblaze only starts the 30 day timer when you unplug an external drive. If you shutdown your computer with the external still connected, Backblaze will not automatically delete the data after 30 days. In the FAQ it is worded this way: “The 30 day countdown is only for drives that have been unplugged.”
Thomas
on May 6, 2012 - 6:41am
How do you know that THEY are backing up your backup in the event that their servers catch fire?
Also, here in NZ all internet plans are data capped and you would chew through that in no time doing this!
on May 6, 2012 - 11:25am
Thank you, Joseph, for your explanation of BackBlaze. I recently subscribed (2-year) but would like to make a few comments for those who might not have a relatively high-speed Internet upload. In my ISP’s 4 tier system, I subscribe to the 2nd fastest, which is 5MBPS down, 0.6MBPS up.
1. Consequently, my initial backup of 1.8GB is taking over two weeks to complete.
2. I must turn-off BackBlaze to stream video, download large files, etc. In fact, it even slows down standard e-mail significantly so I also turn it off for web browsing and email.
3. Although I leave my system on 24/7 with about 2 - 4 hours of pausing per day, the 0.6MBPS upload is a snail’s pace.
4. I am completely satisfied with the backup and security BackBlaze provides, but for those with low speed upload please take note of the time involved plus every time files are changed during the backup, those must be included since it is a dynamic not static backup.
If I am incorrect in any of the above statements, I am sure you or your readers will let me know. Thanks.
on May 7, 2012 - 6:22am
DBMoore — No script, there’s no way to have an AppleScript constantly watching an Aperture Smart Album which is what you would need to do that.
Thomas — That’s great about the external hard drive, I didn’t know that. Obviously I didn’t read the FAQ ;-) Thanks!!
Marcus — they are a company who’s sole purpose is to back up your data. I think if they weren’t backing up, they’d only be one mistake away from going out of business. I think it’s safe to say that whatever they are doing is better than what you or I are doing :)
wmm5 — yep if your connection is that slow then of course it will take longer. Glad to hear you’re satisfied with it though even with your slow connection.
@PhotoJoseph
— Have you signed up for the mailing list?
on May 7, 2012 - 2:00pm
How do you know that THEY are backing up your backup in the event that their servers catch fire?
Also, here in NZ all internet plans are data capped and you would chew through that in no time doing this!
—
Good point Marcus. These telcos are truly pathetic by capping your data use. They need to install the gear anyway, might as well have your customer use it to its fullest. Same should apply to mobile usage. And no, I wouldn’t trust cloud computing as my only storage location. Learn from Microsofts’ Sidekick disaster.
on May 8, 2012 - 12:03am
Folks,
No one we said to use cloud backup as your ONLY solution. It’s one piece of the puzzle. And I think it’s safe to say that statistically the cloud solutions are FAR safer than one at home. How many people lose data, every day, at home — and how many cloud services have suffered catastrophic failure in the last few years?
Have a local backup and a cloud backup and you’re about as safe as you can be.
@PhotoJoseph
— Have you signed up for the mailing list?
on May 9, 2012 - 10:51am
I realize that once I get all my data up - it’s good, but my problem is getting all my data up to the cloud. I wish I could just mail a drive for a small fee and have them copy it over and then get all that drive mailed back. My problem is keeping my external drives tethered to my laptop long enough to ever get there, and also the bandwidth caps that I generally have to deal with. (Be it Comcast or the dorms in college). When I figure out a good for me to get everything online, I’m going to do it. (There’s a lab at berkeley which has 250Mbps both ways, so I may end up spending some time there…)
Joseph, how does backblaze handle open files and system files? Will it back those up? I have a friend who has crashplan and it won’t touch files that are in use or system files, which are two things that I’d like backed up. (It would also be a problem for my aperture library, since it is almost always open.)
When it does subsequent backups - is it like time machine or dropbox where you still have version control, or does it overwrite things? If it does indeed keep versions, Backblaze seems better and better!
on May 9, 2012 - 11:39am
I also have to say that Apple are missing a serious trick by not incorporating this sort of functionality into iCloud.
Of course it would cost more but if they incorporated iCloud Vaults onto Aperture 4 and a “pro Account” for iCloud, they could do well I would say.
on May 10, 2012 - 4:04am
Michael,
You’ll need to check with Bb to see how they handle open files. I’m not sure.
They do keep version though, for four weeks from what I see: [screenshot]
-Joseph
@PhotoJoseph
— Have you signed up for the mailing list?
on May 12, 2012 - 8:51pm
Hi Joseph,
Good article. I had backblaze in my sights for a long time, but eventually got in on the beta for Bitcasa. They have unlimited storage and bandwidth and so far I am very happy with it.
I recently did a diagram of my home setup (without my laptop and IOS devices) to keep track of what I had setup and to visualize it.
As very few online storage services are made to run on NAS servers (and the once that do exist cost an “arm and a leg”) I decided to mount the NAS drive that I was using for backup on my mac mini server, and it would then run the online storage client. Have a look at the before mentioned diagram here: link Note! it is a vector image, so just push “view in my browser” when asked.
This gives me blazing speed on the 4disk raid0 thunderbolt array and 1000mb network connection to my NAS for the aperture vault backup, the mac mini then handles the actual upload to get my backup off location. :)
on May 14, 2012 - 4:14am
Great article Joseph. I love this web site and read it regularly. I, like so many of my friends are Comcast customers and have a 250GB monthly limit. (My monthly usage is currently 60GB.) With about 2TB of data to upload, Comcast would be calling…. and then billing me per GB over the limit. Without the ability to throttle data with Backblaze during the initial uploading period, is a big issue for many of us.
Also, I regularly achieve 24 Mbps in Download Speed. My average is 20.4 Mbps. (Smokin fast, Right?)… However, Upload speeds are RARELY over 5 Mbps. I average 3.5 Mbps… and that’s faster than the average person. Joseph, you said you are getting 10 Mbps? Are you sure that is your UPLOAD speed? If so, that’s really exceptional.
Lastly, with regard to backup, I’ve lost data. I learned the hard way. So I regularly backup. I have a DROBO for my Final Cut media, iTunes, and Aperture vaults (5TB), I also have TimeMachine and Superduper backing up my Mac, User, Aperture data, etc… on two different drives. (My TimeMachine backup drive just died after 21 months.)
I would love to backup to the cloud. It’s the best solution. But with Comcast capping our data limits, our options are limited.
Thanks- Alex
ACS
on May 15, 2012 - 12:28am
We’re in the situation of wmm5 - about 750GB (currently) in the library. But our cable broadband while offering about 7.5Mbps download only runs at .53Mbps upload. A very rough estimate would put us at about 4 months for the initial backup, and of course we’d be playing catchup for some time due to changes to the library as it was being backed up. And honestly there’s a decent chance there will be at least one provider outage or other problem in four months.
However: I seem to recall that Backblaze has an option where they will mail you a USB drive or equivalent of your backup to get it back quickly if you have a real disaster.
So: does Backblaze (or anybody) offer a “prime the pump” solution where you could run the initial backup to a (perhaps vendor-provided) USB drive or equivalent and express-ship it to them for initial loading of your configuration? That would mean maybe a week’s delay before the subsequent sync, and it would either be a raw copy or they’d have to incorporate such a feature into their client – but it would be an absolute winner for Aperture users. I’d be happy to pay well for such an option as it would let us get into the Backblaze world conveniently and quickly.
on May 15, 2012 - 2:11am
Alexander,
I understand that many ISPs do have data caps. You can throttle the backup if your upload performance will push you into exceeding the cap. Go to the Backblaze settings in the System Preferences, and click on Performance. You can turn off Automatic Throttle and set it to manual. It’s not a hard-limit throttle, although I like that idea and will suggest it to their developers. For example, if you know you have 250GB and usually use 60GB, you could set a max of something like 150GB upload, just to be super safe. I’ll recommend it!
And yes, I do get close to 10Mb up. I pay for small business fiber connection of 25/10 provided by the city of Ashland. I usually break 25 on the down, and up is usually in the 7 to 8 range when testing. I push my backup at night, so the performance is usually quite high.
That said, I have a friend in NY who has a 25/25 fiber connection provided by Verizon, I think? I know these are exceptional speeds, but the fact is it’s getting more and more prevalent.
I think options aren’t exactly limited, you just have to be a bit more careful in the initial upload. So what if it takes six months to get your initial backup done… if you start today, in six months you could be backed up in the cloud. If you don’t, then in six months, you’ll still be thinking about it.
Jus’ sayin’ ;-)
-Joseph
@PhotoJoseph
— Have you signed up for the mailing list?
on May 15, 2012 - 2:36am
Dave,
Yes, Backblaze will send a drive to you if you can’t download your backup. There’s a fee of course, but if you can handle the download, that’s free.
As far as sending in a drive to get the backup started, yes there are services that do that however I’ve looked into them and ended up sticking with backblaze. I think the reason was there’s a max to what you can send in, like 1TB or something, and with 4TB to backup it just wasn’t worth the cost or effort.
-Joseph
@PhotoJoseph
— Have you signed up for the mailing list?
on May 15, 2012 - 11:15am
Thanks Joseph for this post. I’ve been debating for way too long if I should go ahead and subscribe to some cloud backup service. I’m wondering how Backblaze handles differential backups, though. The Aperture library is stored in a single file.
So is Backblaze able to identify incremental changes that I make to my library without requiring that I upload the whole library each time?
In any case, I agree with a comment earlier that says that there’s a good opportunity for Apple to come up with an integrated solution at this point, that could combine local (Time Machine) and cloud backups (iCloud). Let’s see if they make interesting announcements along those lines at the WWDC, coming up in June.
Tony Mignot
on May 15, 2012 - 11:52am
Tony,
Actually the Aperture library is NOT a single file. It’s a “package”, which contains thousands of smaller files. I know Backblaze work perfectly for Aperture Libraries because back when they first came out, I was one of their earliest users and worked with their CTO to ensure backups worked properly. They didn’t at first — but that was addresses years ago.
-Joseph
@PhotoJoseph
— Have you signed up for the mailing list?
on May 15, 2012 - 1:04pm
Cool. Thanks for the quick response Joseph.
Tony Mignot
on June 20, 2012 - 2:04am
I’m catching this conversation 2 months late. But perhaps someone would be kind enough to reply to a question. By the way, yesterday I subscribed to Carbonite (due to their effective advertising on key podcasts I follow). Today I find in the Apple’s Aperture discussion forum that their are significant problems with Carbonite and database-based application files, such as Aperture and iPhoto. Oh well, with no refunds, I’ll use it for a year for non-Aperture backups.
My question (now that I’ll look into Backblaze) are:
1. Is it better to backup the Aperture “library” file or an Aperture “vault” file?
2. Would backing up a “vault only” mean that Aperture could be used while Backblaze is backing up, since it would not be impacting the active Aperture library file?
3. Or, is that irrelevant because the impact on Aperture’s performance is more due to CPU drain in general? (I shut down EVERYTHING when working on Aperture for that reason.)
Thanks.
Bill
on June 20, 2012 - 10:22am
William,
I didn’t know Carbonite had that problem. Bummer. When BackBlaze first came out they did too, but I worked with them directly and they made the fixes years ago.
I backblaze my library, not the vault, but you could do either or both.
To be honest I haven’t tested a backup that’s running while the Aperture library is open, but it should be fine. Other than writing the sharing xml file on quit (if you have the sharing pref to update on quit) and writing the preferences on quit, I don’t think Aperture does anything else specific on quit. So even if you were mid-backup and mid-edit and all hell broke loose, I think the worst you’d have is some missing metadata or adjustments. I would think. I have not tested this.
I run BackBlaze manually on my main system because i have so many files backed up and backing up (over 3.5TB) that BB really hogs my system when its running if Aperture is too. But on my Air, which is of course much smaller, I leave Bb running on full auto.
I’d just let it run, and if you see speed hits, then go manual. You can’t go to timed, because that’s almost the same as running auto — Bb will scan your drives on regular intervals prepping for the next backup. The timed backup is more about timing it to not hog internet, not to not hog resources.
@PhotoJoseph
— Have you signed up for the mailing list?
on June 20, 2012 - 11:00am
Hi William, Joseph
I am currently using Bitcasa, which is working out well for me.
http://alf.ikikata.org/2012/05/backup-strategy-visualized
I recommend offloading your library to a different system if you have one. (like a Mac mini running as a server or media center etc.)
That way you can set a max upload on that system and it wont suck all the bandwidth, plus it frees up resources on your own system so you don’t feel a strain when working in Aperature while the backup is running, and it gives you a on-site backup of your library or vault plus the ability to shut down your main workstation when you want, and the backup will still happen in the background through the server. :)
on June 21, 2012 - 6:01am
Did you find out how to back up the only Aperture library? Almost everything is backed up by default, you have to manually exclude.
on July 20, 2012 - 5:04am
Don’t want to be a Debbie Downer but just want to offer a warning to Backblaze users.
I have used Backblaze since late 2008. I had over 1.5TB backed up to Backblaze servers including my Aperture libraries. Like Joseph, I was a very happy Backblaze customer. Then I had a problem and lost my backup. I learned a few lessons that I thought I would share.
If you are a Backblaze user I suggest you immediately check the size of the bzfileids.dat file. This file is located in the /Library/Backblaze.bzpkg (package file) at /Library/Backblaze/bzdata/bzbackup/bzfileids.dat
This file can not exceed 1GB in size. If it exceeds 1GB you will get a
Backblaze Notification -
“WARNING: Backblaze has stopped working properly on this computer! Your bzfileids.dat file is too large, please contact Backblaze customer support.”
In my case I had to restart my backup from scratch (at 30-40 GB/day this will take 1-2 months).
The following is a long explanation of what I learned in dealing with Backblaze Support (Ken and Yev). They have been excellent in answering my questions.
The bzfileids.dat file is a simple file that maps your filenames to a unique integer ID that is used to anonymize your files on the Backblaze servers. The size of the bzfileids.dat file is primarily dependent on the number of files you are backing up and the length of the full path name of those files. Long file names, long folder names and deep folder hierarchies all make this file larger.
I was told that my 1.5TB backup was not at all unusual. They had many users with larger backups. I was backing up almost 2 million files. They said this was very unusual. Less than 1% of Backblaze users are backing up that many files. Digging into this I discovered that my Aperture libraries have lots of files.
I have two major Aperture libraries (managed).
Library 1 - 23,500 images, 297GB - 427,200 files
Library 2 - 22,000 images , 96.4GB - 364,200 files
So my Aperture libraries account for almost 800,000 of my 2 million files backed up.
I can only assume having 16-18 files per image is a function of Aperture and not unusual (???).
This problem occurred immediately after I upgraded Aperture to v3.3. I am not exactly sure why. I understand that the bzfileids.dat file can also grow in size when you move around large number files or change the name of a high level folder. I am not sure what the v3.3 upgrade did to the file/folder names but I know it took a LONG time to upgrade my libraries so I can only assume lots of stuff was changing inside the Aperture libraries. (Looking back thru my Time Machine backups I could see that the bzfileids.dat file had grown steadily over time. It was over 900MB the week before I did the v3.3 upgrade)
The bzfileids.dat file will not shrink. Even if you exclude a bunch of files the bzfileids.dat file will not shrink in size. Once it get’s big, it stays big. Starting your backup over (and exclude some files) is the only option to reduce it’s size.
Backblaze said they were working on a “larger-scale fix” of this bzfileids.dat issue but no ETA. I also suggested they add some warning (email/popup/etc) that the bzfileids.dat file is getting very large and you could lose your backup in the near future.
Suggestions:
If your bzfileids.dat file is no where close to 1GB in size don’t worry about this issue.
If your bzfileids.dat is getting large (hundreds of MBs) I would seriously look at what you are backing up and exclude unnecessary files. Unfortunately this won’t shrink it’s size but it might help keep it from growing as fast. Aperture previews would be one easy folder to exclude. There may be other non-Aperture files that don’t really need to be backed up. You will have to do the sleuthing. I did some digging to better understand my files and made one big discovery (read on if interested)
FYI - I also discovered that of my remaining 1.2 million files, over 1m were related to Evernote (great program that I have also used since 2008 - highly recommend). I found two Evernote folders in my user library folder that contain the local copies of my 64,000 notes. These folders contained over 1m files totalling only 23GB. (I am still trying to figure out if that is a problem or not.) Anyway I should have never been backing up those folders to Backblaze since they are just local copies of what is stored on Evernote’s servers. If I had originally excluded these folders I would have avoided this issue with the bzfileids.dat file size (for a while longer). I uninstalled Backblaze then reinstalled Backblaze excluding these Evernote files and my bzfileids.dat file is now only about 200mb.
FYI - While dealing with issue and worried about losing my cloud backup I researched other cloud backup services. I signed up with Crashplan (free trial). I am really liking it. It is the same price as Backblaze for unlimited backup (maybe cheaper with a family plan). It offers a ton of other options (which are a bit bewilderingly compared to Backblaze’s simplicity). I really like explicitly choosing what I want to backup vs. Backblaze’s choose everything then add file exclusions. The killer feature is that Crashplan has an iPhone and iPad app (Android too) that let’s you access all your backed up files. I have no complaints about Crashplan so far. Some of the reviews have complaints about support issue but they were a few years old (Backblaze support is always great.)
Good luck - hope you don’t have this issue.
Jim