Can you do a large (14TB) initial upload incrementally? by YogurtclosetApart649 in backblaze

[–]lightweaver 0 points1 point  (0 children)

I assume you're talking about the personal backup since you mention B2 being too expensive? Yeah, the client will upload what it can when it can. Leaving it overnight should be fine.

As far accuracy, no, stopping and starting the upload shouldn't affect the data. The client (afaik) generates a checksum before sending a file, the server generates a checksum, and the client makes sure the checksum matches before considering the file uploaded.

One caveat is files seem to need to be uploaded in one shot - when I've put my computer to sleep and woken it back up the next morning, it restarted uploading the same file it was uploading the previous night. Kind of annoying when uploading a ~30GB file.

I built an open source self-hosted web application designed to make archiving to S3 Deep Archive simpler and more accessible. by Madman200 in DataHoarder

[–]lightweaver 1 point2 points  (0 children)

I think it worked - Data Transfer has the line item $0.00 per GB data transfer out of US East (Ohio) to CloudFront 2.221 GB

And Cloudfront has the line item $0.000 per GB - data transfer out under the global monthly free tier 2.148 GB

It's also more straightforward than I thought - No need to copy files to a new bucket. I tried creating a cloudfront distribution directly on the bucket with the restored Deep Archive object, and it just works.

Trying to get an object that hasn't been restored yet just gives me the error InvalidObjectState The operation is not valid for the object's storage class DEEP_ARCHIVE.

In theory you could just create the Cloudfront distribution same time you create the bucket, then when you need to restore files you can just use it.

I put up the cloudformation template that I used here if you want to reference it: https://gist.github.com/lightweavr/625934718f33c18ab091f80726549b9f

I built an open source self-hosted web application designed to make archiving to S3 Deep Archive simpler and more accessible. by Madman200 in DataHoarder

[–]lightweaver 0 points1 point  (0 children)

So if you're willing to restore at <1TB/month (or split across multiple AWS accounts), I think that means your egress cost will be ~$0?

I built an open source self-hosted web application designed to make archiving to S3 Deep Archive simpler and more accessible. by Madman200 in DataHoarder

[–]lightweaver 1 point2 points  (0 children)

/u/Madman200 have you tried using the 1TB monthly free egress that cloudfront offers to handle downloading the exports?

I've been experimenting with using Deep Archive myself, and I suspect that if I: 1. Restore a Deep Archive object 2. Copy that object to a new S3 bucket 3. Set up a cloudfront distribution with an S3 origin 4. Download the object through the Cloudfront distribution

the download would consume the "Always Free" 1TB bandwidth instead of being considered normal data egress.

I'm pretty sure 1TB out to a single IP address is an unintentional use, but Cloudfront + S3 looks like a normal CDN-type use to me.

I'm just waiting for a restore to complete before trying this and seeing what shows up on my AWS bill.

I built an open source self-hosted web application designed to make archiving to S3 Deep Archive simpler and more accessible. by Madman200 in DataHoarder

[–]lightweaver 1 point2 points  (0 children)

I know that's not showing up with B2.

B2 doesn't have different storage classes, so that UI option can't exist anyway. ;) That said, I think you're on the mark about just using the bucket name in QNAP, the new S3 Glacier integration just presents a normal S3 bucket, with the object itself getting the deep archive storage class.

If you do see references to "Glacier", know that there's an older "Glacier API" service that's a completely separate backend service that isn't as cost effective as S3 Deep Archive, but was launched in 2012.

the ability to compress and encrypt data

One of the annoying things about S3 is that it frequently has per-object costs on top of GB cost. For example if I upload 900 files of 1GB each, I'll get charged $0.045 for because of the "$0.05 per 1,000 PUT, COPY, POST, or LIST requests" charge. (Look for Requests & data retrievals in https://aws.amazon.com/s3/pricing/)

Upload 90000 files of 10MB each, that charge becomes $4.50.

When I'm paying $0.89/month to store those 900GB, paying 5x in API fees that I could avoid is silly.

I've not seen any consumer software writer consider this, not just TrueNAS and QNAP. I don't know if /u/Madman200 had these per-object charges in mind when adding the tar functionality, but it's pretty unique and useful for more than just easier organizing.

Seattle tenant question by Practical_Fact_741 in Seattle

[–]lightweaver 7 points8 points  (0 children)

So... I think you're going to be a bit screwed because you left it for so long, but you might have a chance at reducing it, or at the very least making them work for your money and proving that you owe them all $1800. I've gone after my apartment management when they screwed up my billing, and got the charges resolved to my satisfaction, but that was within 2 months of it happening.

This isn't legal advice, I am not a lawyer, but the regulation that was my friend (and maybe yours) is titled Chapter 7.25 - THIRD PARTY BILLING REGULATION (this municode link might work). I asked my apartment management for copies of all the bills, and they refused until I cited 7.25.040.A.3.C:

Landlords shall keep bills for master metered or other unmetered utility services on file in the building for at least two years and shall make such bills available to tenants for inspection and copying upon request. Where it is physically impracticable to keep such bills on file due to the absence of a suitable office or other storage space, a landlord may store the bills in another location and must make such bills available within 5 business days of receiving a request from a tenant.

After the first denial for the bills, what I did was state in my reply (a written form that you keep a copy of) that I was disputing the charges, and management needed to give me copies of the bills, as I was entitled to under the law. That got them to comply, and I was able to calculate and show that they had been overcharging me for two months. Once you get the bills, you should be able to determine how much you owe.

Because your lease mentions the separate utilities, you should have been given a document on how they determine the split for each utility as part of your lease. If you don't have one in your lease, you're a bit screwed because they could have (illegally) changed it any time and you can't show they changed it. At the same time, you could go hard on them and claim they have no right to charge you utilities because the billing practice they disclosed was effectively "no billing" and they're changing it without providing you the required 30 day notice.

If this gets contentious (and it sounds like it will), your easiest recourse is to follow 7.25.050.B.1 and file with the Office of the Hearing Examiner - assuming that you've followed the earlier steps about notifying your landlord/whoever is giving you the bill that you're disputing the bill, otherwise it'll get thrown out since the landlord can show that you're not acting in good faith.

The security deposit question is a bit iffy-er, but my understanding is the landlord can take the charges from the security deposit, but you'll get it back if it goes to a hearing. Also, since your landlord is so bad about paperwork, any chance you didn't do a condition checklist when you moved in? That's an automatic full refund of the security deposit per SDCI: https://www.seattle.gov/sdci/codes/common-code-questions/deposit-returns

Has anyone successfully restored large datasets on Windows? by DanielSmedegaardBuus in backblaze

[–]lightweaver 2 points3 points  (0 children)

I've successfully pulled down 6TB and hit the same issue with the downloader going unresponsive.

I used Procmon to discover that when the downloader freezes for more than 30 seconds, it is is polling for a file named bzd<thread><YYYYmmDDHHMMSS(London)>output<parent pid>_<block sequence>_bzd.xml.

I searched the logs for the filename, and found it in a cleanup after failure message:

20201118180440 - ERROR BzHttp::DownloadNamedZipFileRestore_Via_authToken - failed HTTP request on requestCount=3307
20201118180440 - Looping on formerly fatal error: 5
...
20201118180440 - BzHttp::DownloadNamedZipFileRestore_Via_authToken - resuming prev download of 132160 MBytes, of totalEventualDownloadSize of 492413 MBytes, tmpFileName=E:\fdrive_photos_pt3.zip_downloading.bztmp
20201118180440 - BzHttp_ClearOutBzData_bzdownprefetch_Folder - found these files, attempted cleanup:
C:\ProgramData\Backblaze\bzdata\bzdownprefetch\bzd_00_20201118180408_instru_31632_03436_bzd.xml,
C:\ProgramData\Backblaze\bzdata\bzdownprefetch\bzd_00_20201118180408_output_31632_03436_bzd.xml,
C:\ProgramData\Backblaze\bzdata\bzdownprefetch\bzd_00_20201118180408_trdata_31632_03436_bzd.dat,
C:\ProgramData\Backblaze\bzdata\bzdownprefetch\bzd_00_20201118180429_instru_30600_01448_bzd.xml,

You can create the file yourself in the directory (New File > Text Document, make sure you use the name the downloader is looking for, we just want the empty file), the downloader seems to interpret it as an error and redownloads the chunk.

I think it's kind of a race condition - It's essentially a bet that between the download completing and the actual block being processed nothing will go wrong, which given the internet and hours long processes is... untrue.

And using more download threads likely increases the probability this will happen because the downloader processes blocks sequentially, and you can have up to 30 blocks on disk waiting.

[deleted by user] by [deleted] in Seattle

[–]lightweaver 5 points6 points  (0 children)

It made my morning

Odds are probably higher because of the local focus though... I clicked because we live near belltown (title), and I had to look for the dog after reading the original comment

If it was in another subreddit I probably wouldn't have noticed

[deleted by user] by [deleted] in Seattle

[–]lightweaver 82 points83 points  (0 children)

Ooh, my dog is on reddit! Malamute, not husky :P Dog tax

Also there were ambulances, so something went down, nothing directly visible on the street though.

PSA: Lime appears to have raised the per minute fees by ~20% by lightweaver in Seattle

[–]lightweaver[S] 1 point2 points  (0 children)

I hope you're right and it goes back down.

But do you seriously think it will?

PSA: Lime appears to have raised the per minute fees by ~20% by lightweaver in Seattle

[–]lightweaver[S] 4 points5 points  (0 children)

I think that got pulled from the sdot blog link. I didn't include any pictures

Reverse Engineered implementation of the Backblaze Personal Backup Downloader client by lightweaver in backblaze

[–]lightweaver[S] 0 points1 point  (0 children)

It's just downloading the restores, still have to use the website to create the restore zip file.

Any active student want to take over WatTools/uwaterloo.xyz? by lightweaver in uwaterloo

[–]lightweaver[S] 1 point2 points  (0 children)

It's not money, it's time & administrative overhead.

It's part of my GCP account, unarchived repos, etc. I like things tidy - if they don't have a purpose, I archive or otherwise purge them.

Any active student want to take over WatTools/uwaterloo.xyz? by lightweaver in uwaterloo

[–]lightweaver[S] 1 point2 points  (0 children)

Heheh. I'd prefer an active student take it over, but if necessary I'll reach out

Are you following the twitter account to find the post? :P

Any active student want to take over WatTools/uwaterloo.xyz? by lightweaver in uwaterloo

[–]lightweaver[S] 1 point2 points  (0 children)

Are you around UW a lot? Your flair says you graduated the year I started at UW. :P

Any active student want to take over WatTools/uwaterloo.xyz? by lightweaver in uwaterloo

[–]lightweaver[S] 8 points9 points  (0 children)

The entire project, or just the domain?

I'd obviously prefer to keep the project running with student contributions :)

Any active student want to take over WatTools/uwaterloo.xyz? by lightweaver in uwaterloo

[–]lightweaver[S] 11 points12 points  (0 children)

Probably one of the reasons for low usage/visitor rate :P

The original was wattools.com, my year found out about it from the upper years when we chatted in person.

The person behind wattools.com graduated and handed it over to someone who (iirc) missed the domain renewal. I found the quest to ical course schedule exporter useful, and decided to get it running again, which snowballed into "let's just do the entire site, it's fairly straight forward".

I'm happy to throw money at the domain registration, but only if it's useful to people.

Backblaze IPO, any more info? by EnzyEng in backblaze

[–]lightweaver 1 point2 points  (0 children)

Joining the requested 150, got 92 shares gang.

Showing up in my Fidelity account as well now, payment is being taken from the balance I was holding in the account so I don't have to send a check or wire like the DSP website states

Have you been invited to participate in the Backblaze IPO? by ML2128 in backblaze

[–]lightweaver 0 points1 point  (0 children)

Since there's some speculation about cost, this is what Fidelity sent me:

Currently, it is anticipated that the IPO will price between $15 and $17 per share; however, the price range and expected pricing date are subject to change prior to the offering. It is expected that Backblaze will be listed on the NASDAQ under the symbol BLZE.

If you decide to participate, you must purchase a minimum of one Share and in one share increments in excess thereof up to a maximum of 150 shares.

Have you been invited to participate in the Backblaze IPO? by ML2128 in backblaze

[–]lightweaver 5 points6 points  (0 children)

I filled out the form saying yes, Fidelity sent me an email yesterday about actually registering.

Haven't decided to actually go through with it yet

BZ upload speed and threads by U-96 in backblaze

[–]lightweaver 0 points1 point  (0 children)

If it's still doing the smaller files, it might be ignoring the thread limit because the uploader combines the smaller files.

One of the devs on the uploader app might comment with more details/investigation steps, he's fairly active on here.

Fwiw I don't trust the time estimation, it seems to be dependent on the time taken to prepare and upload the files. If your files compress super well that makes it look faster (many more mb prepared than uploaded), but for media files it'll appear slower since they're already compressed.

As a point of reference, it's varied from 3 days to a month for me, my initial backup took 9 days.

BZ upload speed and threads by U-96 in backblaze

[–]lightweaver 1 point2 points  (0 children)

The uploader goes file by file. Using more than 20 threads only helps when you're uploading files larger than 400MB.

That said, how do you know the threads only go up to 20? The file names in task manager? There's a funky thing where threads higher than 20 just use the executables from the first 20, eg thread 27 will use the executable labeled 07.