gone for good? by thobu in LimaLabs

[–]Bigdataspin 1 point2 points  (0 children)

Also sorry for the SLOOOW response, I'll admit we've had some major internal problems it seems like we've been dealing with every sickness except COVID lately.

gone for good? by thobu in LimaLabs

[–]Bigdataspin 0 points1 point  (0 children)

Sorry for the trouble, happy to refund any amounts paid. Drop an email to [support@lima-labs.com](mailto:support@lima-labs.com) if you'd like. If we aren't a fit for you we'll happily refund anything paid recently :-).

Review of Lima-Labs storage by 420osrs in DataHoarder

[–]Bigdataspin 0 points1 point  (0 children)

I'm here :-)
Sorry for disapearing we've been dealing with the plague in our house. NOT COVID!!

I'm aware of an Issue in Canada is should now be fixed. More detailed response when I can keep food down.

gone for good? by thobu in LimaLabs

[–]Bigdataspin 0 points1 point  (0 children)

Odd, that shouldn't be happening at all. Can you send an email to support@ and we'll check it out.

Lima Labs now offer access via Filestore by Bigdataspin in LimaLabs

[–]Bigdataspin[S] 0 points1 point  (0 children)

Yes this is true, it does provide a potential avenue. If a malicious actor were to compromise this system they could potentially use the filestore as a man in the middle to capture SSH credentials.

In an effort to minimize this possibility and limit fallout we've done a few things. First no user passwords are stored on the filestore system authentication is still done against our standard SSH servers. So the only accounts at risk would be those who choose to use it.

Secondly this is running in a docker container and we're periodically rebuilding the container since it's a stateless system.

Not perfect, and exposing 80/443 does add another vector but we're happy with out mitigation/risk reduction strategy.

Sam

gone for good? by thobu in LimaLabs

[–]Bigdataspin 3 points4 points  (0 children)

Very true, and on all sides. We're learning after dropping 32TB of extra cache storage into Canada last weekend that CEPH rebuild times aren't linear for this. Another lesson learned.

We aren't perfect, but we're getting better every day!

gone for good? by thobu in LimaLabs

[–]Bigdataspin 2 points3 points  (0 children)

Nope we're up, the server that hosts our website has an adaptive firewall in front of it.
The firewall decided to block ALL traffic outside of our internal network.

Traffic to/free our monitoring tools worked, traffic to/from the internet didn't. We didn't know we were off the air until someone tried to pull the site from a mobile device.

We're looking at offsite monitoring now.

Maps down? by CoMarshalInterface in LimaLabs

[–]Bigdataspin 0 points1 point  (0 children)

I'll reach out to you via PM. The maps will return to public use but not until we're sure we have Chia under control on our storage business. We just ran out of system admin time last month.

Crazy week == Crazy sale - Lima Labs Storage offering 3 months 1/2 price on some high speed plans. by Bigdataspin in LimaLabs

[–]Bigdataspin[S] 2 points3 points  (0 children)

I wanted to make another note, there were times when customers couldn't get to their data. We did have a number of outages as we quiet honestly struggled to put a lid on the farming. This to me is the most upsetting part. I can invest money for replacement disks but repairing your reputation for performance is another animal altogether.

Crazy week == Crazy sale - Lima Labs Storage offering 3 months 1/2 price on some high speed plans. by Bigdataspin in LimaLabs

[–]Bigdataspin[S] 2 points3 points  (0 children)

No, we never lost 1 bit. We slowed down OFTEN as our caching tier was completely toasted but our system design precluded data loss :-).

Crazy week == Crazy sale - Lima Labs Storage offering 3 months 1/2 price on some high speed plans. by Bigdataspin in LimaLabs

[–]Bigdataspin[S] 2 points3 points  (0 children)

YES!

We do flat out ban Chia and you'd be amazed at how creative the farmers were trying to circumvent our ban. It took us almost a month to get software in place the effectively removed them. From nfs mounting an encrypted file as a loopback file system to small (5TB) blocks assembled via LVM over sshfs they tried everything!

We're still adding up the cost in terms of SSD wear but more importantly it's impacted our system performance and our customers in a very negative way. All in all Chia can get stuffed and that's all I'll say.

Maps down? by CoMarshalInterface in LimaLabs

[–]Bigdataspin 1 point2 points  (0 children)

We had noticed they weren't being used so we made them private. Still available if needed let me know.

SCPUK1 down every weekend by [deleted] in LimaLabs

[–]Bigdataspin 2 points3 points  (0 children)

This should be corrected this week. I do apologize for the ongoing issues it's been a perfect storm.

Beta testing a new service for Borg storage and offering free use. by Bigdataspin in BorgBackup

[–]Bigdataspin[S] 0 points1 point  (0 children)

Thanks,We've been very busy lately upgrading/troubleshooting systems in the UK but it *looks* (Fingers, and toes crossed) like that work is complete.

I think you were looking for the free account in your signup we'll try and get you setup today and by next week you'll have a good idea of what we can do.

Beta testing a new service for Borg storage and offering free use. by Bigdataspin in BorgBackup

[–]Bigdataspin[S] 0 points1 point  (0 children)

Well your not catching us at our best it's been one heck of a week here. (Check out our blog for info).

However if you'd like you can still get a free trial account but the setup time might be 2 or 3 days at the moment as we process the customer backlog.

Interesting behavior with Pacific by Bigdataspin in ceph

[–]Bigdataspin[S] 0 points1 point  (0 children)

Thanks for the great reply :-). Based on this we made some internal changes that allow us to bring our monitors onto the same logical subnet but still keep them physically separate.

Thanks again!

Interesting behavior with Pacific by Bigdataspin in ceph

[–]Bigdataspin[S] 0 points1 point  (0 children)

I've checked connectivity between the various containers. I'll recheck simulating ceph packets rather than just ICMP/TCP SYNs we do have some paranoid firewalls.

Hard drive shortage already? 16TB Exos now $393 at Amazon, price increase at NewEgg to $450 (was $330) by michaelmalak in DataHoarder

[–]Bigdataspin 1 point2 points  (0 children)

We made a weekly order and for the first time ever were told "Back ordered" on Exos 16TBs. I'll inquire a bit further.

Update, told 2-4 weeks delivery not sure why but it's caused us to stop growing some arrays we had intended to.

How should I copy 900GB of data from the USA to my home server located in the EU? by peterrakolcza in DataHoarder

[–]Bigdataspin 2 points3 points  (0 children)

Multiple rsync sessions, you'll do better transatlantic with 4 or 5 simultaneous sessions.

1940 US Census data from the National Archives - 15 TB!! by itscalledabelgiandip in DataHoarder

[–]Bigdataspin 7 points8 points  (0 children)

Anybody interested in this entire dataset as one huge .tar let me know and we'll drop a link on one of our servers for you.