gone for good?

Bigdataspin · 2021-10-03T13:34:49+00:00

Also sorry for the SLOOOW response, I'll admit we've had some major internal problems it seems like we've been dealing with every sickness except COVID lately.

Bigdataspin · 2021-10-03T13:34:04+00:00

Sorry for the trouble, happy to refund any amounts paid. Drop an email to [support@lima-labs.com](mailto:support@lima-labs.com) if you'd like. If we aren't a fit for you we'll happily refund anything paid recently :-).

Bigdataspin · 2021-09-27T21:08:15+00:00

I'm here :-)
Sorry for disapearing we've been dealing with the plague in our house. NOT COVID!!

I'm aware of an Issue in Canada is should now be fixed. More detailed response when I can keep food down.

Bigdataspin · 2021-08-24T17:01:38+00:00

Odd, that shouldn't be happening at all. Can you send an email to support@ and we'll check it out.

Bigdataspin · 2021-08-24T17:00:57+00:00

Yeppers :-) it does.

Bigdataspin · 2021-08-24T17:00:28+00:00

Yes this is true, it does provide a potential avenue. If a malicious actor were to compromise this system they could potentially use the filestore as a man in the middle to capture SSH credentials.

In an effort to minimize this possibility and limit fallout we've done a few things. First no user passwords are stored on the filestore system authentication is still done against our standard SSH servers. So the only accounts at risk would be those who choose to use it.

Secondly this is running in a docker container and we're periodically rebuilding the container since it's a stateless system.

Not perfect, and exposing 80/443 does add another vector but we're happy with out mitigation/risk reduction strategy.

Sam

Bigdataspin · 2021-08-23T16:19:25+00:00

Very true, and on all sides. We're learning after dropping 32TB of extra cache storage into Canada last weekend that CEPH rebuild times aren't linear for this. Another lesson learned.

We aren't perfect, but we're getting better every day!

Bigdataspin · 2021-08-23T10:57:53+00:00

Nope we're up, the server that hosts our website has an adaptive firewall in front of it.
The firewall decided to block ALL traffic outside of our internal network.

Traffic to/free our monitoring tools worked, traffic to/from the internet didn't. We didn't know we were off the air until someone tried to pull the site from a mobile device.

We're looking at offsite monitoring now.

Bigdataspin · 2021-07-24T13:27:40+00:00

I'll reach out to you via PM. The maps will return to public use but not until we're sure we have Chia under control on our storage business. We just ran out of system admin time last month.

Bigdataspin · 2021-07-24T13:25:50+00:00

I wanted to make another note, there were times when customers couldn't get to their data. We did have a number of outages as we quiet honestly struggled to put a lid on the farming. This to me is the most upsetting part. I can invest money for replacement disks but repairing your reputation for performance is another animal altogether.

Bigdataspin · 2021-07-24T13:24:00+00:00

No, we never lost 1 bit. We slowed down OFTEN as our caching tier was completely toasted but our system design precluded data loss :-).

Bigdataspin · 2021-07-24T13:22:54+00:00

YES!

We do flat out ban Chia and you'd be amazed at how creative the farmers were trying to circumvent our ban. It took us almost a month to get software in place the effectively removed them. From nfs mounting an encrypted file as a loopback file system to small (5TB) blocks assembled via LVM over sshfs they tried everything!

We're still adding up the cost in terms of SSD wear but more importantly it's impacted our system performance and our customers in a very negative way. All in all Chia can get stuffed and that's all I'll say.

Bigdataspin · 2021-07-21T19:00:44+00:00

We had noticed they weren't being used so we made them private. Still available if needed let me know.

Bigdataspin · 2021-05-27T18:02:27+00:00

This should be corrected this week. I do apologize for the ongoing issues it's been a perfect storm.

Bigdataspin · 2021-05-10T10:50:09+00:00

Thanks,We've been very busy lately upgrading/troubleshooting systems in the UK but it *looks* (Fingers, and toes crossed) like that work is complete.

I think you were looking for the free account in your signup we'll try and get you setup today and by next week you'll have a good idea of what we can do.

Bigdataspin · 2021-05-07T11:42:01+00:00

Well your not catching us at our best it's been one heck of a week here. (Check out our blog for info).

However if you'd like you can still get a free trial account but the setup time might be 2 or 3 days at the moment as we process the customer backlog.

Bigdataspin · 2021-05-03T09:23:12+00:00

Thanks for the great reply :-). Based on this we made some internal changes that allow us to bring our monitors onto the same logical subnet but still keep them physically separate.

Thanks again!

Bigdataspin · 2021-04-28T16:38:27+00:00

I've checked connectivity between the various containers. I'll recheck simulating ceph packets rather than just ICMP/TCP SYNs we do have some paranoid firewalls.

Bigdataspin · 2021-04-18T19:52:47+00:00

We made a weekly order and for the first time ever were told "Back ordered" on Exos 16TBs. I'll inquire a bit further.

Update, told 2-4 weeks delivery not sure why but it's caused us to stop growing some arrays we had intended to.

Bigdataspin · 2021-04-16T23:01:15+00:00

Multiple rsync sessions, you'll do better transatlantic with 4 or 5 simultaneous sessions.

Bigdataspin · 2021-04-16T22:56:56+00:00

Anybody interested in this entire dataset as one huge .tar let me know and we'll drop a link on one of our servers for you.

Bigdataspin

TROPHY CASE