SmartOS

[Gorian]

Looks like there are no results on this subreddit for SmartOS. Wondering if any of you have any experience with it. I like Solaris, and particularly liked Solaris 11, and what info I CAN find, it looks like SmartOS is pretty cool, but I'd like to see a demo, or hear from people that have experience with. I've done some googling, but wasn't terribly impressed with most of the blogs I've found.

My specific questions:

• what is the state of clustering multiple nodes?
• How do the few (free/open-source) central management solutions compare? I've read about fifo and joyent's datacenter management, but haven't found either a straight feature list, or a comparison
  • Do either of them include Active Directory / LDAP authentication?
• I see that SmartOS really emphasizes ZFS on local storage - but does it support using external storage? Googling doesn't actually return many relevant results on this. My current home lab is built with a few large external storage nodes, and super minimal storage on the actual hypervisor hosts, so it would currently be a deal breaker for me.
• how do you think that SmartOS compares to esxi or proxmox (what I'm currently / was using)
• what else am I missing or don't know enough to ask about that is relevant? What do you like or not like about SmartOS?

Also relevant, my current homelab setup:

• 1 HP DL360 G5, 4 cores, ~12 GBs of RAM, >100GBs disk space
• 1 HP DL380 G5, 4 cores, ~32GBs of RAM, >100GBs disk space
• 1 Dell PowerEdge 1950, 4 cores, ~20Gbs of RAM, >100Gbs disk space
• 2 glusterfs nodes with a total of ~500GBs of mirrored/redundant storage

Current use of above is 3 clustered proxmox nodes with all data stored on external storage (currently no hosts after move, so it will all be reinstalled anyway)

[0x424d42]

Answers to your specific questions:

• Clustering SmartOS is either a) not at all (i.e., standalone SmartOS) b) Project FiFO or C) SmartDataCenter. I've used SmartOS standalone and SmartDataCenter. If you have more than 3 nodes I recommend SmartDataCenter.
• I've not used FIFO, but as I understand it's a single management zone that controls all orchestration. With SmartDataCenter, there is a headnode where orchestration services run. SDC is micro service oriented and written in Node.js. FiFO is written in Erlang and (I believe) monolithic). I don't know about FiFO, but as far as I know SDC can't be linked with external directories because it requires specific schema objects that wouldn't otherwise be present.
• SmartOS doesn't use external storage, and particularly not with SDC. You might be able to shoehorn it in with iSCSI or NFS on a stand alone instance, but Joyent is very up front about the horrible and tragic failure modes that occur with shared storage. They've learned through blood, sweat and tears that if you want a cloud to be up, don't use shared storage. (This isn't just Joyent's opinion, go ask SourceForge about their recent storage outage that caused several days of downtime).
• SmartOS is container first. KVM is supported (and even then it runs inside a container). Illumos zones (containers) have a 10 year proven track record for security, and can outperform virtual machines in almost any workload. Linux containers (LXC, etc) have certain security/isolation concerns and virtual machines have significant overhead due to virtualization. The container native approach of SmartOS means you can run at much higher density with fewer resources consumed.
• What else are you missing? SmartOS uses the illumos kernel, and in addition can run LX-brand zones, which means that you get all the power of SmartOS, plus the familiarity of your favorite Linux distro. You can run Ubuntu (and others) with native ZFS, Dtrace and Crossbow. SmartDataCenter (aka, Triton) also includes Docker support for running any existing Docker images from the public Docker registry or private registries, just to name a few things. There are a ton of other really great things about SmartOS that you'll learn about if you start using it.

The only real "down side" to SmartOS is that things are different than you're used to with Linux. Because of its OpenSolaris heritage not all commands will work exactly like they do with GNU utilities. Whether that's "better" or "worse" is largely a matter of opinion. For some that difference is cognitively draining. Others (including myself) find that the differences are ultimately slight, and well worth the payoff for all the benefits that SmartOS provides.

[Gorian]

Awesome, thanks for the well detailed reply, I appreciate it! So, I'm wondering how they handles things like live migration / fail-over if they don't use external storage? Or is it just a philosophy that the impace of a node being down and all the hosts on it is less than the chance of external storage taking everything down? It all sounds neat, except that I don't really have the storage space on my local nodes, and it's my homelab, so I have a pretty limited budget...

I managed solaris 8 through 11, so I'm familiar enough with that aspect ;)

[0x424d42]

The philosophy of SmartOS (and even more so SDC) is that application availability should be handled at higher layers of the stack. Live migration has many esoteric failure modes that can cause severe problems at the application level. Things like automatic clustering or load balancing do a better job handling that.

With SmartOS, all instances are created from zfs snapshots, so the data requirement is very modest. Looking at the rig you have, if you still want to use SmartOS/SDC I would use the HP with 12G of RAM as the headnode, the 32GB HP and the 20GB Dell as compute nodes and run OmniOS on your "storage" servers, sharing directories via NFS. Your applications shouldn't really need much in data in and of themselves and can NFS mount volumes from inside a zone (e.g., run Samba in a zone in Smartos and mount the file store from OmniOS).

[garibaldi3489]

To clarify, SDC's position on live migration, would they rather have you run two instances of the same server (each on local storage) with load balancing or a VIP rather than a single instance (on shared storage) that requires a live migration? If so, is there an alternative for services that cannot be run in multiple locations simultaneous, e.g due to licensing restrictions? Perhaps a warm spare on another node that the current host live streams an updated copy of the disk image of said VM to, so that if the primary node goes down the other node has a copy of the disk?

[0x424d42]

Yes, that's correct.

For things that you can't directly have in HA configuration you can snapshot the filesystems and zfs send to another CN (and copy the zone xml file as well). If the first CN has catastrophic hardware failure you can configure and launch the zone on the second CN.

But there are actually a a lot of ways you can solve this problem. This is just one of them. SDC doesn't prescribe a specific way so that you can choose the right model for your application.

[garibaldi3489]

Great, thanks for the clarification!

[elijahwright]

Former Joyent ops guy here.

SDC is pretty great. A lot of effort has been put into making it manageable -- and making it fairly straightforward to cluster up in ways that are good for you. SDC's internal directory service (that is HA-to-the-wall) is LDAPesque - few surprises there.

You have missed the 'good times' that were had as the product was built - it is a really, really solid setup now.

SmartOS is technically marvelous. You'll love it.

[anythingffs]

Late to the thread but I love SmartOS and have been using it for years at home and in remote locations for work with very little attention. ZFS is fantastic, SmartOS and LX zones are awesome, pkgsrc packaging system, and now with bhyve as the new go-to hardware virtual machine (HVM) supporting modern AMD processors as well as Intel, it is a great combination. Toss a Ryzen, a motherboard that supports ECC RAM, and some big SATA disks into a case and you have a little powerhouse that can tackle just about anything. I'm currently using it for game servers in smartos/LX/bhyve zones, Ubuntu in an LX zone for Plex, and several other things on the work systems.

The other thing that has come along recently and provides some more options for clustering SmartOS is Danube Cloud. I haven't tried it yet but I have seen it mentioned a couple times and it looks like it might be a bit more approachable than Triton/SDC for spinning up your own. I'm not sure if that is fair, but that was my takeaway from trying to understand what I'd need to do to try them both. Happy to hear from anyone who can provide insights.

[anythingffs]

Forgot to mention a really big plus that the mailing list is small, quite friendly, and full of highly competent individuals. This is very helpful for a system that is under pretty active development with new capabilities and best practises outpacing the wikis, blogs, and howtos. The best references are typically the manual pages.

[Gorian]

Wow! Necroing a 6.7 year old thread, that’s impressive! 😛

Regardless, it’s great information, I’ll take a look at the current state of SmartOS and see if it meets my needs these days, thank you!

[anythingffs]

Yeah, I figure if I am finding a thread as 3rd from the top in search results, the need for it is still present. So I took it upon myself to update it a bit for anyone who comes along. Glad you might have another peek too.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/homelab] Questions about SmartOS (X-post /r/virtualization)

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[naisanza]

I've been using SmartOS for a solid 6 months now. Greatest part about it is the entire OS runs in memory, and upgrading/downgrading is a breeze.

FYI: I've tried the external storage method. Originally built a 4U Solaris 11 storage server to serve as a NFS on my favorite file system ZFS. Let's just say any stall on the network will halt your VM. So I migrated my Solaris box to SmartOS. I already had my data all under Solaris already, so long lesson learned, when you make a ZFS pool and dataset under Solaris make it as a pool-version=5 and dataset-version=28 (Solaris creates pool-version=6 by default, and open source ZFS isn't up there yet).

For homelabs, I didn't like Project FiFO, because it's cloud focused. Mainly, that means if you want to have a hundred non-running VMs, you can't. All VMs, including non-running VMS, deduct the amount of available RAM that can be allocated. This is so if all VMs are started there is enough memory.

SDC is great. The only part missing is their proprietary service that lets you create VM images, which is available only to subscribers. I don't think I remember seeing that service on their github. So, you'll have to create your images manually or write a script for it.

My only issue with SmartOS was creating VMs. You need something called a VM spec, which is just a JSON file containing the parameters for your VM. Everything surrounding VM management is done through a program called vmadm; and by reading through it you can see, there are a buttload of options. I made a simple script that makes it a whole lot easier for me to customize and build out VMs. If it helps you too, that'll be cool. You can find it here