Kubernetes: One Orchestrator to Rule Them All
James Kempf and Danjue Li, Edge Services, Equinix
At a recent Out of the Box Networking Meetup, Ramki Krishnan from VMWare gave a great talk, “ONAP & the K8s Ecosystem: Toward a Converged Edge Application & Network Function Platform”. He started the talk with a slide showing how Kubernetes has slowly begun to take on orchestration tasks, which previously were the job of other system control components. Kubernetes started as a cluster container orchestration solution. But with the addition of CRDs (custom resource definitions) and kubevirt for managing virtual machines, Kubernetes has broadened out to orchestrate VMs. Kubernetes has already muscled out other container orchestrations systems like Docker Swarm (leaving Docker the company in deep trouble and without a viable business model) and has far more mindshare than Mesos and Marathon. Most companies that previously offered their open-source container orchestration are now also offering some form of Kubernetes. The big public clouds are embracing Kubernetes and now have Kubernetes service offerings. The on-premises companies such as Pivotal and Redhat have also caught up and have offerings for private and hybrid cloud. While old software rarely dies, the prognosis for the other container orchestration solutions is not good.
When will Kubernetes achieve World Domination in Orchestration? When will everything that can be orchestrated, be orchestrated by Kubernetes, and no other platform? We make a few predictions here.
Prediction #1: Medium-to-large size data centers would start exploring using Kubernetes to orchestrate its compute workload, providing cloud management support, and replacing tradition cloud management systems.
With Kubevirt, Kubernetes is well on the way to handling the virtual machine part of traditional cloud management. But there is more to existing conventional cloud management than only managing virtual machines. Bare metal server orchestration, storage, and network provisioning are all part of the underlying cloud management infrastructure, in addition to other services like databases and remote file systems. For instance, Rook, the container-based storage management system for Kubernetes, is working on the storage part. Metal Kubed is working on the server management part. While we are not fully there yet, Kubernetes will unify the management of network, storage, and compute within a cluster.
Further down the stack, Kubernetes seems well-positioned to orchestrate different types of workloads in addition to containers: VMs, micro-VMs, unikernels, and heavyweight, more isolated containers like Kata containers. Many market analysis reports have shown that there is an increasing trend of adopting a multi/hybrid-cloud strategy among enterprises to avoid cloud vendor lock-in and to take advantage of rapid innovations from various public cloud providers. Since such cloud strategies usually call for a deployment environment that consists of one or more public cloud providers, private clouds, and evolving edge clouds, the deployment complexity increases, and is more challenging to pull off. The availability of a unified resource orchestration and management solution across different types of clouds (public, private, or edge) will radically transform the cloudscape. And Kubernetes has already shown its strength in handling such hybrid cloud environments.
Prediction #2 Kubernetes will be reengineered to run directly on the hardware, providing an alternative to Linux for cloud deployments.
When people talk about “Kubernetes on bare metal” today, what they mean is “Kubernetes running on top of the Linux operating system without a virtual machine.” An operating system like Linux provides an abstraction layer for the hardware as a collection of virtual resources (processes, threads, virtual memory, sockets, etc.) that are easier for developers to consume than directly interacting with hardware. The abstractions cover multiple different kinds of vendor hardware (e.g., NICs from different vendors) or provide a more convenient API than interacting directly with server hardware (e.g., sockets instead of PCIe NIC commands). In addition, the operating system also includes orchestration for virtual resources. A VM provides an extra layer of isolation so that multiple operating system images can run on one machine with enough resource isolation. Therefore, organizations running the VM images on the same machine can be independent and potentially not co-operating; in other words, hypervisors and VMs enable a multitenant deployment.
Traditional scalable software infrastructure assumes a deployment model where individual hypervisors are running on the hardware, with a standard guest OS running in virtual machines. While this design has proved highly scalable, it has issues with stranded resources because VM sizes tend to be large because of the guest OS size, and the architecture seems unnecessarily cumbersome. More recently, some researchers have revived the idea of a distributed operating system for cloud software infrastructure, called a single system image (SSI). This idea was popular until the mid-90s when a combination of slow improvement in networking performance and the rise of the Linux hyper-scale mindshare obliterated it. Recent work in this area includes an experimental operating system, Nefele*, using the Linux kernel as a modular component in a distributed operating system and DC/OS, a product from D2IQ that likewise depends on a single machine kernel. HP also has a product, the HP Nonstop OS, built on the work of Tandem Computers, which they acquired, that runs on commodity hardware. But the single system image architecture also doesn’t seem to be achieving widespread mindshare.
The underlying operating system architecture, while it has weathered the radical changes in hardware performance over the last 50 years, is beginning to show its age. This year is the 50th anniversary of the first release of the Unix operating system, and the 28th anniversary of the first release of Linux, the latest and greatest iteration of Unix. While there are other operating systems out there, Linux is by far the choice for most server infrastructure deployments in enterprise data centers. In the 28 years since Linux was first released, hardware has changed quite a bit, not just in performance but also in basic architecture, while the overlying operating system architecture has not.
Some traditional abstractions such as processes, threads, and virtual memory still seem relevant, but others are questionable. For example, with most servers having 32–256GB and up of main memory, “swap space” seems an antiquated concept. Indeed, since most servers today use SSDs and NVMe is coming, at some point, the difference between ephemeral storage and main memory will likely be trivial. New hardware such as SGX enclaves for secure computation cries out for better abstractions, especially for orchestration. And the fit between the traditional operating system architecture and today’s massively multicore chips, to say nothing of data centers with distributed machines, has always been fraught.
In addition to problems with newer hardware architectures, in many ways, the traditional operating system API isn’t a good match for how “born in the cloud” applications are developed and delivered today. There is no support for simple release or rollback, for example, and containers are built on namespaces without any hard tenant isolation (though Kata containers are working on fixing this). In addtion, better support for serverless deployments, perhaps using micro-processes, is desirable. Extending Kubernetes to replace traditional operating systems abstractions with resources explicitly designed for “born in the cloud” applications, including distributed resources, seems like a big challenge for the research community.
Prediction #3: When Kubernetes directly on the hardware has deployment traction, changes in the hardware architecture will be made to increase performance and multitenant isolation.
While the leap from Kubernetes on an OS to Kubernetes as the OS seems like a big one now, once Kubernetes OS deployment momentum takes off, modifications to the standard processor and NIC hardware to accommodate it looks like a no-brainer. We need to look no further than the history of system virtual machines.
By “system virtual machines” we mean what people today call a VM, namely a guest operating system image packaged such that multiple operating system images can run on top of a host hypervisor. System virtualization developed in the late 1990s to provide the same kind of multiuser support offered by timeshare systems in the 1970s and 1980s, but with better isolation between users. With system virtualization, multiple users can share isolated copies of an operating system running on the same machine, even if the users are not from different, non-cooperating organizations. This model proved popular with enterprise customers. By the mid 2000’s, products such as VMWare ESX together with VCenter for orchestration and management became widely deployed in enterprise data centers.
But hypervisors without hardware support experienced performance bottlenecks in addition to the possible security problems that limited the potential for utility-scale, multitenant computing (e.g., public cloud). In 2005 and 2006, Intel and AMD introduced hardware support for virtualization. While the performance of the first generation of processors with virtualization hardware support was no better than software virtualization, multitenant isolation was better. And the performance has improved to the point where hardware virtualization is today the default choice for multitenant cloud deployment.
As the Danish politician Karl Kristian Steincke said in the fourth volume of his autobiography “It is difficult to make predictions, especially about the future,” but the trend is clear. Kubernetes is on the march, and assimilation is inevitable. With its flexible resource definition and orchestration architecture, Kubernetes is on track to become the single “go-to” for all resource orchestration needs.
* One of us (James) was part of the team that developed Nefele.
