What Is Kubernetes?
Originally developed inside Google, Kubernetes has been an open-source project since June 2014 and managed by the Cloud Native Computing Foundation (CNCF) since Google and Linux partnered to found the CNCF in July 2015. Kubernetes is an orchestration system that automates the processes involved in running thousands of containers in production. It eliminates the infrastructure complexity associated with deploying, scaling, and managing containerized applications.
There is a strong correlation between the growth in containers and microservice architectures and the adoption of Kubernetes. According to a recent Gartner report, more than 70% of global organizations will soon be running more than two containerized applications in production, up from less than 20% in 2019. And Kubernetes usage will continue to grow as companies deepen their commitment to containerization. According to a recent survey of 250 IT professionals conducted by Dimensional Insight, “Well over half (59%) are running Kubernetes in a production environment, with one-third (33%) operating 26 clusters or more and one-fifth (20%) running more than 50 clusters.”
This is part of an extensive series of guides about open source.
In this article:
Kubernetes Concepts
Pods
In Kubernetes, a pod is the smallest and simplest unit that you can create and deploy. A pod represents a single instance of a running process in the cluster and can contain one or more containers. Containers within a pod share the same network namespace, IP address, and storage, allowing them to communicate. Pods can run a single application or a set of cooperating applications. They are ephemeral and can be created and destroyed dynamically.
Services
A service in Kubernetes is an abstraction that defines a logical set of pods and a policy by which to access them. Services enable communication between different sets of pods without needing to keep track of individual pod IP addresses. Kubernetes offers several types of services, including ClusterIP, NodePort, and LoadBalancer.
Volumes
Volumes in Kubernetes provide a way to persist data beyond the lifecycle of individual pods. Unlike container storage, which is ephemeral, volumes can outlive the containers that use them. Kubernetes supports various volume types, such as emptyDir, hostPath, and networked storage solutions like NFS, AWS EBS, and Google Persistent Disks. Volumes are useful for applications that require persistent state or shared data among multiple pods.
Namespaces
Namespaces in Kubernetes are a mechanism to partition a single Kubernetes cluster into multiple virtual clusters. They help isolate and organize resources, such as pods, services, and deployments, within the same physical cluster. Namespaces are particularly useful in environments where multiple teams or projects share the same cluster, as they allow for resource segregation and administrative control.
Deployment
A deployment in Kubernetes is a higher-level abstraction that manages the creation, scaling, and updating of pods. Deployments define the desired state of the application, and Kubernetes ensures that the current state matches this desired state. This includes tasks like rolling updates, rollbacks, and scaling of pod replicas.
Key Components of Kubernetes
Here’s an overview of the Kubernetes architecture.
Kubernetes Control Plane Architecture
The control plane includes the following elements:
- API server (kube-apiserver): Exposes the Kubernetes API, serving as the entry point for administrative tasks. Processes RESTful requests from users, components, and other systems, validates them, and updates the etcd store.
- etcd: A distributed key-value store that provides a reliable way to store data across a Kubernetes cluster. Holds all the configuration data, state information, and metadata for the cluster, and is designed to be resilient to failures.
- Controller manager (kube-controller-manager): Runs a set of controllers that regulate the state of the cluster. Each controller is responsible for a specific task, such as node management, replication, and endpoint management. Controllers continuously monitor the desired state of the cluster, as defined in the deployment specifications, and make sure the actual state aligns with the desired state.
- Scheduler (kube-scheduler): Assigns pods to nodes in the cluster. Evaluates the requirements of each pod, such as resource needs and constraints, and matches them with the available nodes based on factors like resource availability, policy constraints, and affinity rules.
Kubernetes Node Components
Kubernetes nodes include the following elements:
- kubelet: An agent that runs on each node in the Kubernetes cluster. Ensures that containers are running in a pod by interacting with the container runtime. Monitors the state of the pods assigned to its node, reports back to the control plane, and takes corrective actions to maintain the desired state. It plays a vital role in the node's lifecycle management and resource monitoring.
- Kube-proxy: Runs on each node in the Kubernetes cluster and maintains network rules on the nodes, allowing communication to and from pods. Enables services to be exposed within and outside the cluster by managing IP addresses and ports. Supports networking modes such as userspace, iptables, and IPVS.
- Container runtime: The software responsible for running containers on a Kubernetes node. Interfaces with the kubelet to manage the lifecycle of containers and supports operations like starting, stopping, and deleting containers. Kubernetes supports multiple container runtimes, including Docker, containerd, and CRI-O
Best Practices for Designing Your Kubernetes Architecture
Here are some of the main considerations when building a Kubernetes architecture.
High Availability
High availability in Kubernetes ensures that the cluster and applications remain operational despite failures. It involves designing the infrastructure to handle and recover from failures seamlessly. Availability is critical for minimizing downtime and maintaining service continuity, especially in production environments.
To ensure high availability:
- Multi-master setup: Deploy multiple master nodes to eliminate a single point of failure. Use an odd number of master nodes to maintain quorum in etcd.
- Node redundancy: Distribute workloads across multiple nodes to ensure that if one node fails, others can take over its responsibilities.
- Load balancers: Use external load balancers to distribute traffic across multiple master nodes and ensure that API server requests are always handled.
- Persistent storage: Implement highly available storage solutions like distributed storage systems (e.g., Ceph, GlusterFS) to avoid data loss during node failures.
- Monitoring and alerting: Use tools like Prometheus and Grafana to monitor the health of the cluster and set up alerts to quickly respond to issues.
Portability
Portability in Kubernetes refers to the ability to deploy and manage applications across different environments with minimal modification. This involves creating configurations and container images that work uniformly across various Kubernetes clusters. Portability is important for developing scalable and flexible applications that can move between development, staging, and production environments.
To enable portability:
- Standardized configurations: Use Kubernetes configuration files (YAML or JSON) to define resources in a consistent manner, enabling easy deployment across different environments.
- Container images: Build container images that are environment-agnostic and ensure they include all dependencies needed to run the application.
- CI/CD pipelines: Implement continuous integration and continuous deployment (CI/CD) pipelines to automate the process of building, testing, and deploying applications across various environments.
- Abstraction layers: Leverage Helm charts and Kubernetes Operators to abstract and manage the complexities of applications, making them easier to deploy and manage on different clusters.
Namespace Management
Kubernetes namespace management involves organizing and isolating resources within a cluster using namespaces. This allows for better resource segregation and administrative control.
To properly manage namespaces:
- Segregation by environment: Create separate namespaces for different environments (development, staging, production) to prevent resource conflicts and enhance security.
- Resource quotas: Implement resource quotas to control the amount of resources (CPU, memory, storage) that can be consumed within a namespace, preventing any single namespace from exhausting cluster resources.
- Network policies: Use network policies to control the communication between pods in different namespaces, enhancing security and isolation.
- Access control: Apply Role-Based Access Control (RBAC) to define permissions and restrict access to resources within specific namespaces.
Storage and Data Management
Proper storage and data management is essential for applications that require data persistence beyond the lifecycle of individual pods. It ensures data durability, availability, and performance, which are especially important for stateful applications like databases.
To ensure adequate management of data and storage:
- Persistent volumes: Use Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) to manage storage independently from the lifecycle of pods.
- Data backup: Regularly back up persistent data using tools like Velero to protect against data loss.
- Dynamic provisioning: Implement dynamic storage provisioning to automatically create PVs based on PVC requests, simplifying storage management.
- Storage classes: Define storage classes to provide different types of storage (e.g., SSD, HDD) and allow users to request the type of storage that meets their performance and cost requirements.
Deployment Strategies
Deployment strategies in Kubernetes define how applications are released and updated. Choosing the right deployment strategy is key to minimizing downtime and mitigating risks during updates. It also supports quick rollbacks in case of issues.
Examples of deployment strategies include:
- Rolling updates: Use rolling updates to gradually replace old pod instances with new ones, minimizing downtime and ensuring continuous service availability.
- Canary deployments: Deploy new versions to a small subset of users first to test and validate changes before a full rollout.
- Blue-green deployments: Maintain two identical environments (blue and green). Direct traffic to the blue environment while the green is updated. Once the update is validated, switch traffic to the green environment.
- A/B testing: Deploy different versions of the application to separate user groups simultaneously to compare performance and gather feedback.
Security
Security in Kubernetes requires practices and configurations that protect the cluster and applications from threats. Stringent security measures protect sensitive data, ensure compliance with regulations, and prevent unauthorized access to the cluster, minimizing the risk of a breach.
To ensure security in Kubernetes:
- RBAC: Implement Role-Based Access Control to enforce the principle of least privilege, ensuring users and applications have only the necessary permissions.
- Network policies: Use network policies to control traffic between pods and prevent unauthorized access.
- Pod security policies: Apply pod security policies to define security standards for pod creation, such as running as non-root and restricting privileged access.
- Image security: Scan container images for vulnerabilities using tools like Trivy or Clair before deployment, and use trusted registries to pull images.
- Encryption: Enable encryption for data at rest and in transit to protect sensitive information.
- Audit logging: Enable audit logging to track access and modifications to the cluster, helping detect and respond to suspicious activities.
How Does Kubernetes Address Data Science Challenges?
Containers and the Kubernetes ecosystem have been embraced by developers for their ability to abstract modern distributed applications from the infrastructure layer. Declarative deployments, real-time continuous monitoring, and dynamic service routing deliver repeatability, reproducibility, portability, and flexibility across diverse environments and libraries.
These same Kubernetes features address many of the most fundamental requirements of data science workloads:
- Reproducibility across a complex pipeline: Machine/deep learning pipelines consist of multiple stages, from data processing through feature extraction to training, testing, and deploying models. With Kubernetes, research and operations teams can confidently share a combined infrastructure-agnostic pipeline.
- Repeatability: Machine/deep learning is a highly iterative process. With Kubernetes data scientists can repeat experiments with full control over all environmental variables including data sets, ML libraries, and infrastructure resources.
- Portability across development, staging, and production environments: When run with Kubernetes, ML-based containerized applications can be seamlessly and dynamically ported across diverse environments.
- Flexibility: Kubernetes provides the messaging, deployment, and orchestration fabric that is essential for packaging ML-based applications as highly modular microservices capable of mixing and matching different languages, libraries, databases, and infrastructures.
Considerations for Successful Kubernetes Architecture for AI Workloads
With all of the advantages described above, it is not surprising that Kubernetes has become the de facto container orchestration standard for data science teams. This section provides best practices for optimizing how data science workloads are run on Kubernetes.
Kubernetes Monitoring
Monitoring Kubernetes clusters is essential for right-scaling Kubernetes applications in production and for maintaining system availability and health. However, legacy tools for monitoring monolithic applications cannot provide actionable observability into distributed, event-driven, and dynamic Kubernetes applications. The new monitoring challenges raised by Kubernetes deployments include:
- With seamless deployment across complex infrastructures, diverse streams of compute, store, and network data must be normalized, analyzed, and visualized to achieve real-time actionable insight into environment topology and performance.
- Highly ephemeral containers make it tricky to capture and track important metrics such as the number of containers currently running, container restart activity, and each container’s CPU, storage, memory usage, and network health.
- Effectively harnessing Kubernetes’ rich array of internal logs for quick detection and remediation of cluster performance issues, including node and control plane component metrics.
The current gold standard for monitoring Kubernetes ecosystems is Prometheus, an open-source monitoring system with its own declarative query language, PromQL. A Prometheus server deployed in the Kubernetes ecosystem can discover Kubernetes services and pull their metrics into a scalable time-series database. Prometheus’ multidimensional data model based on key-value pairs aligns well with how Kubernetes structures infrastructure metadata using labels.
The Prometheus metrics, which are published using the standard HTTP protocol, are human-readable and easily accessed via API calls by, for example, visualization and dashboard-building tools such as Grafana. Prometheus itself provides basic visualization capabilities by displaying the results of PromQL queries run on the aggregated time-series data as tables or graphs. Prometheus can also issue real-time alerts to the relevant teams when predefined performance thresholds are breached.
- Run batch AI workloads as jobs and interactive sessions as replicas
- Use CronJobs for better scheduling
Traditionally, when used for applications and services, K8s containers are run as replicas, not as jobs. But for ML and DL workloads, running as jobs is a better fit. This is because jobs run to completion and can support parallel processing. Jobs can run at the same time multiple pods, enabling set up of a parallel processing workflow while making sure those pods terminate and free their resources when the job runs to completion. Replicas are not set up to enable this functionality, which is critical for batch experimentation and for increasing resource utilization and reducing cloud spending. Replicas are a better fit for interactive sessions where users build and debug their models or experiment with data.
Kubernetes architecture includes CronJob, which is the native way to trigger jobs in a schedule. CronJobs are used when creating periodic and recurring tasks. CronJobs can also schedule specific tasks at determined times, such as scheduling a Job for when your cluster is likely to be idle.
Kubernetes Architecture and Run:ai
Run:ai’s platform is built as a plug-in to the Kubernetes architecture to enable automated orchestration of high-performance AI workloads.
Run:ai simplifies and optimizes orchestration of AI workloads on Kubernetes to help data scientists accelerate their productivity and the quality of their models. Learn more about the Run:ai platform.
Learn More About Kubernetes Architecture
The Challenges of Scheduling AI Workloads on Kubernetes
