My blog link: http://blog.geekidentity.com/k8s/concepts/cluster-administration/federation-cn/

Federation

This page explains why and how to use federation to manage multiple Kubernetes clusters.

Why federation
- Caveats
- Hybrid cloud capabilities
Setting up federation
API resources
Cascading deletion
Scope of a single cluster
Selecting the right number of clusters
What’s next

why federal

Federation makes it easy to manage multiple clusters. It does this by providing 2 main building blocks:

Synchronize resources across clusters: Federation can keep resources in multiple clusters in sync. For example, you can ensure that the same program is deployed across multiple clusters.
Discovery across clusters: Federation provides automatic configuration of DNS servers and load balancers with all cluster backends. For example, you can ensure that you can use global VIP or DNS records to access the backends of multiple clusters.

Some other uses of federation are as follows:

High Availability: Federation minimizes the impact of cluster failures by spreading load across clusters and automatically configuring DNS servers and load balancers.
Avoid provider lock-in: Federation prevents cluster provider lock-in by making it easier to migrate applications across clusters.

Federations are not useful unless you have multiple clusters. Some reasons why you might need multiple clusters are:

Low Latency: Having clusters in multiple regions minimize latency by serving the cluster closest to them.
Fault isolation: It is better to have multiple small clusters rather than a single large cluster for fault isolation (eg: multiple clusters in different Availability Zones of the cloud provider).
Scalability: A single kubernetes cluster has scalability limitations (most users shouldn't, see Kubernetes Scaling and Performance Goals for more details ).
Hybrid cloud : You can have multiple clusters on different cloud providers or on-premises data centers.

Precautions

While federation has many attractive uses, there are some caveats:

Increased network bandwidth and costs: The federation console monitors all clusters to ensure the current state is as expected. This can lead to significant network costs if the cluster is running in different regions of the cloud provider or different cloud providers.
Reduce cross-cluster isolation: A bug in the federation console can affect all clusters. This problem can be mitigated by keeping the logic in the federation console minimal. Whenever possible, it is mostly delegated to the console's kubernetes cluster. The design and implementation also takes a lot of thought into security and avoids multi-cluster downtime in the event of an error.
Maturity: Federal programs are relatively new and less mature. Not all resources are available and many are still in alpha status. Issue 88 lists known issues with the system that the team is busy working on.

Hybrid cloud capabilities

Kubernetes cluster federation can run clusters in different cloud providers (e.g. Google Cloud, AWS) and on-premises (e.g. OpenStack). Kubefed is the recommended way to deploy federated clusters.

Thereafter, your API resources can span different clusters and cloud providers.

set up federation

To be able to federate multiple clusters, you first need to set up the federation console. Follow the setup guide to set it up.

API resources

Once the console is set up, the creation of federated API resources can begin. The following guides explain some of the resources in detail:

The API reference documentation lists all the resources supported by the federated apiserver.

Cascade delete

Kubernetes version 1.6 supports cascading deletion of federated resources. When a resource is deleted from the federation console, the corresponding resource in all underlying clusters is also deleted.

Cascading deletes are not enabled by default when using the REST API. To enable it, set the DeleteOptions.orphanDependents=false option when deleting a resource from the federation console using the REST API. Cascading deletes are enabled by default using kubectl delete. It can also be disabled by running kubectl delete --cascade=false

Note: Kubernetes version 1.5 includes cascading delete support for subsets of federated resources.

Scope of a single cluster

In IaaS providers such as Google Compute Engine or Amazon Web Services, virtual machines exist in zones or AZs. We recommend that all virtual machines in a Kubernetes cluster should be in the same Availability Zone because:

There are fewer single points of failure than having a single global Kubernetes cluster.
It is easier to infer availability properties for single-zone clusters than for clusters that span Availability Zones.
When Kubernetes developers are designing systems (such as making assumptions about latency, bandwidth, or related failures), they assume that all machines are located in a single data center or are very closely connected.

It is recommended to run fewer clusters of virtual machines per Availability Zone; however, it is possible to run multiple clusters per Availability Zone.

The reasons for choosing fewer clusters per Availability Zone are:

In some cases, having more nodes (fewer resources) in a cluster can improve pod packing.
Reduced operational overhead (though as operational tools and processes mature, the benefits are less pronounced).
Reduce the cost of fixed resources per cluster, for example, apiserver virtual machines (but a small percentage of the overall cluster cost for medium and large clusters).

Reasons for having multiple clusters include:

Strict security policies require isolation of one class of work from another (but see partitioned clusters below)
Test the cluster canary to a new Kubernetes version or other cluster software.

Choose the correct number of clusters

The number of Kubernetes clusters to choose from generally does not change, but is revisited occasionally. In contrast, the number of nodes in a cluster and the number of pods in a service may change frequently with load and business growth.

To choose the number of clusters, you first need to determine which regions you need to deploy in so that services running on Kubernetes provide the lowest latency for all end users, (if using a content delivery network, CDN-hosted content does not need to be considered ). Legal issues may also play into this. For example, a company with a global customer base might decide to have clusters in the US, EU, AP, and South Africa regions. The number of regions that need to be used is R.

Second, determine how many clusters can be simultaneously unavailable without affecting the overall business. Set the maximum number of unavailable clusters to U. If you are not sure, then U=1 is a good choice.

At least a larger R or U+1 cluster is required if load balancing is allowed to direct traffic to any region in the event of a cluster failure. If not (for example, if you want to ensure low latency for all users in the event of a cluster failure), you need to have an R*(U+1) cluster (U+1 in each R region). Anyway, try to put each cluster into a different region.

Finally, if you need to build a cluster with more nodes than Kubernetes' maximum recommended number of nodes, you may need multiple clusters. Kubernetes v1.3 supports clusters of up to 1000 nodes. Kubernetes v1.8 supports clusters of up to 5000 nodes. For more guidance, see Building Large Clusters .

What’s next

Learn more about federal programs .
See this setup guide for cluster federation.
See this Kubecon2016 talk on federation
See this Kubecon2017 Europe update on federation

Introduction to Kubernetes Federation Mechanism