Deploy a Multicluster Application with GitOps

OpenShift GitOps implements Argo CD as a controller so that it continuously monitors application definitions and configurations defined in a Git repository. Then, Argo CD compares the specified state of these configurations with their live state on the cluster.

The ApplicationSet leverages the cluster decision generator to interface Kubernetes custom resources that use custom resource-specific logic to decide which managed clusters to deploy to. A cluster decision resource generates a list of managed clusters, which are then rendered into the template fields of the ApplicationSet resource. This is done using duck-typing, which does not require knowledge of the full shape of the referenced Kubernetes resource.

ApplicationSet is a sub-project of Argo CD that adds multicluster support. You will go through this section to create ApplicationSets from the Red Hat Advanced Cluster Management and from the ArgoCD CLI.

We are gonna deploy 4 applications to understand the how it works.

Application 1 - Deploying a ChatBot application

The first Application will be created from the Advanced Cluster Management console and it will be placed on those Clusters matching labels:

  • Vendor=Openshift

  • Platform=AWS

Go throught the Advanced Cluster Management Console and login with Openshift user:

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet

application02

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > General

  • ApplicationSet name: rhte2023-application-01

  • Argo server: openshift-gitops

  • Requeue time: 180

application03

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Template

application04

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Sync policy

  • Automatically sync when cluster state changes (syncPolicy/automated/selfHeal=true)

  • Delete resources that are no longer defined in the source repository (syncPolicy/automated/prune=true)

  • Delete resources that are no longer defined in the source repository at the end of a sync operation (syncOptions/CreateNamespace=true)

  • Automatically create namespace if it does not exist (syncOptions/CreateNamespace=true)

application05

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Placement

  • Placement: rhte2023-gitops-clusters

application06

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Review and Submit

application07

Advanced Cluster Management Console > Applications > rhte2023-application-01 (filter by ApplicationSet) > Topology

application08

Advanced Cluster Management Console > Applications > Advanced Configuration > Placements

Check the number of cluster matching with Placement > rhte2023-gitops-clusters, so we are expecting see the Application rhte2023-application-01 deployed in all of them.

application08b

Also, let’s check from the ArgoCD Console what is the Application status and from oc client

  • Argo CD Console > Applications

application09

  • Application deployment pods on both clusters

oc --context acm get pod -n rhte2023-application-01
oc --context cluster01 get pod -n rhte2023-application-01

Finally, let’s see how the Applications looks like checking the Route and copying on your browser.

oc --context acm get route -n rhte2023-application-01

URL: https://rhte2023-application-01-rhte2023-application-01.<your_domain_cluster>;

oc --context cluster01 get route -n rhte2023-application-01

URL: https://rhte2023-application-01-rhte2023-application-01.<your_domain_cluster>;

application09b

This is a GPT3 Chatbot, so just make questions on the chat to get bot’s answers.

Application 1 - Change number of replicas

In order to check how ArgoCD and ACM are monitoring the Application, let’s make changes and see what happens.

  • Change application replicas

Let’s change the number of replicas in both clusters and check how it is fixed automatically.

oc --context acm scale deployment/rhte2023-application-01 -n rhte2023-application-01 --replicas=0
oc --context cluster01 scale deployment/rhte2023-application-01 -n rhte2023-application-01 --replicas=0
oc --context acm get pod -n rhte2023-application-01
oc --context cluster01 get pod -n rhte2023-application-01

After a while, application will scale automatically to the initial number of replicas (5).

application09c

  • Make a change on the deployment yaml file

Now, let’s make a persistent change on the Git repository:

cd 04_deploy_applications/rhte2023-application-01/base
sed -i 's/replicas: 5/replicas: 3/g' 100-deployment.yaml
git add 100-deployment.yaml
git commit -m "decreasing number of replicas"
git push origin main
See how configure a token GitHub Creating a personal access token to push changes on your git repository.

Once the change has been pushed, sync the application again from ArgoCD Console and checking the ACM Topology

In case you’ve got this error Unable to deploy revision: permission denied: applications, sync, please fix as follows:

oc --context acm adm groups new cluster-admins
oc --context acm adm groups add-users cluster-admins opentlc-mgr
oc --context acm adm policy add-cluster-role-to-group cluster-admin cluster-admins
application09d
application15

Application 2 - Deploying a ChatDraw application

The second Application will be created from the Advanced Cluster Management console and it will be deployed on those Clusters matching label:

  • environment=development

Before create the application, let’s label the cluster rhte2023-cluster01 with the label environment=development.

oc --context acm label ManagedCluster rhte2023-cluster01 environment=development --overwrite

Let’s create the Application from Advanced Cluster Management:

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet

application02

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > General

  • ApplicationSet name: rhte2023-application-02

  • Argo server: openshift-gitops

  • Requeue time: 180

application10

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Template

application11

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Sync policy

  • Automatically sync when cluster state changes (syncPolicy/automated/selfHeal=true)

  • Delete resources that are no longer defined in the source repository (syncPolicy/automated/prune=true)

  • Delete resources that are no longer defined in the source repository at the end of a sync operation (syncOptions/CreateNamespace=true)

  • Automatically create namespace if it does not exist (syncOptions/CreateNamespace=true)

application12

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Placement

  • Placement: rhte2023-gitops-clusters-environment

application13

Advanced Cluster Management Console > Applications > Create Application > ApplicationSet > Review and Submit

application14

Advanced Cluster Management Console > Applications > rhte2023-application-02 (filter by ApplicationSet) > Topology

application15

Advanced Cluster Management Console > Applications > Advanced Configuration > Placements

Check the number of cluster matching with Placement > rhte2023-gitops-clusters-environment, so we are expecting see the Application rhte2023-application-02 deployed in rhte2023-cluster01.

application16

Also, let’s check from the ArgoCD Console what is the Application status and from oc client

  • Argo CD Console > Applications

application17

  • Application deployment pods

oc --context cluster01 get pod -n rhte2023-application-02

Finally, let’s see how the Applications looks like checking the Route and copying on your browser.

oc --context cluster01 get route -n rhte2023-application-02

URL: https://rhte2023-application-02-rhte2023-application-02.<your_domain>;

application18

This is a GPT3 Chatdraw, so just give a description about what you want to see.

Application 2 - Change container image

Once the rhte2023-application-02 is already deployed, let’s change the application image:

  • Change application image commit and push your changes

cd ../../../04_deploy_applications/rhte2023-application-02/base
sed -i 's/chatdraw:latest/chatdraw:rhte2023/g' 100-deployment.yaml
git add 100-deployment.yaml
git commit -m "changing the application image"
git push origin main

Once the change has been pushed, sync the application again and verify that the application has been changed (background red color)

application19

Application - Deploying multiple Applications

The third and fourth Applications will be created from the ArgoCD command line grouping both Applications rhte2023-application-03 (RedHat Offices Photos) and rhte2023-application-04 (Tetris) and deploying at the same time:

  • Application rhte2023-application-03 will match Location=eu-west-2 and area=fringe labels

  • Application rhte2023-application-04 will match clusters with the largest allocatable CPU and memory allocatable CPU and memory.

  • Apply new label to local-cluster

oc --context acm label ManagedCluster local-cluster area=fringe --overwrite

  • Change ApplicationSet according to your settings:

Change repoURL variable with your git repository https://github.com/<your_github_account>/rhte-2023-acm-apps.git

cd ../../..
vi 04_deploy_applications/argocd/rhte2023-application-03.yaml
vi 04_deploy_applications/argocd/rhte2023-application-04.yaml

  • Change Location according to your settings:

vi 04_deploy_applications/argocd/placement_location.yaml

  • Commit and push changes:

git status
git add *
git commit -m "changing repository"
git push origin main

  • Get ArgoCD password

oc --context acm get secret/openshift-gitops-cluster -n openshift-gitops -o jsonpath='{.data.admin\.password}' | base64 -d

  • Get ArgoCD Route

oc --context acm get route -n openshift-gitops

  • Login into ArgoCD

argocd login openshift-gitops-server-openshift-gitops.apps.<your_domain> --username admin --password <your_password> --insecure

  • List ArgoCD Clusters

argocd cluster list

  • Create an ArgoCD Application:

argocd app create rhte2023-application-gitops \
--project default \
--repo <your_forked_repository> \
--path rhte2023/04_deploy_applications/argocd \
--sync-policy automated \
--dest-namespace openshift-gitops \
--dest-server https://api.<your_domain>:6443

  • Get ArgoCD Application details:

In order to check the deployment status run:

argocd app list
argocd app get rhte2023-application-gitops

  • Check Application pods

In order to see where the Applications have been placed, go to Advanced Cluster Management > Applications > Advanced Configuration > Placements, and check the number of the clusters matching Placements

  • Application rhte2023-application-03 > rhte2023-gitops-clusters-location

  • Application rhte2023-application-04 > rhte2023-gitops-clusters-cpu

Or going to Advanced Cluster Management > Applications and filter per Application name and click on the Topology where you can see the Application deployed and the information about the targeted clusters.

Once you know where Applications are deployed, just check the Application pods status changing the context according to your environment:

oc --context acm get pod -n rhte2023-application-03
oc --context cluster01 get pod -n rhte2023-application-04

  • Applications routes

Let’s see how the Applications looks like checking the Route and copying on your browser. In this example, rhte2023-application-03 is placed on local-cluster and rhte2023-application-04 is placed in rhte2023-cluster01:

oc --context acm get route -n rhte2023-application-03
oc --context cluster01 get route -n rhte2023-application-04

  • Application rhte2023-application-03 > Red Hat Offices

URL: https://rhte2023-application-03-rhte2023-application-03.<your_domain>;

application20

URL: https://rhte2023-application-04-rhte2023-application-04.<your_domain>;

  • Application rhte2023-application-04 > Tetris Game

application21

Fix Application 03 - Sync and Diff

Let’s see a common issue with Application deployments and autoscalers and how to fix it. The third Application rhte2023-application-03 is deployed with an HPA - horizontal pod autoscaler that let you specify the minimum and maximum number of pods you want to run. It means that this application will change the number of replicas and it will not match what is defined in Git repository, so the application will be out of sync. Let’s see how can fix this kind of issues:

  • Check and Sync the Application rhte2023-application-03-<cluster> from the ArgoCD UI. The Application status will be flapping between synced and of of sync status.

  • Make changes on replicas definition:

cd 04_deploy_applications/rhte2023-application-03/base
sed -i '/replicas:/d' 100-deployment.yaml
git add 100-deployment.yaml
git commit -m "removing the replicas definition"
git push origin main

Commit, push your changes, sync the Application again.

This issue could also be solved adding ignoreDifferences into the deployment definition.

Fix Application 04 - allocatable CPU and Memory

This is the last exercice from this section. The fourth application rhte2023-application-04 will be deployed on that cluster with the largest allocatable CPU and Memory. Verify that in your environment is deployed on rhte2023-cluster01, if not change the score weight as follows:

  • Change the score weight:

cd ../../../04_deploy_applications/argocd/
sed -i 's/weight: 10/weight: -9/g' placement_cpu.yaml
git add *
git commit -m "changing score weight"
git push origin main

Commit, push your changes, sync the Application again.

Great! You’ve completed the 4 Applications deployment! Congratulations!