Since its inception in 2017 by Weaveworks, GitOps has caused quite some fuss on Twitter and KubeCon. This site aggregates the essence of GitOps to help clear up the confusion about the topic.
GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools.
The core idea of GitOps is having a Git repository that always contains declarative descriptions of the infrastructure currently desired in the production environment and an automated process to make the production environment match the described state in the repository. If you want to deploy a new application or update an existing one, you only need to update the repository - the automated process handles everything else. It’s like having cruise control for managing your applications in production.
GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping.
Okay, to be fair, probably every Continuous Deployment technology promises to make deploying faster and allows you to deploy more often. What is unique about GitOps is that you don’t have to switch tools for deploying your application. Everything happens in the version control system you use for developing the application anyways.
When we say “high velocity” we mean that every product team can safely ship updates many times a day — deploy instantly, observe the results in real time, and use this feedback to roll forward or back.
Oh no! Your production environment is down!
With GitOps you have a complete history of how your environment changed over time.
This makes error recovery as easy as issuing a
git revert and watching your environment being restored.
The Git record is then not just an audit log but also a transaction log. You can roll back & forth to any snapshot.
GitOps allows you to manage deployments completely from inside your environment. For that, your environment only needs access to your repository and image registry. That’s it. You don’t have to give your developers direct access to the environment.
kubectl is the new ssh. Limit access and only use it for deployments when better tooling is not available.
Have you ever SSH’d into a server and wondered what’s running there? With GitOps, every change to any environment must happen through the repository. You can always check out the master branch and get a complete description of what is deployed where plus the complete history of every change ever made to the system. And you get an audit trail of any changes in your system for free!
Using Git to store complete descriptions of your deployed infrastructure allows everybody in your team to check out its evolution over time. With great commit messages everybody can reproduce the thought process of changing infrastructure and also easily find examples of how to set up new systems.
GitOps is the best thing since configuration as code. Git changed how we collaborate, but declarative configuration is the key to dealing with infrastructure at scale, and sets the stage for the next generation of management tools.
There are two ways how you can organize your Continuous Deployment process with Git repositories as the central element: Push-based GitOps and Pull-based GitOps. They both have in common, that they use at least two repositories: the application repository and the environment repository. The application repository contains the source code of your application and the deployment manifests to deploy the application, while the environment repository contains all deployment manifests of the currently desired infrastructure in an environment. The difference between the two types of GitOps, is how it is ensured, that the environment actually resembles the desired infrastructure.
Push-based GitOps works with regular CI/CD tools, such as Jenkins, CircleCI or Travis CI. The source code of your application lives inside the application repository along with your Kubernetes YAMLs needed to deploy your app. Whenever your application code is updated, the build pipeline is triggered, which builds your container images, and finally the environment repository is updated with new deployment descriptors.
Tip: You can also just store templates of your YAMLs in your application repository. When a new version is built, you can use the template to generate the YAML in your environment repository.
Changes to the environment repository trigger the deployment pipeline. This pipeline is responsible for applying all manifests in the environment repository to your infrastructure.
One important thing to keep in mind when using this approach, is that the deployment pipeline only is triggered when your environment repository changes. It can not automatically notice any deviations of the environment and its desired state. This means, you need some way of monitoring in place, so that you can intervene if your environment doesn’t match what is described in the environment repository.
Want to see how to set it up? Check out Google’s Tutorial on how to set up Push-based GitOps with their Cloud Builds and GKE.
Pull-based GitOps uses the same concepts as the push-based variant, but changes how the deployment pipeline works. Traditional CI/CD pipelines are triggered by an external event, for example when new code is pushed to an application repository. With this approach, the operator is introduced. It takes over the role of the pipeline by continuously comparing the desired state in the environment repository with the actual state in the deployed infrastructure. Whenever differences are noticed, the operator updates the infrastructure to match the environment repository. Additionally, when working with Kubernetes clusters, the image registry can be monitored to find new versions of images and deploy them to the cluster.
Just like push-based GitOps, this variant updates the environment whenever the environment repository changes. However, with the operator, changes can also be noticed in the other direction. Whenever the deployed infrastructure changes in any way not described in the environment repository, these changes are reverted. This ensures that all changes are made tracable in the Git log, by making all direct changes to the cluster impossible.
This change in direction solves the problem of push-based GitOps, where the environment is only updated when the environment repository is updated. However, this doesn’t mean you can completely do without any monitoring in place. Most operators support sending mails or Slack notifications if it can not bring the environment to the desired state for any reason, for example if it can not pull a container image. Additionally, you probably should set up monitoring for the operator itself, as there is no longer any automated deployment process without it.
Want to see how to set it up? Check out our Tutorial about setting up Pull-based GitOps on Google’s GKE with WeaveWorks Flux.
Of course working with just one application repository and only one environment is not realistic for most applications. When you are using a microservices architecture, you probably want to keep each service in its own repository.
GitOps can also handle such a use case. You can always just set up multiple build pipelines that update the environment repository. From there on the regular automated GitOps workflow kicks in and deploys all parts of your application.
Managing multiple environments with GitOps can be done by just using separate branches in the environment repository. You can set up the operator or the deployment pipeline to react to changes on one branch by deploying to the production environment and another to deploy to staging.
Most likely: Yes! The cool thing about GitOps is that you don’t need to write any code differently. All you need to get started is infrastructure that can be managed with declarative Infrastructure as Code tools.
Yes! GitOps is not limited to Kubernetes. In principle, you can use any infrastructure that can be observed and described declaratively, and has Infrastructure as Code tools available. However, currently most operators for pull-based GitOps are implemented with Kubernetes in mind.
Is GitOps just a new name for Infra as Code?
No. Declarative Infrastructure as Code plays a huge role for implementing GitOps, but it’s not just that. GitOps takes the whole ecosystem and tooling around Git and applies it to infrastructure. Continuous Deployment systems guarantee that the currently desired state of the infrastructure is deployed in the production environment. Apart from that you gain all the benefits of code reviews, pull requests, and comments on changes for your infrastructure.
First of all, never store secrets in plain text in git! Never!
That being said, you have have secrets created within the environment which never leave the environment. The secret stays unknown, and applications get the secrets they require but they aren’t exposed to the outside world. For example, you provision a database within the environment and give the secret to the applications interacting with the database only.
Another approach is to add a private key once to the environment (probably by someone from a dedicated ops team) and from that point you can add secrets encrypted by the public key to the environment repository. There’s even tool support for such sealed secrets in the K8s ecosystem.
GitOps doesn’t provide a solution to propagating changes from one stage to the next one. We recommend using only a single environment and avoid stage propagation altogether. But if you need multiple stages (e.g., DEV, QA, PROD, etc.) with an environment for each, you need to handle the propagation outside of the GitOps scope, for example by some CI/CD pipeline.
DevOps is all about the cultural change in an organization to make people work better together. GitOps is a technique to implement Continuous Delivery. While DevOps and GitOps share principles like automation and self-serviced infrastructure, it doesn’t really make sense to compare them. However, these shared principles certainly make it easier to adopt a GitOps workflow when you are already actively employing DevOps techniques.
You can use GitOps to implement NoOps, but it doesn’t automatically make all operations tasks obsolete. If you are using cloud resources anyway, GitOps can be used to automate those. Typically, however, some part of the infrastructure like the network configuration or the Kubernetes cluster you use isn’t managed by yourself decentrally but rather managed centrally by some operations team. So operations never really goes away.
In a way, yes. In principle, you can use any version control system you want. One of the core ideas of GitOps is letting developers use the tools they are familiar with to operate your infrastructure. If you prefer SVN over Git, that’s cool! However, you may need to put more effort into finding tools that work for you or even write your own. All available operators only work with Git repository — sorry!
No! There are no GitOps engineers. GitOps is not a role (and neither is DevOps). GitOps is a set of practices. You can look for a developer who has experience practicing GitOps — or simply let your developers try out those practices.
Also check out Weavework’s Awesome-GitOps.
Florian Beetz is currently studying International Software Systems Science at University of Bamberg. He is interested in cloud computing, clean code, and software engineering techniques. In his free time, he likes to go rock climbing.