Tools for an efficient DevOps workflow

Tools for an efficient DevOps workflow

The DevOps workflow is about efficiently producing useful outcomes—in both software and process—that inform and improve the next cycle of work from dev-to-deploy. This is how you can create more opportunities for digital transformation in your business. The goal is not just velocity, not to simply move faster through the steps of the workflow, but also to create opportunities to learn through experimentation or failure at each step so the next big idea can shine through.

A daily workflow based on DevOps concepts enables team members to deliver value (by shipping software and fixing bugs) faster, be flexible enough to both experiment and deliver value, and help every part of the organization adopt a learning mentality. The specific workflow your team uses will employ a wide variety of tools and steps tailored to your project and team, but the framework and purpose will be similar to whatever you are working with.

In this post, we’ll take a closer look at some of the tools you might use in each step, how they integrate with each other, and how communication is key throughout the workflow. We’ll start with local development, which is one of the areas where today’s technologies can make a significant impact on your developers’ day-to-day work and value delivery. Then we’ll move on through other steps and practices: CI/CD, project management, and finally hosting—another area where there’s a lot of improvement possible through technology choices today.

Conceptually, of course, once your software is live, your team would be pulling in a copy of the live code to their local development environments to implement the next set of features that will deliver value … and so the cycle continues.

What comes first?

The DevOps movement emphasizes a holistic approach to your work practices, but it might be difficult to see where you can get started. The first step to adopting DevOps practices is to understand how you work now. Discover and document your existing systems, processes, and workflows; list the tools you use and how they interact. From there, you can create checklists to make changes before the next cycle. It’s important that these cycles are short so you can integrate new insights into the next cycle, and see improvements as soon as possible.

Once you have a clear picture of your overall process, you can begin optimizing it all by adjusting steps, implementing new tools or automation, measuring, and analyzing the results. If you reduce the overhead on any workflow step, you’ll increase the speed and accuracy of the whole. That’s the foundation of a smart technical practice. A workflow should be designed to reduce friction, enable rapid iterations, deliver software at a higher velocity, and reinforce best practices—security, auditability, efficiency, etc.—and wherever possible automate them.

With all this in mind, let’s have a look at the workflow steps and some of the tools you might use along the way.

Image showing stages of dev > dest > deploy and the tools you'd encounter.

Local Development

Local development is the first step in a solid DevOps workflow and it is the most critical to all the other pieces fitting together and avoiding errors in production. “Local development” means working with a copy of the project on your own computer, rather than on a live website or application in production. Some pieces you might need:

Local development environment: The key to developer efficiency and effectiveness is a robust, portable, production-like environment, and an up-to-date copy of the project for anyone who wants to work on it, test, or view progress. Choose between these options depending on how much bandwidth you have for DIY setup and configuration, portability needs, and whether it will integrate smoothly with your other tools. DDEV-Local is DRUD’s own contribution to this space. It is a fast, open source, containerized local development environment that runs on Mac, Windows or Linux and integrates neatly with our hosting solution, DDEV-Live, and can be set up to deploy to others, too.

  • Host machine *AMP Stacks
  • MAMP/WAMP
  • VMs
  • Docker
  • DDEV-Local

Configuration-Management and Provisioning: This encompasses the entirety of the product’s lifecycle, including identifying, managing, monitoring, and auditing everything from processes to documentation, team members, tools, software, and systems involved. The book, Effective DevOps is an excellent resource on this topic. A few tools to help you manage software builds and provisioning, which you might choose between depending on your local environment choice:

  • Chef/Puppet
  • Ansible
  • Docker Images

Code Editors: Depending on what sort of project you’re working with, you might want a simple text editor or a more robust integrated development environment (IDE) with a compiler, interpreter, and debugger included. The IDE you choose will depend on which one offers the tools you need for your specific project:

  • Textmate/Sublime
  • PHPStorm
  • Visual Studio Code

Services: These tools will need to be compatible with your project and your local development environment. A development environment like DDEV-Local offers MailHog and Xdebug built-in, others offer the option of adding your preferred services. Audit your project to see what you might need, then choose an environment that will support those services. Here are a few popular services:

  • Task Runners
  • MailHog
  • Xdebug
  • SQL
  • Package Managers

Testing (Automated): Test-driven development is a major part of communicating the intent of the code and the project to all stakeholders, and it eliminates errors starting very early in a project. Enabling developers to run tests on their own prevents code from becoming stale while a test instance is requested. Choose your tools based on the type of project you’re working on and automate as much as possible from the start in order to speed future cycles. Some options to consider:

  • PHPUnit
  • Behat
  • Selenium
  • Phantom JS

Testing (Manual): While you should automate as much as possible, one of the most straightforward steps you can take to assure the quality of your web application is to make sure it works on all browsers and devices. To catch these issues early on, before they affect other work, we recommend:

  • Browserstack

CI/CD

Continuous Integration (CI) is the process of regularly incorporating small new pieces of work to the main codebase. It is essential to include automated testing to minimize risks when making a large number of small changes on an ongoing basis. Continuous Delivery (or Continuous Deployment) (CD) are the processes in which new releases are cleared for deployment and are actually sent to production. In both parts of the process, the value is in small, frequent changes that can be rolled back easily, taken into account by other work in progress (WIP). Doing things this way lets team members see their work in action soon after it is finished and learn from it in near-real-time. One goal of DevOps practices it to shorten feedback loops.

Version Control: A good version control system (VCS) will allow your team to work collaboratively and provide a safe environment to make and roll back changes, and receive credit for their work. Look for a VCS and code repository hosting solution that best suits your team’s needs for privacy, collaboration, and autonomy for contributors who just want to get to work. Popular options include:

  • GitHub
  • GitLab
  • Bitbucket

ChatOps: You team will benefit from clearly defined procedures on how to communicate in different situations. For example “texting at 2 a.m.” is not an option unless it’s an emergency and even then, it’s probably not a good option. You need robust procedures in place—for example, “If a mission-critical application goes down, open a new channel in your chat client, inform team leads.” Furthermore, to learn from problems and incidents, you need the support of a system that can archive your messages so that they can serve as a learning (and later prevention) tool. Team members need to be able to reference previous conversations and search through an organizational knowledge base. Some good systems include:

  • Slack
  • IRC
  • SMS
  • Emails

Builds/Testing/Tasks: Build-automation includes tasks that assemble the software from source files, including its dependencies, as well as tasks that automatically generate a build when new code is committed. Automating these steps removes risk for human error and forgetfulness. Choose from these depending on the exact tasks you need to accomplish:

  • Circle CI
  • Travis CI
  • Jenkins
  • DDEV Live

Deployments: When it comes to deploying your application, you need a tool that will connect with your hosting providers API to push your tested and approved application code from your version control repository to the live hosting environment. DDEV-Local can be set up not only to deploy to deploy to DRUD’s DDEV-Live container-based hosting but other providers as well.

Project Management

Ticketing: Clear communication of project status helps drive work forward and eliminate blockers and blame. Ticketing systems like these offer several visual methods to track the status of work in progress using Agile practices like a Kanban board and some include tools to schedule delivery and orchestrate collaboration.

  • Jira
  • Basecamp
  • Trello
  • Asana

Documentation: Part of a culture of learning, this is one of the most important things you can do to increase efficiency and flow! Shared documentation can help onboarding new team members, reducing the time spent trying to figure out intent, or what tool was used in the past, and much more. Everything should be replicable and documentation can even be considered a second line of defense right behind automation. Some tools you might use to sync and share documentation:

  • Dropbox Apps
  • Google Apps
  • Evernote

Hosting

A strong DevOps workflow has feedback loops guiding the flow of data from production back to development where it can inform the next iteration, so whatever and however you deploy, integration with the kinds of tools below is crucial.

Monitoring: You may need tools to monitor at various levels, including system infrastructure,  up-time, and resource usage, as well as application, memory, and database performance. All of this data can be used to find where your project can be more efficient, whether you need to change infrastructure, and whether the project meets expectations. Monitoring tools include:

  • Prometheus
  • Datadog
  • Pingdom
  • New Relic

Logging: Data from application traffic, errors, and other events within the software can be logged and used as a reference when something goes wrong. Both your application and third-party components can generate logs constantly, leaving with you a lot to sift through. There are tools to help you manage log information:

  • Manual logging (files on a server)
  • Splunk
  • ELK Stack (Elastic Search, Logstash, Kibana)

Analysis, Alerts, and Reporting: Automated, human-readable results from monitoring, testing or logging sent to directly to designated team members. Known as alerts or reports, these are valuable because they can notify you of potential trouble areas as soon as they are detected, allowing you to remedy issues before they escalate. Try out:

  • New Relic
  • Prometheus Alert Manager
  • VictorOps
  • Grafana

Backups: Reliable, high-quality, near-fail-proof products are the outcome of a well-tooled DevOps workflow and practices. Once you have version control and testing in place—and you are always and only committing tested code to your repos—you have built yourself a safety net. When you need a fresh copy of your project, you can pull from the Master branch (or whatever appropriate commit point), re-import the database and off you go. Spinning up a local copy of a project is especially quick with a containerized local development environment, and if you’re working with DDEV-Local, you can use snapshots to restore a previous version of your database, too.

DDEV-Live

DDEV-Live is a managed hosting solution developed, monitored, and maintained by DRUD that runs a custom Kubernetes cluster on any public or private cloud, white-label or OEM and supports auto-scaling, high availability, CDN compatibility, logs, monitoring and SSL/TLS encryption. It includes tools and solutions for all of the points listed here under “Hosting.” DDEV-Live might not meet all your requirements and it’s worth comparing solutions to be certain you’re using the solution that is right for you. Here are a few other hosting providers to consider:

  • DDEV-Live
  • AWS
  • Acquia
  • Pantheon
  • Platform.sh
  • Digital Ocean
  • … and many others

DevOps is greater than the sum of its parts

We’ve seen here that individual tools all deliver value at different points of your dev-to-deploy workflow. None of these tools—or even the DevOps workflow—will get your team to delivering true digital transformation on their own. But they will give you a strong framework on which to innovate and drive your project forward. The value to you and your business of these tools is greatly multiplied when used together in a carefully managed DevOps workflow and in organizations with a culture of learning.

Learn more: In our recent post, DevOps – the Culture v Software Question, we go further into this, including getting buy-in from the top, building a partnership between business and IT, being able to roll with disruption and more. Want to talk more about your organization’s dev-to-deploy workflow and tooling? Get in touch with us! We’d love to help.


Photo by Sasha • Stories on Unsplash

Please follow and like us: