How A Telecom Giant Accelerated Infrastructure Management by 90%

Customer

An Operations-focused division inside a large multinational conglomerate, which is also one of the largest telecommunications company in the world.

Challenge

The Operations team was responsible for provisioning and maintaining Openstack infrastructure for all applications across the organization. This included creating and managing several internal networks, provisioning different components like security VNFs, virtual machines, Kubernetes clusters, etc.

The team used several tools for these activities, such as Ansible, Terraform, and Packer, in addition to CLIs and SDKs from different providers. All infrastructure was managed in a private Openstack cloud with no access to the Internet.

Internal teams could request new infrastructure through a ticketing system. Each ticket went through a triage process, and if approved, the Operations team ran the right combination of scripts to provision or update infrastructure. While IT automation tools helped the team avoid UI-driven configuration, the process of executing these various scripts was still manual. This frequently caused long, unpredictable delays, depending on the availability of the Operations team and ticket priority. Internal Dev and Test teams complained that the process took anywhere between 1-3 weeks, which slowed them down and affected their ability to ship effectively.

The Operations team was struggling to keep up with the constantly-increasing demand for infrastructure. They identified the following main causes of delays:

  • The Operations team treated any Production-related tasks as higher priority, and frequently had no additional bandwidth to service non-Production related tickets.
  • The effort to service each ticket was non-trivial, since the workflow spanned several tools and configurations that had to be updated and executed manually in specific sequences.
  • A lot of time was spent in process-related manual activities such as maintaining an inventory of all infrastructure or deprovisioning unused infrastructure, for example.
  • New hires on the Operations team took a long time to ramp up due to numerous manual processes.

Chief Architect, a key member of the Operations team, was given the charter of finding a solution that would reduce average resolution time per ticket without scaling up the team.

Solution

Chief Architect approached Shippable while researching different solutions. Excited by what he saw during a live demo, he signed up for the Shippable Pilot program. This enabled him to get a free Shippable Server trial license for 30 days, during which the Shippable team helped him implement one pilot scenario of his choice. Chief Architect was concerned that no platform could work around the complexities of their Openstack-only environment with highly isolated internal networks.

The Shippable team worked closely with Chief Architect to achieve the following in less than 2 weeks:

  • Installed Shippable Server in their internal Openstack environment with multiple proxies and complex network topology. Configured authentication with Bitbucket Server.
  • Created a mono repo in Bitbucket Server with folders for each workflow. This helped the team keep all scripts in one place and create common templates which could be reused across several workflows.
  • Configured all workflows-as-code with a simple YAML configuration file. Automation code was committed to source control and versioned.
  • Configured Shippable to execute a specific workflow, depending on the activity that triggered an event.
  • Automated the ticket approval process. When a ticket was approved, this triggered Shippable and sent a payload, which triggered the appropriate workflow. If execution succeeded, the ticket would be closed automatically.
  • Inventory and audit of all actions was maintained as part of the platform. This included things like Terraform state, inventory of provisioned infrastructure, etc.
  • Security patching was automated, since the platform already had an inventory of all machines that needed to be patched.
  • Secrets were managed centrally with an included secret store. Teams no longer needed to exchange sensitive information through email and Slack conversations.

Results

With Shippable, the Customer automated all infrastructure management related activities, including integrating with ticket approval process, with no manual steps. As a result, the Operations team saw the following benefits:

  • Ticket resolution time went from 1-3 weeks to less than one day.
  • Security patches rollout takes a couple of hours, as opposed to taking 2 weeks on an average.
  • Adding new workflows takes less than half a day.
  • Number of scripts needed to manage infrastructure has shrunk by 40%, due to code reuse.
  • Execution errors have reduced by an estimated 80%, since nobody is manually copying strings around..

Chief Architect summarized the impact of this DevOps transformation: “I love that we can maintain all our automation and workflow scripts in one repository, and the platform intelligently figures out what needs to be done. Our team just needs to approve a ticket, and boom! Infrastructure is provisioned and the stakeholders are notified. We can get an inventory of our entire network, as well as notifications to consider deprovisioning underutilized infra. If DevOps is about culture and respect across Dev and Ops, Shippable is instrumental in fostering that culture by enabling a high-performing, transparent, and efficient Dev-Ops interaction."

Try Shippable now and take your project to a new level