DevOps / SRE — Top Links Last Week

Week 26 Issue #33

Announcing the 2021 State of DevOps Report Sponsors

Google Cloud and the DORA research team are excited to announce our eight sponsors for the 2021 State of DevOps report. The report is the largest and longest running research of its kind. It provides an independent view into how teams and companies can drive powerful business outcomes, no matter what stage of the DevOps journey your team is in. The report captures how teams are driving excellence in technology delivery by implementing DevOps practices. The survey is a 25-min survey for the Dev Ops community to share how they are using DevOps to improve software delivery performance.

A Step by Step Guide to Building A Distributed, Spot-based Training Platform on AWS Using…

A Step by Step Guide to Building A Distributed, Spot-based Training Platform on AWS Using TorchElastic and Kubernetes. This is part II of a two-part series, describing our solution for running distributed training on spot instances. This post assumes you’ve read Part I of this series, and that you have a decent level of experience working with Kubernetes and AWS infrastructure. We will create a new EKS cluster called ‘EKS’ inside it, we will create two node groups.

Please consider supporting the Weekly DevOps / SRE Report. Subscribe to the phpops Newsletter on our website!

Kubernetes a black hole of unpredictable spend, according to new report

A report on Kubernetes expenditure from the Cloud Native Computing Foundation (CNCF), in association with the FinOps Foundation, shows that costs are rising and companies struggle to predict them accurately. Less than 25% of those surveyed said they could accurately predict how much they’d spend on the technology to within 5% of actual cost. The report is based on a survey of the CNCF and FinOp Foundation communities and although it had only 195 respondents, there was a "strong enterprise representation"

Lens 5 Released

With Lens 5, we are introducing a unified catalog to bring all your clusters, services, workloads, tools, tools and automations together for easy access. Hotbar is now the main navigation, allowing users to pick and choose the most important and commonly used functions (such as open cluster) from the catalog and assign those to the hotbar. Seamlessly manage access to clusters across local, on-prem, or public clouds all from within Lens. Secure access to the Kubernetes cluster is not always easy and make it super easy for Lens users to access and share access.

This Week in Programming: GitHub Copilot, Copyright Infringement and Open Source Licensing

GitHub introduced GitHub Copilot, a new AI-powered tool that autocompletes lines of code. The tool is built using OpenAI Codex, a system that the company says is “significantly more capable than GPT-3 in code generation” One of the first questions that have come up regarding Copilot has focused on copyright issues. Experts in intellectual property law took up the issue and, at least in their preliminary analysis, tended to agree with the idea of “fair use”

Tableau's latest update aims for AI enhanced augmented analytics, broader access to data

Tableau 2021.2 includes Ask Data with a new user interface and natural language search tools. Ask Data is getting tools to guide users on how to ask relevant questions as well as personalized dashboards. Tableau is integrating more artificial intelligence into its platform to broaden the base of people, roles and skill levels able to access analytics. The company is also adding an integration tab that will make it easier to connect and analyze data from Salesforce Sales and Service Cloud on Tableau Online. The new tools are aimed at democratizing data and creating an analytics culture.

10 GitHub Repositories for the Self-Taught Developer

This repository aims to create an open source curriculum for learning computer science. The repository covers prerequisites, core, and advanced courses required to have a well-rounded understanding of the fundamentals of the field. It also provides a strategy to learn these skills in a structured manner. This repository was created by John Washam, who works at Amazon as a software development engineer, to create a study plan to transition from a software/web developer into a software engineer at a FAANG company. The topics covered in the repository include: daily plan, Big-O notation, design patterns, threads/processes, networking, testing and testing.

What happens when Kubectl Run is executed?

Kubernetes uses a versioned API that is categorized into ‘API groups’ to categorize similar resources so that they’re easier to reason about. The API group of a Deployment is named apps, and its most recent version is v1. Therefore you need type APIVersion: apps/v1 at the top of Deployment manifests. To deploy a nginx pod, the below will be executed, which starts 3 nginx pods. To send the request successfully, kubectl needs to authenticate.

Journey to Container Orchestration!

There are 9 layers in a Cloud Stack and it is logically grouped into 3 areas Infrastructure, Platform, and Software. The coming of containers simplified the way a development & a deployment is performed. Containers are very light-weight, it has only what an application expects; nothing extra. As compared to the VM image, container images are 100x times smaller, hence lesser deployment time and more importantly lesser security vulnerabilities. Container requires only little resources as compared to VM’s image. Kubernetes is one of the leading container orchestration tools widely used in the market.

Application Containers, A closer Look

This article aims to demystify the concept of the application container. Containers allow us to run multiple containers within our host machine. The container is a standardized package of code, and all its dependencies and configurations needed to run in an isolated processes in user space — on an OS kernel. The OS kernel connects the system hardware to the application software by way of the OS system hardware. The kernel provides an API to the software that lets the application application run into the system’s hardware. We can split the OS from a bigger abstract view into two.

The best Infrastructure as Code tools for 2021

Infrastructure as Code tools are bridging the gap between IT infrastructure and IT infrastructure. The advent of cloud computing has revolutionized every IT segment. Using IaC config files, you can automate deployments of networks, virtual machines, servers, databases, etc. Using infrastructure as code tools, administrators are now able to write predefined configurations for each IT resource in the source code and automate IT infrastructure management. An unpatched vulnerability or a misconfigured template can enable hackers to gain access to sensitive resources.

What Are Microservices All About?

Microservices are a way to keep domain knowledge and business rules in one place. This way, each service (or person) has one task they know how to do best instead of one service knowing everything. Microservices — also known as the microservice architecture — is an architectural style that structures an application as a collection of services that are:. Highly maintainable and testable. Independently deployable. Decentralization of services. Separation of concerns (SOC) is at the heart of this architecture.

3 Enemies of Developer Velocity

Nica helps teams adopt serverless and optimize their costs on AWS. She is a serverless developer advocate for New Relic. In competitive markets, teams need to get code written and have new features built, tested and deployed at a regular rate. Unmanual consequences of tool selection and organizational architecture hamper some teams, making it harder for developers to get codes out the door. These mistakes aren’t just going to lead to delayed product releases; they can also have unexpected human costs.

Do Faster Development and Testing on Kubernetes Apps With Telepresence

Do Faster Development and Testing on Kubernetes Apps with Telepresence? Do it with the tools available on the market? The tools are available for developers and testers to improve their experience. They can be used to run apps locally and test them in the cloud using the tools that enable them to communicate with other services. The tools can also be used for testing and testing on the cloud. They are available in Amazon and Google's Cloud Kubernio, with the latest version of version of this article being published on Wednesday, October 1.

Why I built a Serverless Web Crawler

A technology studio tasked with a job to crawl and acquire product data from an eCommerce website. The parameters provided were to minimise the cost of computation as much as possible. The data only needed to be scraped from the website daily. The single target meant that we could deliver on the job with a single crawl script. AWS Glue was a consideration but couldn’t provision an instance as thin as the Crawl Script demanded. AWS Fargate was the most cost-effective compute engine for our small crawler.

How to Set Up AWS RDS

Sign in to the AWS console if you already have or else create a new account and log in. You will automatically be navigated to the region. For most cases, the Easy create option should work. I’ll be configuring my own database here in this tutorial. You can also configure your own DB. The creation of the database will take around 5 mins to set up and it should be up and running in the next 5 mins. The DB’s master username and password below are set up below.

Use Kool to Dockerize Your Local Development Environment the Right Way

Using Docker containers in local development environments has become commonplace for web development. Using Docker locally is still a challenge, oftentimes resulting in a frustrating developer experience. Kool (kool-dev/kool) offers a better way to use Docker locally. It simplifies the way you use Docker in your local environment by removing the Docker "speed bumps" that slow you down, greatly reducing the learning curve and error-prone area, and helping teams leverage containers at a lower cost of entry.

How I Deploy AWS CloudFormation Stack using Github Action

How to automate and deploy AWS CloudFormation using Github Actions is easier than you could imagine. There are a lot of open-source actions powered by the community which makes CI/CD easy & convenient. The project contains 2 key files: test-stack.yaml and a Github Action template. Add the Required AWS Credentials into the Github Secret. Write the triggers, deployment steps in the Github Actions template. Initialize the SDK using the SDK to deploy the CloudFormations stack to AWS Cloud.

Cloud Computing: An Overview

The practice of renting computing, storage and networking infrastructure from a vendor is called cloud computing. Rather than managing files on a local storage device, individuals and large corporations alike can now do so over the internet without having to purchase, install, run, and maintain a software application locally. Cloud computing is growing faster than ever: AWS, Azure and Google Cloud account for about 60% of the market. Cloud providers are turning to artificial intelligence programs, which rely on built-in algorithms to seek out and identify possible vulnerabilities in security measures.

Improved Volume Management, Docker Dev Environments and more in Desktop 3.5

Docker Desktop 3.5 is here and we can’t wait for you to try it! We’ve introduced some exciting new features including improvements to the Volume Management interface, a tech preview of Docker Dev Environments, and enhancements to Compose V2. We've added in the modified date, kind, and size of files so that you can quickly identify what is taking up all that space and decide if you can part with it. We're continuing to add more to volume management like the ability to share your volumes with your colleagues.

5 unusual Docker container use cases

Docker compose introduces a declarative way of running containers. Run a UI app in a container and connect to it via a web browser! Deployment is very easy. We are using Open Source software called digikam as an example, but this can be almost anything, like web browsers, Libre Office etc. Run a Linux desktop in a. container, that runs in a program, that. runs on Linux, which in turn runs on Windows… and we will access the Linux server via a browser!

The most important part of a successful software project

CICD is all about automating away tasks from developers to make their lives easier. There’s by no means a “best pipeline”, or the ‘best” or “the best” pipeline is the one made by your team, to serve your team. To show you the evolution of a simple pipeline, let's take a look at how the pipeline evolved in the project I mentioned earlier. You’ll see the process of continual improvement that provided developers with more and more confidence in what they were making.

An Elixir adoption success story

How a team that was new to Elixir over-delivered a big project in just three months. Elixir is only the right tool for the job if your engineers can wield it well. The Flatiron School chose Elixir for a new application that needed to be built. This decision wasn't just the right technical fit for the tasks at hand, it's also the reason why we were so successful in delivering on this project. The team had no Elixir experience at all, so a choice to build a Phoenix app represented a decision to adopt Elixir.

Onboarding Applications to Vault Using Terraform: A Practical Guide

Learn how to build an automated HashiCorp Vault onboarding system with Terraform using sensible naming standards, ACL policy templates, pre-created application entities, and workflows driven by VCS and CI/CD. At this stage, we recommend codifying Vault configurations and using infrastructure as code to manage Vault. The benefits of using IaC are truly realized when you are able to onboard applications to Vault hundreds of times through a single, automated workflow. Here is a flow chart outlining the key milestones of the onboarding process: The key stages of app onboarding onto Vault.