Azure Infrastructure: Cost Optimization

On the week of November 13th through 15th, we held a workshop with Microsoft to discuss different ways through which washu could save on its Azure resource cost. The meeting was very engaging and knowledgeable, few members of the platform engineering team were present in the working session as well. This document is just a summary of the strategies that I learned will be implementing within our Platform engineering team.

Azure Workbooks:

• Azure workbooks provide a customizable way to visualize data within azure. It is also great when wanting to visualize cost-related data, it can also identify inefficient & orphan resources that can help us save on cost.
• During the meeting, Microsoft provided three customizable samples that we could use to save on cost.
• To access our list of workbooks, follow this link

Steps to create Workbooks:

• To create a workbook, go to Azure Monitor > Workbooks > select + New to create the workbook.

• There are currently about 25+ workbooks within our Azure environment.
• Additionally, there are 3 workbooks that were provided by the Microsoft team.
• We are currently utilizing Orphaned_Resource_Groups, Tags_governance, VBD Cost, Storage accounts Overview.
  • With these workbooks, you can filter high-cost resources or high cost resources by regions
  • Additionally, these workbooks can also be used to view resource utilization and orphaned resources within our environment.

Orphaned Resources Workbook:

• The orphaned resource workbook lets you know what resources are not recently been utilized.

• Based on the screenshot, there are currently 13 orphaned disks, 3 network interface, 9 public ip addresses, 10 resource groups that are harboring no resources, 17 Network security groups that are orphaned resources.
• To remediate these orphaned resources, I will put together a list & discuss it with the team.

VBD Cost Workbook:

• The VBD cost workbook provides cost saving recommendations per subscription.
• Few of these recommendations consist of downsizing Database and utilizing reserved instance for our virtual machines.

Tag-Governance Workbook

• The Tag-Governance workbook provides a visual interpretation of what geographical location where most of our resources are deployed.

• The workbook also provides a list of utilized tags and their location.

Storage Accounts Workbook:

• The storage account workbook provides an insight into our storage account utilization such as the number of transactions, errors and server latency.

Azure Budgets

• Another tool that was discuss during the working session was Amazon Budget.
• It is a service that helps track our spending and can alert us when our cost exceeds a threshold.

Steps to create budget

• Go to Cost Management > Monitoring > Select Budgets > + add.
• As shown, below, we there’s currently an alert for our redcap subscription that sets the monthly limit to $20,000 based on the forecast amount.

• The budgets resource can also provide daily, monthly and yearly expenditures that can be categorized by Subscriptions, resource groups.

Reservations + Hybrid Benefits

• Within Cost Management, there’s another resource called Reservation + Hybrid.
• This resource allows us to optimize cloud expenditures.
• Azure reservations allows you to commit to one-or-three-year & a pay-as-you-go plans for specific azure services that allows you to save up to 70% of the running cost.

Azure Tagging

• Another way to save on cost is to utilize resource tags.
• Resource tag allows us to categorize and track Azure resources by their purpose, owner or department.
• We currently have a tag that is tracked by databricks, this tag is wusmdepartment.
• The goal moving forward is for PE to utilize resource tagging to ease tracking of all resources thus making it easier for us to monitor cost and resource non utilization.

Tagging Strategy:

• The goal moving forward is to implement 4 types of tagging:
  • Environment: This includes the environment which the resource will launch in.
    • Environments are typically made up of POC, DEV, QA & PROD.
  • Owner: Who owns the projects i.e., individuals or team (PE = platform engineering)
  • CostCenter: Department or project cost center.
  • wusmdepartment: This tag allows databricks to track resource usage and billing.
  • Purpose: This is optional as it can be used to define the purpose of the service been lunched.
• We can either apply these tags at the time of resource creation or I can develop a policy to automate the application of tags.

Analyze cost by tags:

• You can analyze tags by utilizing Cost Management > use the Cost by Resource view and filter by tags.

Optimize Tagged Resources:

• We can shut down and resize non-critical resources by tags i.e., resources tagged as DEV, POC that are running after hours.