Data-driven cost management for a cloud-based service across autonomous teams

Sammanfattning: Spotify started to use the cloud-based data warehouse BigQuery in 2016 with a business model of pay-as-you-go. Since then, usage has increased rapidly in volume and amount of users across the organisation which is a result of ease of use compared to previous data warehouse solutions. The technology procurement team lacks an overview of how BigQuery is used across Spotify and a strategy of how to maintain an environment where users make cost informed decisions when designing queries and creating tables. Incidents resulting in unexpected high bills are currently handled by a capacity analyst using billing data which is lacking the granularity of how cost maps to the users of BigQuery. The objective of this research is to provide recommendations on how audit data can enable a data driven cost-effective environment for BigQuery across the matrix formed engineering organisation at Spotify. First an overview of the current usage patterns is presented based on audit data which is modeled with regards to volume, complexity and utilization. Different patterns are identified using K-means clustering, including high-volume consuming squads and underutilized tables. Secondly, recommendations on transparency of audit data for cost-effectiveness are based on insights from cluster analysis, interviews and characteristics of organisation structure. Recommendations include transparency of data consumption to producers to prevent paying for unused resources and transparency to consumers on usage patterns to avoid paying for unexpected bills. Usage growth is recommended to be presented to the technology procurement squad which enables better understanding and mitigates challenges on cost forecasting and control.