Skip to main content

GCP Cost Management Onboarding Guide

This guide will walk you through onboarding your GCP account for CloudYali Cost Management. This process involves exporting GCP billing data to BigQuery and allowing CloudYali access to that data for cost analysis.

Prerequisites​

Before starting the onboarding process, make sure you have:

  • A GCP account with Organization Administrator and Organization Role Administrator roles.
  • An active Cloud Billing Account.
  • A GCP project to contain the cloud billing export data. We recommend naming it something descriptive like cloudyali-realtime-integration.
  • A BigQuery dataset to store the billing information.
  • Cloud Billing Export enabled for detailed billing data.

Permissions Needed:​

  • Permission to create a project.
  • Permission to create a BigQuery dataset.
  • Billing Account Administrator role for the Cloud Billing account.
  • BigQuery User role for the project that contains the BigQuery dataset.

Onboarding Steps​

Step 1: Create a Host Project​

Create a GCP project to host all your billing data:

  1. Go to the GCP Console and create a new project named cloudyali-realtime-integration.
  2. Link the project to the Cloud Billing account you want to use for billing export.
    • Click the menu icon on the top-left corner and navigate to Billing.
    • If the project has no billing account, click Link A Billing Account.
    • Select the billing account you would like to associate the project with from the dropdown box.
note

If you have multiple billing accounts, you will need to enable Cloud Billing exports individually on each account.

Step 2: Create a Service Account​

From the dropdown menu on the top, switch to the organization’s view. You must be at the organization’s viewto set up organization-wide permissions for the service account.

  1. Navigate to IAM & Admin > Service Accounts.
  2. Click Create Service Account and name it CloudYali GCP Realtime Integration. Add appropriate description.e.g.This Service Account is used for CloudYali real-time cost monitoring as the Service account description.
  3. Click on CREATE AND CONTINUE Create Service Account
  4. Assign the following roles:
    • Pub/Sub Subscriber
    • Pub/Sub Viewer
    • Security Reviewer
    • Compute Network Viewer

Grant Service Account Access

  1. Note the email address of the service account you just created (CloudYali GCP Realtime Integration). Click on the menu icon (3 dots) under the action column for the service account that was just created and click on “Create Key”.

Create Service Account Key

  1. Keep the default JSON option selected and click "Create."

Save Service Key

note

Save the JSON, this will be used as input for CloudYali integration.

tip

If you face errors such as ‘Service account Key Creation is disabled’, update the organization policies to allow this operation.

Step 3: Create a BigQuery Dataset​

Create Bigquery dataset

  1. Navigate to BigQuery in the GCP console.
  2. In the Explorer panel, click on the three vertical dots next to your project ID and select Create dataset.
  3. Set the following values:
    • Dataset ID: We recommend something descriptive like billing_data.
    • Location type: Choose US(multiple regions in United States) or EU(multiple regions in European Union).
    • Encryption: Set to Google-managed encryption key.
    • Table expiration: Leave the Enable table expiration checkbox unchecked to ensure that data never expires.

Configure Bigquery dataset

Step 4: Enable Cloud Billing Export​

  1. In the GCP Console, while still in BigQuery, click the hamburger menu (three horizontal lines) at the top left of the console and select Billing.
  2. In the left Billing navigation menu, select Billing export. (You can also search for and navigate to Billing export from the main search bar at the top of the GCP console).
  3. Under Detailed usage cost, select the BigQuery dataset you just created and click EDIT SETTINGS.
  4. Configure the following settings:
    • From the Projects list, select the project you set up (e.g. cloudyali-realtime-integration) to contain your billing data.
    • For Dataset, select the dataset that you set up to contain your exported Cloud Billing data (e.g., billing_data).
  5. Click Save to enable the billing export.

Bigquery Billing Export

Part 2: Granting Access to CloudYali​

Step 1: Grant Service Account Permission to Access BigQuery​

  1. Go to IAM & Admin in the GCP console.
  2. At the top of the IAM console, select the project that hosts the BigQuery dataset with your Cloud Billing export data (e.g., cloudyali-realtime-integration).
  3. In the center of the page, under Permissions for the project you created earlier (e.g., cloudyali-realtime-integration), Click + GRANT ACCESS.
  4. In the New Principals field, under Add Principals, select the service account you created earlier.
  5. In the Role field, under Assign roles, search for and select BigQuery Job User and SAVE.

Bigquery Job User Role

Step 2: Grant the Service Account Permission to Access the BigQuery Dataset​

Go back to BigQuery, and complete the steps below to grant the service account permission to access the BigQuery dataset.

  1. At the top of the BigQuery console, ensure the project that you set up to contain your billing data is selected e.g. cloudyali-realtime-integration.
  2. In the Explorer panel, select your project (cloudyali-realtime-integration) to expand it.
  3. Select the three vertical dots next to the dataset name (eg. billing_data), then click Open. The Dataset info will be displayed on the right. Keep this screen open to later obtain your project ID and dataset name.
  4. Select the three vertical dots next to the dataset name again, then click Share.

Bigquery Dataset Permissions

  1. On the Share permissions panel that appears on the right, click + ADD PRINCIPAL.
  2. In the New principals field, under Add principals, search for and select the Service Account.
  3. In the Role field, under Assign roles, search for and select BigQuery Data Viewer and click SAVE.

BigQuery Data Viewer

Step 3: Grant Viewer Role at Organization Level​

  1. Navigate to IAM at the organizational level.

Organization Level Permissions

  1. In the center of the page, under Permissions for organizations, click + Grant Access.
  2. Under Add principals, add Service Account for the CloudYali integration project.
  3. Under Assign roles, click Basic and assign the Viewer role to allow read-only access to all projects.

Grant Viewer Access

Step 4: Provide Your GCP Configuration Information to CloudYali​

Please keep the following information ready:

  • Billing Account ID
    • Go to GCP Billing.
    • Copy the value for your billing account, displayed in the Billing account ID column. Eg. in the below image the billing account id is 01507E-FD2170-C43264 Billing Account ID
  • Project ID, where the billing dataset is created. e.g. cloudyali-relatime-integration
  • The BigQuery Dataset Name, eg. billing_data, that you created earlier. Billing Dataset Information

1: Log in to CloudYali Console

  • Click on your username in the upper right corner of the CloudYali console.
  • Select Settings from the dropdown menu.
  • Click on Add AWS Accounts button.

CloudYali GCP Integration

note

It usually takes a few hours for data to begin appearing. When detailed usage cost data is configured, Cloud Billing data is added retroactively for both the current and previous month. Please note that it can take up to five days for the full data for the current and previous month to be completely available.

Summary​

After following the above steps, your GCP account will be onboarded for CloudYali Cost Management. It may take a few hours for billing data to appear, and data from the current and previous month may take up to five days to fully propagate.

© 2024 CloudYali. All rights reserved.

Last updated on by Nishant Thorat