GCP Cost Optimization Recommendations

After completing the GCP Cost Management Onboarding, you can further enhance CloudYali's cost optimization capabilities by exporting GCP recommendations to the same BigQuery dataset. This allows CloudYali to analyze and present actionable cost-saving opportunities based on Google's built-in recommendation engine.

Prerequisites

Before setting up GCP recommendations export to BigQuery, ensure that:

• You have completed the GCP Cost Management Onboarding process
• You have the necessary permissions in your GCP organization:
  • recommender.resources.exportRecommendations permission
  • bigquery.datasets.create permission (if you need to create a new dataset)
  • bigquery.tables.create permission
• BigQuery API is enabled in your project

Setting Up GCP Recommendations Export to BigQuery

Step 1: Identify Your BigQuery Dataset

Use the same dataset that you configured during the GCP Cost Management Onboarding process:

1. In the Google Cloud Console, navigate to BigQuery
2. Select the project where you've already set up billing export (as specified in the GCP Cost Management Onboarding)
3. Verify that the billing_export dataset (or your custom dataset name) exists

Step 2: Configure Recommendation Export in Google Cloud Console

1. Sign in to Google Cloud console

2. From the Home screen, click the Recommendations tab

3. Click Export to view the BigQuery export form

4. Select a Destination Project to store the recommendation data and click Next

   Note: If you can't find the project, then you either don't have the necessary permissions, or there aren't any projects available.

5. Click Enable APIs to enable the BigQuery APIs for the export. This can take several seconds to complete. Once done, click Continue

   Note: If the APIs are already enabled for this project, this step is skipped.

6. On the Configure Transfer form, provide the following details:

   • In the Transfer config name section, for Display name, enter CloudYali transfer

   • In the Schedule options section:

     • For Schedule, leave the default value (Start now)
     • For Repeats, select Daily
     • For Start date and run time, select UTC 00:00 as the default time

   • In the Destination settings section, for Destination dataset, choose the billing_export dataset (or your custom dataset name that you created during GCP Cost Management Onboarding)

   • In the Data source details section:

     • The default value for organization_id is the organization that you are currently viewing recommendations for. If you want to export recommendations to another organization, you can change this on top of the console in the organization viewer

     Note: Recommendations for projects without organizations are not exported.

7. Click Create to create the transfer

8. Once the transfer is created, you are directed back to the Recommendation Hub. You can click the link to access the transfer configuration details. Alternatively, you can access the transfers by doing the following:

   • Go to the BigQuery page in the Google Cloud console
   • Click Data Transfers. You can view all the available data transfers

It's important to use the same dataset that was configured during the GCP Cost Management Onboarding process to ensure CloudYali can access all your billing and recommendation data in one place.

Step 3: Verify Export Configuration

After setting up the export:

1. Navigate to the BigQuery studio in the Google Cloud Console
2. In the Explorer panel, expand your project and the billing_export dataset
3. Wait for the recommendations_export table to appear (this may take up to 24 hours after configuration)
4. Once the table appears, run a sample query to verify data is being exported:

SELECT 
  name, 
  recommender_subtype, 
  primary_impact.category,
  primary_impact.cost_projection.cost.currency_code,
  primary_impact.cost_projection.cost.units,
  primary_impact.cost_projection.cost.nanos,
  state,
  last_refresh_time
FROM 
  `PROJECT_ID.billing_export.recommendations_export` 
LIMIT 10

Replace PROJECT_ID with your actual Google Cloud project ID.

Step 4: Set Appropriate Permissions

Ensure that CloudYali has access to the recommendation data:

1. In the Google Cloud Console, navigate to IAM & Admin > IAM
2. Find the service account that you configured during GCP Cost Management Onboarding
3. Verify it has the BigQuery Data Viewer role for the project containing your dataset
4. If not, click the pencil icon to edit the service account's permissions and add the role
5. Click Save

If the service account doesn't have proper permissions, CloudYali won't be able to access your recommendation data, and the cost optimization features will be limited.

Available Recommendation Types

CloudYali processes all GCP recommendation types. The most common cost optimization recommendation types include:

Recommender Type	Description
google.compute.commitment.UsageCommitmentRecommender	Recommendations for committed use discounts
google.compute.disk.IdleResourceRecommender	Identifies idle or underutilized disk resources
google.compute.instance.IdleResourceRecommender	Identifies idle VM instances
google.compute.instance.MachineTypeRecommender	Suggests right-sizing VM instances
google.cloudsql.instance.IdleRecommender	Identifies idle Cloud SQL instances
google.cloudsql.instance.OverprovisionedRecommender	Identifies overprovisioned Cloud SQL instances
google.cloudsql.instance.OutOfDiskRecommender	Recommends disk size increases for Cloud SQL instances
google.iam.policy.Recommender	Identifies excess IAM permissions
google.gke.acceleratorVersionRecommender	Suggests GKE accelerator upgrades
google.gke.nodeVersionRecommender	Recommends GKE node version upgrades
google.loggingPrivateLogExclusion.Recommender	Suggests log exclusions to reduce logging costs
google.monitoring.metricFilterRecommender	Recommends metric filters to reduce monitoring costs

Accessing Recommendations in CloudYali

Once the recommendation data is exported to BigQuery and processed by CloudYali:

1. Log in to your CloudYali dashboard
2. Navigate to the Cost Optimization section
3. Select the GCP Recommendations tab
4. View all GCP cost recommendations prioritized by potential savings
5. Filter recommendations by type, resource, or potential impact
6. Track the lifecycle of recommendations, including when they were created, modified, or dismissed
7. Assign recommendations to team members for follow-up actions

Troubleshooting

If recommendations are not appearing in CloudYali after configuring the BigQuery export:

1. Verify that the export is working correctly in BigQuery by running sample queries
2. Check that CloudYali has the necessary permissions to access your BigQuery dataset:
   • The service account should have the BigQuery Data Viewer role
   • Verify the service account has access to both the dataset and the project
3. Ensure all recommendation types are included in your export configuration
4. Note that new exports may take up to 24 hours to generate initial data
5. Confirm that the recommendations_export table is being created in your BigQuery dataset
6. Check for any errors in the GCP Recommender logs or BigQuery logs

Next Steps

After setting up GCP recommendation exports to BigQuery:

• Review the CloudYali Cost Optimization Dashboard documentation to learn how to use these insights
• Explore how to set up automated alerts for new high-impact recommendations
• Learn about tagging governance to better attribute resources and recommendations to teams
• Set up scheduled reports to regularly review cost optimization opportunities with your team