Self-Serve Cody infrastructure operations
This document describes operational guidance for Self-Serve Cody infrastructure. This service is operated on the Managed Services Platform (MSP).
If you need assistance with MSP infrastructure, reach out to the Core Services team in #discuss-core-services.
Service overview
| PROPERTY | DETAILS |
|---|---|
| Service ID | sams (specification) |
| Owners | cody-plg |
| Service kind | Cloud Run service |
| Environments | dev, prod |
| Docker image | us-central1-docker.pkg.dev/sourcegraph-dev/sams/accounts-server |
| Source code | github.com/sourcegraph/self-serve-cody - cmd/accounts-server |
Rollouts
| PROPERTY | DETAILS |
|---|---|
| Delivery pipeline | sams-us-central1-rollout |
| Stages | dev -> prod |
Changes to Self-Serve Cody are continuously delivered to the first stage (dev) of the delivery pipeline.
Promotion of a release to the next stage in the pipeline must be done manually using the GCP Delivery pipeline UI.
Environments
dev
| PROPERTY | DETAILS |
|---|---|
| Project ID | sams-dev-bfec |
| Category | test |
| Deployment type | rollout |
| Resources | dev Redis, dev PostgreSQL instance, dev BigQuery dataset |
| Slack notifications | #alerts-sams-dev |
| Alert policies | GCP Monitoring alert policies list, Dashboard |
| Errors | Sentry sams-dev |
| Domain | cody.sgdev.org |
| Cloudflare WAF | ✅ |
MSP infrastructure access needs to be requested using Entitle for time-bound privileges. Test environments may have less stringent requirements.
| ACCESS | ENTITLE REQUEST TEMPLATE |
|---|---|
| GCP project read access | Read-only Entitle request for the ‘Engineering Projects’ folder |
| GCP project write access | Write access Entitle request for the ‘Engineering Projects’ folder |
For Terraform Cloud access, see dev Terraform Cloud.
dev Cloud Run
The Self-Serve Cody dev service implementation is deployed on Google Cloud Run.
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud Run service |
| Service logs | GCP logging |
| Service traces | Cloud Trace |
| Service errors | Sentry sams-dev |
You can also use sg msp to quickly open a link to your service logs:
sg msp logs sams dev
dev Redis
| PROPERTY | DETAILS |
|---|---|
| Console | Memorystore Redis instances |
dev PostgreSQL instance
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud SQL instances |
| Databases | accounts, cody_management |
To connect to the PostgreSQL instance in this environment, use sg msp in the sourcegraph/managed-services repository:
# For read-only access
sg msp pg connect sams dev
# For write access - use with caution!
sg msp pg connect -write-access sams dev
dev BigQuery dataset
| PROPERTY | DETAILS |
|---|---|
| Dataset Project | sams-dev-bfec |
| Dataset ID | sams |
| Tables | user_emails, events, cody_events, subscription_events |
dev Architecture Diagram
dev Terraform Cloud
This service’s configuration is defined in sourcegraph/managed-services/services/sams/service.yaml, and sg msp generate sams dev generates the required infrastructure configuration for this environment in Terraform.
Terraform Cloud (TFC) workspaces specific to each service then provisions the required infrastructure from this configuration.
You may want to check your service environment’s TFC workspaces if a Terraform apply fails (reported via GitHub commit status checks in the sourcegraph/managed-services repository, or in #alerts-msp-tfc).
To access this environment’s Terraform Cloud workspaces, you will need to log in to Terraform Cloud and then request Entitle access to membership in the “Managed Services Platform Operator” TFC team. The “Managed Services Platform Operator” team has access to all MSP TFC workspaces.
The Terraform Cloud workspaces for this service environment are grouped under the msp-sams-dev tag, or you can use:
sg msp tfc view sams dev
prod
| PROPERTY | DETAILS |
|---|---|
| Project ID | sams-prod-ywuz |
| Category | external |
| Deployment type | rollout |
| Resources | prod Redis, prod PostgreSQL instance, prod BigQuery dataset |
| Slack notifications | #alerts-sams-prod |
| Alert policies | GCP Monitoring alert policies list, Dashboard |
| Errors | Sentry sams-prod |
| Domain | cody.sourcegraph.com |
| Cloudflare WAF | ✅ |
MSP infrastructure access needs to be requested using Entitle for time-bound privileges.
| ACCESS | ENTITLE REQUEST TEMPLATE |
|---|---|
| GCP project read access | Read-only Entitle request for the ‘Managed Services ’ folder |
| GCP project write access | Write access Entitle request for the ‘Managed Services’ folder |
For Terraform Cloud access, see prod Terraform Cloud.
prod Cloud Run
The Self-Serve Cody prod service implementation is deployed on Google Cloud Run.
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud Run service |
| Service logs | GCP logging |
| Service traces | Cloud Trace |
| Service errors | Sentry sams-prod |
You can also use sg msp to quickly open a link to your service logs:
sg msp logs sams prod
prod Redis
| PROPERTY | DETAILS |
|---|---|
| Console | Memorystore Redis instances |
prod PostgreSQL instance
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud SQL instances |
| Databases | accounts, cody_management |
To connect to the PostgreSQL instance in this environment, use sg msp in the sourcegraph/managed-services repository:
# For read-only access
sg msp pg connect sams prod
# For write access - use with caution!
sg msp pg connect -write-access sams prod
prod BigQuery dataset
| PROPERTY | DETAILS |
|---|---|
| Dataset Project | sams-prod-ywuz |
| Dataset ID | sams |
| Tables | user_emails, events, cody_events, subscription_events |
prod Architecture Diagram
prod Terraform Cloud
This service’s configuration is defined in sourcegraph/managed-services/services/sams/service.yaml, and sg msp generate sams prod generates the required infrastructure configuration for this environment in Terraform.
Terraform Cloud (TFC) workspaces specific to each service then provisions the required infrastructure from this configuration.
You may want to check your service environment’s TFC workspaces if a Terraform apply fails (reported via GitHub commit status checks in the sourcegraph/managed-services repository, or in #alerts-msp-tfc).
To access this environment’s Terraform Cloud workspaces, you will need to log in to Terraform Cloud and then request Entitle access to membership in the “Managed Services Platform Operator” TFC team. The “Managed Services Platform Operator” team has access to all MSP TFC workspaces.
The Terraform Cloud workspaces for this service environment are grouped under the msp-sams-prod tag, or you can use:
sg msp tfc view sams prod
Alert Policies
The following alert policies are defined for each of this service’s environments.
Cloud SQL - Connections
The number of Cloud SQL connections are approaching the maximum number of connections.
This can be caused by an increase in the number of active service instances.
Try increasing the 'resource.postgreSQL.maxConnections' configuration parameter.
Severity: WARNING
Cloud SQL - CPU Utilization
Cloud SQL instance CPU utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Disk Utilization
Cloud SQL instance disk utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Memory Utilization
Cloud SQL instance memory utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Server Availability
Cloud SQL instance is down.
Severity: WARNING
Cloud SQL - Spike in Per-Query Lock Time
Cloud SQL database queries encountered lock times well above acceptable thresholds.
Severity: WARNING
Cloud SQL - Sustained Per-Query Lock Times
Cloud SQL database queries are encountering lock times above acceptable thresholds over a window.
Severity: WARNING
High Container CPU Utilization
High CPU Usage - it may be neccessary to reduce load or increase CPU allocation
Severity: WARNING
High Container Memory Utilization
High Memory Usage - it may be neccessary to reduce load or increase memory allocation
Severity: WARNING
Container Startup Latency
Service containers are taking longer than configured timeouts to start up.
Severity: WARNING
Cloud Redis - System CPU Utilization
Redis Engine CPU Utilization goes above the set threshold. The utilization is measured on a scale of 0 to 1.
Severity: WARNING
Cloud Redis - Standard Instance Failover
Instance failover occured for a standard tier Redis instance.
Severity: WARNING
Cloud Redis - System Memory Utilization
Redis System memory utilization is above the set threshold. The utilization is measured on a scale of 0 to 1.
Severity: WARNING
High Ratio of 400 Responses
400 responses coming from the application. Does NOT include requests that did not reach the instance, e.g. if no host is available to receive a request - check GCP logs and error reports instead.
Severity: WARNING
High Ratio of 401 Responses
401 responses coming from the application. Does NOT include requests that did not reach the instance, e.g. if no host is available to receive a request - check GCP logs and error reports instead.
Severity: WARNING
High Ratio of 403 Responses
403 (forbidden) responses coming from the application. Does NOT include requests that did not reach the instance, e.g. if no host is available to receive a request - check GCP logs and error reports instead.
Severity: WARNING
High Ratio of 500 Responses
500 responses coming from the application. Does NOT include requests that did not reach the instance, e.g. if no host is available to receive a request - check GCP logs and error reports instead.
Severity: WARNING
Cloud Run Pending Requests
There are requests pending - we may need to increase Cloud Run instance count, request concurrency, or investigate further.
Severity: WARNING
Cloud Run Instance Precondition Failed
Cloud Run instance failed to start due to a precondition failure.
This is unlikely to cause immediate downtime, and may auto-resolve if no new instances are created and/or we return to a healthy state, but you should follow up to ensure the latest Cloud Run revision is healthy.
Severity: WARNING
External Uptime Check
Service is failing to repond on https://cody.sourcegraph.com - this may be expected if the service was recently provisioned or if its external domain has changed.
Severity: CRITICAL
Container Instance Count
There are a lot of Cloud Run instances running - we may need to increase per-instance requests make make sure we won't hit the configured max instance count
Severity: WARNING