Release Registry infrastructure operations
This document describes operational guidance for Release Registry infrastructure. This service is operated on the Managed Services Platform (MSP).
If you need assistance with MSP infrastructure, reach out to the Core Services team in #discuss-core-services.
Service overview
| PROPERTY | DETAILS |
|---|---|
| Service ID | releaseregistry (specification) |
| Owners | dev-experience |
| Service kind | Cloud Run service |
| Environments | prod, dev |
| Docker image | us.gcr.io/sourcegraph-dev/releaseregistry |
| Source code | github.com/sourcegraph/releaseregistry - . |
Rollouts
| PROPERTY | DETAILS |
|---|---|
| Delivery pipeline | releaseregistry-us-central1-rollout |
| Stages | dev -> prod |
Changes to Release Registry are continuously delivered to the first stage (dev) of the delivery pipeline.
Promotion of a release to the next stage in the pipeline must be done manually using the GCP Delivery pipeline UI.
Environments
prod
| PROPERTY | DETAILS |
|---|---|
| Project ID | releaseregistry-prod-5421 |
| Category | internal |
| Deployment type | rollout |
| Resources | prod PostgreSQL instance |
| Slack notifications | #alerts-releaseregistry-prod |
| Alert policies | GCP Monitoring alert policies list, Dashboard |
| Errors | Sentry releaseregistry-prod |
| Domain | releaseregistry.sourcegraph.com |
| Cloudflare WAF | ✅ |
MSP infrastructure access needs to be requested using Entitle for time-bound privileges.
| ACCESS | ENTITLE REQUEST TEMPLATE |
|---|---|
| GCP project read access | Read-only Entitle request for the ‘Internal Services’ folder |
| GCP project write access | Write access Entitle request for the ‘Internal Services’ folder |
For Terraform Cloud access, see prod Terraform Cloud.
prod Cloud Run
The Release Registry prod service implementation is deployed on Google Cloud Run.
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud Run service |
| Service logs | GCP logging |
| Service traces | Cloud Trace |
| Service errors | Sentry releaseregistry-prod |
You can also use sg msp to quickly open a link to your service logs:
sg msp logs releaseregistry prod
prod PostgreSQL instance
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud SQL instances |
| Databases | releaseregistry |
To connect to the PostgreSQL instance in this environment, use sg msp in the sourcegraph/managed-services repository:
# For read-only access
sg msp pg connect releaseregistry prod
# For write access - use with caution!
sg msp pg connect -write-access releaseregistry prod
prod Architecture Diagram
prod Terraform Cloud
This service’s configuration is defined in sourcegraph/managed-services/services/releaseregistry/service.yaml, and sg msp generate releaseregistry prod generates the required infrastructure configuration for this environment in Terraform.
Terraform Cloud (TFC) workspaces specific to each service then provisions the required infrastructure from this configuration.
You may want to check your service environment’s TFC workspaces if a Terraform apply fails (reported via GitHub commit status checks in the sourcegraph/managed-services repository, or in #alerts-msp-tfc).
To access this environment’s Terraform Cloud workspaces, you will need to log in to Terraform Cloud and then request Entitle access to membership in the “Managed Services Platform Operator” TFC team. The “Managed Services Platform Operator” team has access to all MSP TFC workspaces.
The Terraform Cloud workspaces for this service environment are grouped under the msp-releaseregistry-prod tag, or you can use:
sg msp tfc view releaseregistry prod
dev
| PROPERTY | DETAILS |
|---|---|
| Project ID | releaseregistry-dev-6bac |
| Category | test |
| Deployment type | rollout |
| Resources | dev PostgreSQL instance |
| Slack notifications | #alerts-releaseregistry-dev |
| Alert policies | GCP Monitoring alert policies list, Dashboard |
| Errors | Sentry releaseregistry-dev |
| Domain | releaseregistry.sgdev.org |
| Cloudflare WAF | ✅ |
MSP infrastructure access needs to be requested using Entitle for time-bound privileges. Test environments may have less stringent requirements.
| ACCESS | ENTITLE REQUEST TEMPLATE |
|---|---|
| GCP project read access | Read-only Entitle request for the ‘Engineering Projects’ folder |
| GCP project write access | Write access Entitle request for the ‘Engineering Projects’ folder |
For Terraform Cloud access, see dev Terraform Cloud.
dev Cloud Run
The Release Registry dev service implementation is deployed on Google Cloud Run.
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud Run service |
| Service logs | GCP logging |
| Service traces | Cloud Trace |
| Service errors | Sentry releaseregistry-dev |
You can also use sg msp to quickly open a link to your service logs:
sg msp logs releaseregistry dev
dev PostgreSQL instance
| PROPERTY | DETAILS |
|---|---|
| Console | Cloud SQL instances |
| Databases | releaseregistry |
To connect to the PostgreSQL instance in this environment, use sg msp in the sourcegraph/managed-services repository:
# For read-only access
sg msp pg connect releaseregistry dev
# For write access - use with caution!
sg msp pg connect -write-access releaseregistry dev
dev Architecture Diagram
dev Terraform Cloud
This service’s configuration is defined in sourcegraph/managed-services/services/releaseregistry/service.yaml, and sg msp generate releaseregistry dev generates the required infrastructure configuration for this environment in Terraform.
Terraform Cloud (TFC) workspaces specific to each service then provisions the required infrastructure from this configuration.
You may want to check your service environment’s TFC workspaces if a Terraform apply fails (reported via GitHub commit status checks in the sourcegraph/managed-services repository, or in #alerts-msp-tfc).
To access this environment’s Terraform Cloud workspaces, you will need to log in to Terraform Cloud and then request Entitle access to membership in the “Managed Services Platform Operator” TFC team. The “Managed Services Platform Operator” team has access to all MSP TFC workspaces.
The Terraform Cloud workspaces for this service environment are grouped under the msp-releaseregistry-dev tag, or you can use:
sg msp tfc view releaseregistry dev
Alert Policies
The following alert policies are defined for each of this service’s environments.
Cloud SQL - Connections
The number of Cloud SQL connections are approaching the maximum number of connections.
This can be caused by an increase in the number of active service instances.
Try increasing the 'resource.postgreSQL.maxConnections' configuration parameter.
Severity: WARNING
Cloud SQL - CPU Utilization
Cloud SQL instance CPU utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Disk Utilization
Cloud SQL instance disk utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Memory Utilization
Cloud SQL instance memory utilization is above acceptable threshold.
Severity: WARNING
Cloud SQL - Server Availability
Cloud SQL instance is down.
Severity: WARNING
Cloud SQL - Spike in Per-Query Lock Time
Cloud SQL database queries encountered lock times well above acceptable thresholds.
Severity: WARNING
Cloud SQL - Sustained Per-Query Lock Times
Cloud SQL database queries are encountering lock times above acceptable thresholds over a window.
Severity: WARNING
High Container CPU Utilization
High CPU Usage - it may be neccessary to reduce load or increase CPU allocation
Severity: WARNING
High Container Memory Utilization
High Memory Usage - it may be neccessary to reduce load or increase memory allocation
Severity: WARNING
Container Startup Latency
Service containers are taking longer than configured timeouts to start up.
Severity: WARNING
Cloud Run Pending Requests
There are requests pending - we may need to increase Cloud Run instance count, request concurrency, or investigate further.
Severity: WARNING
Cloud Run Instance Precondition Failed
Cloud Run instance failed to start due to a precondition failure.
This is unlikely to cause immediate downtime, and may auto-resolve if no new instances are created and/or we return to a healthy state, but you should follow up to ensure the latest Cloud Run revision is healthy.
Severity: WARNING
External Uptime Check
Service is failing to repond on https://releaseregistry.sgdev.org - this may be expected if the service was recently provisioned or if its external domain has changed.
Severity: CRITICAL