Connect Multi FileFeeds to Google Cloud Storage

Availability: Multi FileFeeds with Google Cloud Storage connections enabled.

This guide walks through three things:

1. Creating a GCS connection in OneSchema, including running the setup script that provisions Workload Identity Federation in your GCP project.
2. Using that connection as a source on a Multi FileFeed, so OneSchema polls a bucket on a schedule and ingests new files.
3. Using that connection as a destination on a Multi FileFeed, so OneSchema writes processed output files into a bucket.

A single GCS connection can back both directions — you do not need separate connections for source and destination usage.

Terminology

How it works

OneSchema does not store long-lived service account keys. Instead, OneSchema's workloads in AWS exchange their AWS identity for short-lived GCP credentials that impersonate a service account in your GCP project (this is called Workload Identity Federation, or WIF).

sequenceDiagram
    participant OS as OneSchema (AWS)
    participant STS as Google STS
    participant SA as Your GCP Service Account
    participant GCS as Your GCS Bucket

    OS->>OS: Acquire AWS identity (IMDS)
    OS->>STS: Exchange AWS identity for federated token<br/>(via Workload Identity Pool + AWS provider)
    STS-->>OS: Federated token
    OS->>SA: Impersonate service account<br/>(scoped: devstorage.read_only or read_write)
    SA-->>OS: Short-lived OAuth token
    OS->>GCS: List / Get / Put objects
    GCS-->>OS: Response

A few things follow from this design:

• No keys to rotate. OneSchema never sees a JSON key. Credentials live for the duration of one operation.
• Scopes are issued per operation. Source reads use devstorage.read_only; destination writes use devstorage.read_write. A single service account backs both directions.
• Setup is one-time per project. The setup script provisions the Workload Identity Pool, the AWS provider inside it, the service account, and the impersonation grant. Once it's in place, you can create as many OneSchema connections as you like against the same project. You separately grant the service account access to whichever buckets it should read from or write to.

Prerequisites

You will need:

You will also need to know which buckets the connection should be able to read from and write to. After the setup script finishes, grant the service account roles/storage.objectViewer on each source bucket and roles/storage.objectAdmin (or equivalent) on each destination bucket. The setup script does not grant any bucket access for you.

Step 1 — Create a GCS connection in OneSchema

In the OneSchema dashboard, go to Connections → GCS → Add GCS connection.

In the connection modal, fill in:

• Name — a human-readable label (e.g. Acme — prod GCP).
• GCP Project ID — the string ID, like acme-data-prod. Not the numeric project number.

If you want non-default WIF resource names, expand Advanced and customize:

• Pool name (default oneschema) — the Workload Identity Pool ID OneSchema will look for in your project.
• Provider name (default aws-provider) — the AWS provider ID inside the pool.
• Service account name (default oneschema-gcs) — the service account OneSchema will impersonate (without the @PROJECT.iam.gserviceaccount.com suffix).

The defaults are fine for most setups. If you change them here, the generated script will match.

Click Generate script. OneSchema renders a bash script tailored to your inputs.

Step 2 — Run the setup script in GCP

Before running the script, make sure the project has the required APIs enabled. The script includes the command as a comment at the top — uncomment and run it once per project, or run it separately:

gcloud services enable iam.googleapis.com sts.googleapis.com iamcredentials.googleapis.com --project=PROJECT_ID

Then copy the rest of the generated script into a terminal authenticated to the correct GCP project and run it. The script will:

1. Look up the GCP project number and echo it (along with the other variables) for a sanity check before doing anything.
2. Create a Workload Identity Pool (using the pool name you chose).
3. Create an AWS provider inside that pool, trusting OneSchema's AWS account.
4. Create the service account (using the name you chose).
5. Grant the federated identity permission to impersonate the service account.
6. Print the project number on the final line, prefixed with Project Number:.

Copy the project number from the script output and paste it into the GCP Project Number field in the OneSchema dashboard.

After the script finishes, grant the service account access to the buckets it will read from or write to — the script does not do this for you. For example:

# Source bucket
gsutil iam ch \
  serviceAccount:oneschema-gcs@PROJECT_ID.iam.gserviceaccount.com:objectViewer \
  gs://your-source-bucket

# Destination bucket
gsutil iam ch \
  serviceAccount:oneschema-gcs@PROJECT_ID.iam.gserviceaccount.com:objectAdmin \
  gs://your-destination-bucket

If the script fails partway through (most commonly because a principal lacks roles/iam.workloadIdentityPoolAdmin), grant the missing permission and re-run only the remaining commands by hand. The script uses plain gcloud … create commands, so re-running it as-is will fail on resources that already exist.

Step 3 — Test the connection

Click Test connection. OneSchema performs a real token exchange (AWS → Google STS → service account impersonation) without saving anything. You'll see one of:

• Success — the WIF chain is wired up correctly.
• Access denied — the pool's trust policy doesn't accept OneSchema's AWS identity, or the impersonation grant on the service account is missing. Re-run the setup script.
• Platform error — typically a transient network issue or a misconfiguration in the pool/provider naming. Verify the advanced fields match what the script created.

Click Save. The connection appears in the GCS tab of the Connections page and is ready to use on Multi FileFeeds.

Step 4 — Use GCS as a source on a Multi FileFeed

A GCS source tells OneSchema to poll a bucket on a schedule, find new files matching a filter, and ingest them into Multi FileFeed imports.

In the Multi FileFeed settings, open the Source section and choose GCS as the source type. Fill in:

Behavior to know about

• Polling order. OneSchema lists objects in lexicographic key order, not in upload order.
• Deduplication is exact. OneSchema tracks every file it has ingested by (bucket, object key, generation). Reuploading a file under the same key creates a new generation, which OneSchema treats as a new file.
• Maximum file size: 1 GiB. Files larger than this are skipped, and the source shows a failure status on its next run until the offending file is removed or its size brought under the limit.
• Errors are per-source. A failure on one source does not stop other sources on other Multi FileFeeds. The source's status in the dashboard reflects the most recent run.
• Manual pulls. A source with no Pull frequency set is only ever polled when you click Pull now in the dashboard.

Step 5 — Use GCS as a destination on a Multi FileFeed

A GCS destination tells OneSchema to write the processed output files of a Multi FileFeed run into a bucket. There is no polling and no filter — the destination simply receives whatever the run produces.

In the Multi FileFeed settings, open the Destination section and choose GCS as the destination type. Fill in:

Behavior to know about

• Output path. Each output file is written to gs://BUCKET/PREFIX/{file_name} (or gs://BUCKET/{file_name} if prefix is empty). The filename is determined by the Multi FileFeed run.
• Existing objects. If an object already exists at the same key, GCS creates a new generation. Whether the prior generation is retained depends entirely on your bucket's Object Versioning setting — OneSchema does not enforce or change it. If you need each run to land at a unique key, include uniqueness in your output filenames or use a unique prefix per run.
• Content type. OneSchema infers Content-Type from the file extension (e.g. .csv → text/csv). Files without a recognized extension are written as application/octet-stream.
• Failure behavior. If a write fails (bucket missing, permission denied, transient platform error), the Multi FileFeed run is marked failed and the destination's status badge reflects the cause.

Troubleshooting

Limits and behavior