Upload files from S3 into Multi FileFeeds

Availability: Multi FileFeeds with S3 connections enabled.

Use Upload from S3 when a file is uploaded into OneSchema by being read from your S3 bucket. OneSchema assumes your configured IAM role, reads the source object, and copies it into OneSchema import storage. No file data passes through your application.

This is the recommended upload method when files already live in S3, especially for files larger than the 5 GiB presigned/direct upload limit. Upload from S3 supports files up to 20 GiB.

Looking for the opposite direction? See Import into S3 when OneSchema should write processed files into your S3 bucket.

Terminology

How it works

sequenceDiagram
    participant Client as API Client
    participant API as OneSchema API
    participant SrcS3 as Source S3 Bucket<br/>(your AWS account)
    participant DstS3 as OneSchema Imports Bucket<br/>(OneSchema AWS account)

    Client->>API: POST /upload-from-s3<br/>{s3_account_id, object_uri}
    API->>API: Assume OneSchemaS3ConnectionReader
    API->>API: Assume your IAM role<br/>(external ID)
    API->>SrcS3: HeadBucket + HeadObject
    SrcS3-->>API: Object metadata
    alt File ≤ 5 GiB
        API->>DstS3: CopyObject
        DstS3-->>API: 200 OK
    else File > 5 GiB
        API->>DstS3: Multipart copy
        DstS3-->>API: 200 OK
    end
    API-->>Client: 200 OK

1. Your API client sends the S3 URI and s3_account_id to OneSchema.
2. OneSchema assumes the intermediate OneSchemaS3ConnectionReader role in the OneSchema AWS account.
3. From that intermediate role, OneSchema assumes your customer IAM role using the configured external ID.
4. OneSchema verifies the source object and copies it directly from your S3 bucket into OneSchema's imports bucket.
5. The file is attached to the Multi FileFeed import.

The copy happens within AWS using your assumed role credentials. OneSchema does not download the file to your API client.

Prerequisites

Replace these placeholders with your values:

Your OneSchema team can provide ONESCHEMA_ACCOUNT_ID and ONESCHEMA_IMPORTS_BUCKET.

Step 1 — Create an S3 account in OneSchema

Create an S3 account with the IAM role ARN and external ID that OneSchema should use.

curl -X POST "https://api.oneschema.co/v0/s3-accounts" \
  -H "X-API-KEY: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production S3 Source",
    "role_arn": "arn:aws:iam::SOURCE_ACCOUNT_ID:role/CUSTOMER_S3_ROLE_NAME",
    "external_id": "YOUR_EXTERNAL_ID"
  }'

Save the returned id; this is the s3_account_id used when uploading from S3.

API reference: Create S3 Account

You can also create or edit S3 accounts from the Connections page. The Test connection account button verifies that OneSchema can assume the configured role. It does not validate access to a specific source bucket or file.

Step 2 — Configure the IAM role trust policy

Allow OneSchema's intermediate role to assume your customer IAM role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOneSchemaS3ConnectionReader",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ONESCHEMA_ACCOUNT_ID:role/OneSchemaS3ConnectionReader"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR_EXTERNAL_ID"
        }
      }
    }
  ]
}

The external ID in this trust policy must exactly match the external ID saved on the OneSchema S3 account.

Step 3 — Grant source bucket read permissions

Attach an IAM policy to CUSTOMER_S3_ROLE_NAME that allows OneSchema to find and read source objects.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListSourceBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::SOURCE_BUCKET"
    },
    {
      "Sid": "ReadSourceObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::SOURCE_BUCKET/*"
    }
  ]
}

If the source objects use a customer-managed KMS key, also grant:

{
  "Sid": "DecryptSourceObjects",
  "Effect": "Allow",
  "Action": ["kms:Decrypt"],
  "Resource": "arn:aws:kms:REGION:SOURCE_ACCOUNT_ID:key/YOUR_KMS_KEY_ID"
}

Step 4 — Grant destination copy permissions

Upload from S3 uses S3 CopyObject or multipart copy. AWS checks permissions on both sides of the copy:

1. Your role must be allowed to read from SOURCE_BUCKET.
2. Your role must be allowed to write into ONESCHEMA_IMPORTS_BUCKET.
3. The OneSchema imports bucket policy must allow that role or its assumed-role sessions to write.

Attach this IAM policy to CUSTOMER_S3_ROLE_NAME:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteOneSchemaImportsBucket",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::ONESCHEMA_IMPORTS_BUCKET/*"
    }
  ]
}

OneSchema also needs a matching bucket policy on ONESCHEMA_IMPORTS_BUCKET. Your OneSchema team handles this setup; share your role ARN:

arn:aws:iam::SOURCE_ACCOUNT_ID:role/CUSTOMER_S3_ROLE_NAME

If the role trust test succeeds but upload fails with an error like Unable to copy S3 object, this destination copy permission is the most common missing step.

Step 5 — Create a Multi FileFeed import

Create a new import on the target Multi FileFeed.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports" \
  -H "X-API-KEY: YOUR_API_KEY"

Response:

{
  "id": 101,
  "multi_file_feed_id": 42,
  "status": "initialized",
  "created_at": "2026-03-02T12:00:00Z"
}

Save the returned id as multi_file_feed_import_id.

API reference: Create a Multi FileFeed import

Step 6 — Upload files from S3

Call the upload-from-S3 endpoint for each source object. The object_uri must point to one file.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports/{multi_file_feed_import_id}/upload-from-s3

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports/101/upload-from-s3" \
  -H "X-API-KEY: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "s3_account_id": 1,
    "object_uri": "s3://SOURCE_BUCKET/path/to/orders.csv"
  }'

A 200 OK response means the file was copied into the import. Repeat for each file.

API reference: Upload a file from S3 to a Multi FileFeed import

Step 7 — Submit the import

After all files have been uploaded into OneSchema, submit the import for processing.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports/{multi_file_feed_import_id}/submit

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports/101/submit" \
  -H "X-API-KEY: YOUR_API_KEY"

API reference: Submit a Multi FileFeed import

Full example

import requests

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.oneschema.co"
MFF_ID = 42
S3_ACCOUNT_ID = 1
HEADERS = {"X-API-KEY": API_KEY}

s3_files = [
    "s3://SOURCE_BUCKET/data/orders.csv",
    "s3://SOURCE_BUCKET/data/customers.csv",
]

resp = requests.post(f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports", headers=HEADERS)
resp.raise_for_status()
import_id = resp.json()["id"]

for object_uri in s3_files:
    resp = requests.post(
        f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports/{import_id}/upload-from-s3",
        headers=HEADERS,
        json={"s3_account_id": S3_ACCOUNT_ID, "object_uri": object_uri},
    )
    resp.raise_for_status()

resp = requests.post(
    f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports/{import_id}/submit",
    headers=HEADERS,
)
resp.raise_for_status()

Troubleshooting

Limits