Upload files from S3 into Multi FileFeeds

Availability: Multi FileFeeds with S3 connections enabled.

Use Upload from S3 when a file is uploaded into OneSchema by being read from your S3 bucket. OneSchema assumes your configured IAM role, reads the source object, and writes it into OneSchema import storage. No file data passes through your application.

This is the recommended upload method when files already live in S3, especially for files larger than the 5 GiB presigned/direct upload limit. Upload from S3 supports files up to 20 GiB.

For OneSchema-hosted deployments, the recommended setup only requires source bucket read access. Self-hosted deployments may optionally use direct bucket copy, which also requires write access to OneSchema import storage.

Looking for the opposite direction? See Import into S3 when OneSchema should write processed files into your S3 bucket.

Terminology

How it works

For OneSchema-hosted deployments, OneSchema reads from your source bucket using your configured IAM role and writes to OneSchema import storage using OneSchema-controlled storage credentials.

sequenceDiagram
    participant Client as API Client
    participant API as OneSchema API
    participant SrcS3 as Source S3 Bucket<br/>(your AWS account)
    participant Storage as OneSchema Import Storage

    Client->>API: POST /upload-from-s3<br/>{s3_account_id, object_uri}
    API->>API: Assume OneSchemaS3ConnectionReader
    API->>API: Assume your IAM role<br/>(external ID)
    API->>SrcS3: HeadBucket + HeadObject
    SrcS3-->>API: Object metadata
    alt File ≤ 100 MiB
        API->>SrcS3: GetObject
        SrcS3-->>API: Object bytes
        API->>Storage: PutObject<br/>(OneSchema credentials)
    else File > 100 MiB
        API->>SrcS3: Ranged GetObject parts
        SrcS3-->>API: Object bytes
        API->>Storage: Multipart upload<br/>(OneSchema credentials)
    end
    API-->>Client: 200 OK

1. Your API client sends the S3 URI and s3_account_id to OneSchema.
2. OneSchema assumes the intermediate OneSchemaS3ConnectionReader role in the OneSchema AWS account.
3. From that intermediate role, OneSchema assumes your customer IAM role using the configured external ID.
4. OneSchema verifies and reads the source object.
5. OneSchema writes the file into OneSchema import storage and attaches it to the Multi FileFeed import.

Prerequisites

Replace these placeholders with your values:

Your OneSchema team can provide ONESCHEMA_ACCOUNT_ID.

Step 1 — Create an S3 account in OneSchema

Create an S3 account with the IAM role ARN and external ID that OneSchema should use.

curl -X POST "https://api.oneschema.co/v0/s3-accounts" \
  -H "X-API-KEY: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production S3 Source",
    "role_arn": "arn:aws:iam::SOURCE_ACCOUNT_ID:role/CUSTOMER_S3_ROLE_NAME",
    "external_id": "YOUR_EXTERNAL_ID"
  }'

Save the returned id; this is the s3_account_id used when uploading from S3.

API reference: Create S3 Account

You can also create or edit S3 accounts from the Connections page. The Test connection account button verifies that OneSchema can assume the configured role. It does not validate access to a specific source bucket or file.

Step 2 — Configure the IAM role trust policy

Allow OneSchema's intermediate role to assume your customer IAM role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOneSchemaS3ConnectionReader",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ONESCHEMA_ACCOUNT_ID:role/OneSchemaS3ConnectionReader"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "YOUR_EXTERNAL_ID"
        }
      }
    }
  ]
}

The external ID in this trust policy must exactly match the external ID saved on the OneSchema S3 account.

Step 3 — Grant source bucket read permissions

Attach an IAM policy to CUSTOMER_S3_ROLE_NAME that allows OneSchema to find and read source objects.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListSourceBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::SOURCE_BUCKET"
    },
    {
      "Sid": "ReadSourceObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::SOURCE_BUCKET/*"
    }
  ]
}

If the source objects use a customer-managed KMS key, also grant:

{
  "Sid": "DecryptSourceObjects",
  "Effect": "Allow",
  "Action": ["kms:Decrypt"],
  "Resource": "arn:aws:kms:REGION:SOURCE_ACCOUNT_ID:key/YOUR_KMS_KEY_ID"
}

Step 4 — Create a Multi FileFeed import

Create a new import on the target Multi FileFeed.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports" \
  -H "X-API-KEY: YOUR_API_KEY"

Response:

{
  "id": 101,
  "multi_file_feed_id": 42,
  "status": "initialized",
  "created_at": "2026-03-02T12:00:00Z"
}

Save the returned id as multi_file_feed_import_id.

API reference: Create a Multi FileFeed import

Step 5 — Upload files from S3

Call the upload-from-S3 endpoint for each source object. The object_uri must point to one file.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports/{multi_file_feed_import_id}/upload-from-s3

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports/101/upload-from-s3" \
  -H "X-API-KEY: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "s3_account_id": 1,
    "object_uri": "s3://SOURCE_BUCKET/path/to/orders.csv"
  }'

A 200 OK response means the file was copied into the import. Repeat for each file.

API reference: Upload a file from S3 to a Multi FileFeed import

Step 6 — Submit the import

After all files have been uploaded into OneSchema, submit the import for processing.

POST /v0/multi-file-feeds/{multi_file_feed_id}/imports/{multi_file_feed_import_id}/submit

curl -X POST "https://api.oneschema.co/v0/multi-file-feeds/42/imports/101/submit" \
  -H "X-API-KEY: YOUR_API_KEY"

API reference: Submit a Multi FileFeed import

Full example

import requests

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.oneschema.co"
MFF_ID = 42
S3_ACCOUNT_ID = 1
HEADERS = {"X-API-KEY": API_KEY}

s3_files = [
    "s3://SOURCE_BUCKET/data/orders.csv",
    "s3://SOURCE_BUCKET/data/customers.csv",
]

resp = requests.post(f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports", headers=HEADERS)
resp.raise_for_status()
import_id = resp.json()["id"]

for object_uri in s3_files:
    resp = requests.post(
        f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports/{import_id}/upload-from-s3",
        headers=HEADERS,
        json={"s3_account_id": S3_ACCOUNT_ID, "object_uri": object_uri},
    )
    resp.raise_for_status()

resp = requests.post(
    f"{BASE_URL}/v0/multi-file-feeds/{MFF_ID}/imports/{import_id}/submit",
    headers=HEADERS,
)
resp.raise_for_status()

Troubleshooting

Limits

Self-hosted deployments: direct bucket copy

For self-hosted or dedicated deployments where your role is allowed to write into OneSchema import storage, Upload from S3 can use direct S3 bucket copy.

sequenceDiagram
    participant Client as API Client
    participant API as OneSchema API
    participant SrcS3 as Source S3 Bucket<br/>(your AWS account)
    participant DstS3 as OneSchema Imports Bucket<br/>(OneSchema AWS account)

    Client->>API: POST /upload-from-s3<br/>{s3_account_id, object_uri}
    API->>API: Assume OneSchemaS3ConnectionReader
    API->>API: Assume your IAM role<br/>(external ID)
    API->>SrcS3: HeadBucket + HeadObject
    SrcS3-->>API: Object metadata
    alt File ≤ 5 GiB
        API->>DstS3: CopyObject
        DstS3-->>API: 200 OK
    else File > 5 GiB
        API->>DstS3: Multipart copy
        DstS3-->>API: 200 OK
    end
    API-->>Client: 200 OK

In this mode, the copy happens within AWS using your assumed role credentials. OneSchema does not download the file to your API client.

Self-hosted prerequisites

Replace these additional placeholders with your values. Your OneSchema team provides ONESCHEMA_IMPORTS_BUCKET.

Grant destination copy permissions

Direct bucket copy uses S3 CopyObject or multipart copy. AWS checks permissions on both sides of the copy:

1. Your role must be allowed to read from SOURCE_BUCKET.
2. Your role must be allowed to write into ONESCHEMA_IMPORTS_BUCKET.
3. The OneSchema imports bucket policy must allow that role or its assumed-role sessions to write.

Attach this IAM policy to CUSTOMER_S3_ROLE_NAME:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteOneSchemaImportsBucket",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::ONESCHEMA_IMPORTS_BUCKET/*"
    }
  ]
}

OneSchema also needs a matching bucket policy on ONESCHEMA_IMPORTS_BUCKET. Your OneSchema team handles this setup for self-hosted deployments; share your role ARN:

arn:aws:iam::SOURCE_ACCOUNT_ID:role/CUSTOMER_S3_ROLE_NAME

If the role trust test succeeds but direct bucket copy fails with an error like Unable to copy S3 object, this destination copy permission is the most common missing step.

Self-hosted troubleshooting