science.alt.dataset.storageS3 Documentation (EN)

main object

S3 or S3-compatible storage for WebDataset tar archives. Supports custom endpoints for MinIO, Cloudflare R2, and other S3-compatible services.

Properties

bucket string Required

S3 bucket name

maxLength: 255 bytes

endpoint string uri Optional

Custom S3-compatible endpoint URL (e.g., for MinIO, Cloudflare R2). Omit for standard AWS S3.

maxLength: 500 bytes

region string Optional

AWS region (e.g., 'us-east-1'). Optional for S3-compatible services.

maxLength: 50 bytes

shards array of ref#shardEntry Required

Array of shard entries with object key and integrity checksum

minLength: 1 items

View raw schema

{
  "type": "object",
  "required": [
    "bucket",
    "shards"
  ],
  "properties": {
    "bucket": {
      "type": "string",
      "maxLength": 255,
      "description": "S3 bucket name"
    },
    "region": {
      "type": "string",
      "maxLength": 50,
      "description": "AWS region (e.g., 'us-east-1'). Optional for S3-compatible services."
    },
    "shards": {
      "type": "array",
      "items": {
        "ref": "#shardEntry",
        "type": "ref"
      },
      "minLength": 1,
      "description": "Array of shard entries with object key and integrity checksum"
    },
    "endpoint": {
      "type": "string",
      "format": "uri",
      "maxLength": 500,
      "description": "Custom S3-compatible endpoint URL (e.g., for MinIO, Cloudflare R2). Omit for standard AWS S3."
    }
  },
  "description": "S3 or S3-compatible storage for WebDataset tar archives. Supports custom endpoints for MinIO, Cloudflare R2, and other S3-compatible services."
}

shardEntry object

A single S3 object shard with integrity checksum

Properties

checksum ref science.alt.dataset.entry#shardChecksum Required

Content hash for integrity verification

key string Required

S3 object key for this WebDataset tar shard

maxLength: 1024 bytes

View raw schema

{
  "type": "object",
  "required": [
    "key",
    "checksum"
  ],
  "properties": {
    "key": {
      "type": "string",
      "maxLength": 1024,
      "description": "S3 object key for this WebDataset tar shard"
    },
    "checksum": {
      "ref": "science.alt.dataset.entry#shardChecksum",
      "type": "ref",
      "description": "Content hash for integrity verification"
    }
  },
  "description": "A single S3 object shard with integrity checksum"
}