science.alt.dataset.storageHttp

lexicon.store View official

Documentation

HTTP/HTTPS storage for WebDataset tar archives. Each shard is listed individually with a checksum for integrity verification. Consumers build brace-expansion patterns on the fly when needed.

main object

HTTP/HTTPS storage for WebDataset tar archives. Each shard is listed individually with a checksum for integrity verification. Consumers build brace-expansion patterns on the fly when needed.

Properties

shards array of ref#shardEntry Required

Array of shard entries with URL and integrity checksum

minLength: 1 items
View raw schema 
{
  "type": "object",
  "required": [
    "shards"
  ],
  "properties": {
    "shards": {
      "type": "array",
      "items": {
        "ref": "#shardEntry",
        "type": "ref"
      },
      "minLength": 1,
      "description": "Array of shard entries with URL and integrity checksum"
    }
  },
  "description": "HTTP/HTTPS storage for WebDataset tar archives. Each shard is listed individually with a checksum for integrity verification. Consumers build brace-expansion patterns on the fly when needed."
}
shardEntry object

A single HTTP-accessible shard with integrity checksum

Properties

url string uri Required

HTTP/HTTPS URL for this WebDataset tar shard

maxLength: 2000 bytes
View raw schema 
{
  "type": "object",
  "required": [
    "url",
    "checksum"
  ],
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "maxLength": 2000,
      "description": "HTTP/HTTPS URL for this WebDataset tar shard"
    },
    "checksum": {
      "ref": "science.alt.dataset.entry#shardChecksum",
      "type": "ref",
      "description": "Content hash for integrity verification"
    }
  },
  "description": "A single HTTP-accessible shard with integrity checksum"
}

Lexicon Garden

@