Small-File Object Storage Economics: Why Millions of 4KB Objects Inflate Class A PUT and LIST Costs

The Small-Object Cost Amplification Paradox (1 TB in 4 KB vs. 64 MB Files)

Consider two applications that both ingest and read exactly **1.0 TB (1,000,000,000,000 bytes)** of data per month on AWS S3 Standard (`$0.023/GB` storage, `$5.00/1M` Class A `PUT`, `$0.40/1M` Class B `GET`):

Complete S3 & R2 API Operation Classification Matrix

Understanding which S3 REST API actions trigger Class A versus Class B billing is essential before deploying backup agents (`rclone`, `restic`, `velero`) or data lake compaction jobs:

The `ListObjectsV2` Pagination Multiplier: The S3 `ListObjectsV2` API returns a maximum of **1,000 keys per response page**. Listing a bucket containing 100,000,000 small objects requires **100,000 sequential Class A `LIST` calls** ($0.50 per full scan). If a misconfigured cron job or Kubernetes operator scans that bucket every 5 minutes, that single `LIST` loop costs **$4,320/month**. Use **AWS S3 Inventory** (which outputs a daily CSV/Parquet manifest of all keys for $0.0025 per million objects) instead of running live `ListObjectsV2` scans on massive buckets.

Multipart Upload Chunk Sizing & Orphan Part Cleanup

When uploading large files via S3 Multipart Upload, every individual `UploadPart` call—plus the initial `CreateMultipartUpload` and final `CompleteMultipartUpload`—is billed as a separate Class A operation:

• Avoid 5 MB Minimum Part Sizes on Huge Files: If you upload a 50 GB database dump using the minimum 5 MB part size, your client executes **10,002 Class A requests** for a single file. Increasing `--s3-chunk-size` to **64 MB or 128 MB** reduces the upload to just 400 Class A calls while saturating 10 Gbps network links.
• Orphaned Incomplete Multipart Uploads: If a client crashes mid-upload without calling `AbortMultipartUpload`, the uploaded parts remain invisible to standard `ListObjectsV2` queries while **billing you indefinitely for standard storage every month**. Always enforce an `AbortIncompleteMultipartUpload` lifecycle policy on every production bucket:

{
  "Rules": [
    {
      "ID": "Purge-Orphaned-Multipart-Uploads-After-7-Days",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

Architectural Pattern: Indexed Shard Packing & HTTP Range Reads

To eliminate small-file Class A/B costs without losing random-access read capability, pack 10,000 small blobs into a single 64 MB uncompressed `.tar` or binary packfile accompanied by a lightweight byte-offset index:

1. Ingestion Buffering: Buffer incoming JSON events in memory (or via Vector / Kinesis Firehose / Kafka Connect) for 60 seconds or until reaching 64 MB before executing a single `PutObject` call.
2. Random Access via Byte-Range `GET`: When an application needs a single 8 KB sub-item inside a 100 MB packfile, issue a standard `GetObject` request with the HTTP header `Range: bytes=4194304-4202495`. Both S3 and Cloudflare R2 return `HTTP 206 Partial Content` containing only the requested 8 KB slice, billing just 1 Class B operation and 8 KB of transfer.

import boto3

s3 = boto3.client("s3", endpoint_url="https://<ACCOUNT_ID>.r2.cloudflarestorage.com")

def fetch_packed_record(bucket: str, shard_key: str, offset: int, length: int) -> bytes:
    # S3 HTTP Range header is inclusive on both start and end byte offsets
    byte_range = f"bytes={offset}-{offset + length - 1}"
    resp = s3.get_object(Bucket=bucket, Key=shard_key, Range=byte_range)
    return resp["Body"].read()

# Reads exactly 4,096 bytes from a 64 MB shard in ~18ms (1 Class B GET)
payload = fetch_packed_record("telemetry-shards", "2026-07-03/pack-0042.bin", 4194304, 4096)