11.2.1: Object Storage

Object storage manages data as discrete units called objects in a flat, scalable address space. Unlike file systems, there are no folders or hierarchical paths – each object is identified by a unique key (ID or name) and stored in a single-level “bucket” or container. Each object consists of the data itself (binary content of a file) plus associated metadata. Metadata in object storage is customizable and stored with the object, enabling rich descriptions (tags, attributes) that can be used for indexing and search. Access to objects is typically via an API over HTTP(S), rather than through a traditional file system interface. For example, to retrieve a photo stored in Amazon S3 or Google Cloud Storage, an application uses a REST API call with the object’s key rather than opening a file path.

Object storage is highly scalable and durable. Cloud object stores like Amazon S3, Google Cloud Storage, and Azure Blob Storage can scale essentially without limit – handling millions or billions of objects – and automatically replicate data across multiple devices or even multiple facilities for durability. In fact, object storage has largely displaced tape libraries for backups and archives due to its scalability and accessibility. Providers often advertise “eleven nines” (99.999999999%) durability for object storage, meaning data is redundantly stored so that loss is extremely unlikely. The trade-off is that object storage may have higher latency for retrieval of individual items compared to block or file storage, because data is accessed via network calls and there’s no in-place modification (updating an object requires writing a new version). This makes object storage less ideal for transactional workloads that require frequent, small updates (e.g. a database system). Instead, object storage excels at storing large amounts of unstructured data – such as media files, backups, logs, and big data – where each object is often large and accessed in whole.

Use cases: Due to its scalability and low cost, object storage is used for data lakes (central repositories of raw data), multimedia repositories, archives, and as the backing store for content delivery networks. For instance, modern enterprises build data lakes on systems like Amazon S3 – ingesting raw datasets (clickstreams, IoT sensor data, images, etc.) into a single massive object store. This decouples storage from compute, allowing various analytics engines or AI/ML tools to run on the data as needed. Object storage also supports versioning and retention features (e.g., S3 Object Versioning) which are valuable for backup and compliance uses, like preserving multiple versions of files or ensuring write-once-read-many (WORM) protection for archival records. In practice, companies like Netflix rely on object storage to hold vast media libraries and user data backups. In Netflix’s case, “the primary backups of all data are generated into S3,” so that any corruption or human error can be mitigated by restoring from the S3 copy. This demonstrates object storage’s role as a resilient, central repository in a cloud architecture.

---