8.6.1: Scalability Patterns

Introduction to Cloud Computing · LibreTexts

0%

Contents

    Text

    Size
    100%
    Theme
    Typeface
    Line length
    Spacing

    Read aloud

    How

    Speed: 1×
    Light up
    Highlight

    Symbols: Mulberry Symbols, © 2018–2026 Steve Lee, CC BY-SA 4.0 (licence, the set), the site's own copies.

    From LibreTexts, under CC BY 4.0, as the source states it; its equations are written in TeX, which the PDF shows typeset. This page at LibreTexts.

    8.6.1: Scalability Patterns

    Scalability is the ability of the system to handle increased load by making use of additional resources. Cloud-native apps prefer horizontal scaling (adding more instances) over vertical scaling (using bigger machines), as horizontal scaling tends to be more elastic and cost-effective in cloud environments. Here are some patterns and considerations:

    • Stateless Services and Horizontal Scaling: As mentioned earlier, designing services to be stateless (not relying on in-memory sessions or local storage) is foundational. Stateless services can be replicated arbitrarily behind a load balancer. For example, a web service can run 10 instances on Kubernetes; the load balancer will distribute requests. Since no instance stores unique session data, any instance can serve any request. This yields near-linear scalability until other bottlenecks (like a database) are hit. Using external caches or databases for state (sessions, user data) avoids tying state to any one instance.
    • Auto-Scaling Policies: Cloud-native apps leverage auto-scaling on multiple levels. Infrastructure auto-scaling can add more VM instances to a cluster when CPU is high (e.g., AWS Auto Scaling Groups or Kubernetes Cluster Autoscaler). Application auto-scaling adds more service instances when needed (Kubernetes Deployments’ HPA, ECS Service Auto Scaling). There’s also the notion of scaling down to zero for cost savings – serverless platforms do this inherently (e.g., Cloud Run or Lambda will not consume resources when idle). A well-tuned auto-scaler uses the right metric signals; CPU is common, but in some cases better signals are request queue length or even custom app metrics. For example, if we have a message processor service reading from a queue, auto-scale based on queue depth (scale out if backlog growing, scale in when backlog is small).
    • Data Scalability (Sharding): The database or stateful part often becomes the bottleneck after scaling stateless tiers. Cloud-native strategy is to use databases that scale horizontally or shard data. NoSQL databases like Cassandra or DynamoDB scale by partitioning data across nodes. If relational DBs are needed, techniques like read replicas or sharding by key (e.g., customers by region) might be used. Another pattern is CQRS, splitting read-heavy and write-heavy workloads with different models (and using something like ElasticSearch or BigQuery for heavy reads, separate from the transactional store).
    • Caching: Caching is a classic scalability pattern and still vital in cloud-native design. Services might use in-memory caches (like Redis via AWS ElastiCache or GCP Memorystore) to store frequent results or session data, thereby reducing load on databases or expensive computations. At the edge, CDNs cache static content and even dynamic content with appropriate cache keys. For example, if a microservice generates a report that’s expensive, caching those results for a short time can handle bursts of requests. Netflix has caching everywhere – even device-level caches in their clients – to minimize calls to the backend.
    • Bulkheads and Workload Isolation: To scale different parts independently, sometimes systems are partitioned. A pattern called bulkhead involves isolating components so that a failure or slowdown in one doesn’t overwhelm others. For example, within a service you might use a separate thread pool for different categories of tasks (so that a slow external API doesn’t consume all threads and prevent the service from processing other requests). On a larger scale, you might split a service into two services if one function needs to scale differently than another.
    • Multi-Region Scaling: For truly massive scale and also for latency reduction, cloud-native apps might be deployed in multiple geographic regions. Traffic is routed via DNS or global load balancing to the nearest region. Each region can scale horizontally as needed. This is how global services (like Netflix, YouTube, etc.) handle millions of users – by not relying on one data center, but many, each scaling out. Multi-region adds complexity (especially keeping data in sync), but many cloud services (like DynamoDB global tables, or Spanner, or Kafka MirrorMaker) can replicate data globally.
    • Efficient Resource Utilization: Container orchestration helps pack workloads efficiently. In a microservices environment, at any given time some services might be busier than others. Kubernetes can bin-pack many small services on the same node up to its capacity. This flexibility means you can often achieve high utilization of machines (thus scaling cost-effectively). Contrast this to siloed monolithic stacks where each might have idle capacity for peak that is not shared.

    Real-world perspective on scalability: Netflix can quickly deploy thousands of instances on AWS when a surge occurs (e.g., a new season of a popular show is released). They use a lot of automation to predict and pre-scale for expected spikes, but also rely on the AWS auto-scaling to add capacity within minutes. Spotify moved to Google Cloud and uses Kubernetes; they built an internal Platform-as-a-Service on top of GKE that allows teams to easily scale their services by just changing a parameter. Spotify caches frequently accessed data in memory and uses asynchronous processing for heavy tasks (so user-facing parts remain responsive and can scale with user traffic, while background jobs scale separately).


    This page titled 8.6.1: Scalability Patterns was last modified on Tue, 22 Sep 2026 21:13:38 GMT and is shared under a CC BY 4.0 license and was authored, remixed, and/or curated by Felix Amoruwa.