Computer Β· Part E - Embedded, Distributed and Specialized Computing Β· Major topic 26 of 39

Distributed Systems, Parallel Computing, Cloud, Edge and HPC

Coordinating computation across many processors or machines, including failure, consistency, scale, locality, cost and latency.

198 items insideAsked 1Γ— in the RPSC/RSSB papers we analysed, most recently 2013COM-26

Also look at: Science & Technology: Cloud Computing, Big Data, Supercomputing, IoT and Smart SystemsScience & Technology: Emerging Technologies β€” Concepts for ExamsPhysics: Computational Physics, Modelling & Scientific Computing

  1. 1

    Historical evolution β€” key dates

    • Time-sharing, multiprocessors and early distributed operating-system ideas develop.1960s-1970s
    • Packet networks and remote procedure calls enable geographically separated computation.1970s-1980s
    • Cluster computing and grid computing support larger workloads.1990s
    • Large-scale web services and commercial cloud infrastructure expand.2000s
    • Containers, orchestration, serverless computing and managed data services proliferate.2010s
    • Edge/cloud continuum, AI accelerators, confidential computing and distributed AI workloads expand.2020s
  2. 2

    Parallel computing fundamentals

    • Concurrency vs parallelism; task/data parallelism; shared/distributed memory; SIMD/SIMT; work/span
      5 more points
      • Concurrency vs parallelism
      • Task/data parallelism
      • Shared/distributed memory
      • SIMD/SIMT
      • Work/span
    • Speedup; scalability; Amdahl’s law; Gustafson’s law; load balance; granularity; communication overhead
      7 more points
      • Speedup
      • Scalability
      • Amdahl’s law
      • Gustafson’s law
      • Load balance
      • Granularity
      • Communication overhead
  3. 3

    Synchronization and parallel patterns

    • Threads; locks; atomics; barriers; fork-join; map/reduce; pipeline; task graph; work stealing
      9 more points
      • Threads
      • Locks
      • Atomics
      • Barriers
      • Fork-join
      • Map/reduce
      • Pipeline
      • Task graph
      • Work stealing
    • Race conditions; false sharing; memory consistency; deadlock; nondeterminism; deterministic replay
      6 more points
      • Race conditions
      • False sharing
      • Memory consistency
      • Deadlock
      • Nondeterminism
      • Deterministic replay
  4. 4

    Distributed-system model

    • Nodes; messages; partial failure; network partition; clock drift; failure detector; retry; timeout
      8 more points
      • Nodes
      • Messages
      • Partial failure
      • Network partition
      • Clock drift
      • Failure detector
      • Retry
      • Timeout
    • Idempotency; duplicate delivery; at-most/at-least/exactly-once claims; backpressure; flow control
      5 more points
      • Idempotency
      • Duplicate delivery
      • At-most/at-least/exactly-once claims
      • Backpressure
      • Flow control
  5. 5

    Consistency and coordination

    Asked 1Γ— Β· last 2013

    • Linearizability; sequential/eventual consistency; causal consistency; read-your-writes; quorum
      5 more points
      • Linearizability
      • Sequential/eventual consistency
      • Causal consistency
      • Read-your-writes
      • Quorum
    • Leader election; consensus; replication; partitioning; membership; distributed lock; lease; fencing token
      8 more points
      • Leader election
      • Consensus
      • Replication
      • Partitioning
      • Membership
      • Distributed lock
      • Lease
      • Fencing token
    • CAP and PACELC models; split-brain; clock assumptions; CRDT concepts; conflict-free merging
      5 more points
      • CAP and PACELC models
      • Split-brain
      • Clock assumptions
      • CRDT concepts
      • Conflict-free merging
  6. 6

    Distributed architecture patterns

    • Client/server; service discovery; API gateway; message broker; event-driven system; saga; outbox pattern
      7 more points
      • Client/server
      • Service discovery
      • API gateway
      • Message broker
      • Event-driven system
      • Saga
      • Outbox pattern
    • CQRS; event sourcing; stream processing; microservices; serverless; batch orchestration; workflow engine
      7 more points
      • CQRS
      • Event sourcing
      • Stream processing
      • Microservices
      • Serverless
      • Batch orchestration
      • Workflow engine
  7. 7

    Cloud computing

    • IaaS/PaaS/SaaS; public/private/hybrid cloud; region/zone; VPC/VNet; IAM; object/block/file storage
      6 more points
      • IaaS/PaaS/SaaS
      • Public/private/hybrid cloud
      • Region/zone
      • VPC/VNet
      • IAM
      • Object/block/file storage
    • Compute instances; managed databases; load balancing; autoscaling; serverless functions; managed Kubernetes
      6 more points
      • Compute instances
      • Managed databases
      • Load balancing
      • Autoscaling
      • Serverless functions
      • Managed Kubernetes
    • Availability zones; quotas; cost estimation; shared responsibility; landing zone; governance guardrails
      6 more points
      • Availability zones
      • Quotas
      • Cost estimation
      • Shared responsibility
      • Landing zone
      • Governance guardrails
  8. 8

    Containers and orchestration

    • Image/layer/registry; build; runtime; namespace; cgroup; volume; network; container security
      8 more points
      • Image/layer/registry
      • Build
      • Runtime
      • Namespace
      • Cgroup
      • Volume
      • Network
      • Container security
    • Pods; deployments; stateful sets; services; ingress; config maps; secrets; probes; operators; helm concepts
      10 more points
      • Pods
      • Deployments
      • Stateful sets
      • Services
      • Ingress
      • Config maps
      • Secrets
      • Probes
      • Operators
      • Helm concepts
  9. 9

    Edge and fog computing

    • Data locality; low-latency inference; offline operation; synchronization; model delivery; fleet management
      6 more points
      • Data locality
      • Low-latency inference
      • Offline operation
      • Synchronization
      • Model delivery
      • Fleet management
    • Edge gateway; near-device processing; privacy/local retention; constrained compute; intermittent connectivity
      5 more points
      • Edge gateway
      • Near-device processing
      • Privacy/local retention
      • Constrained compute
      • Intermittent connectivity
  10. 10

    High-performance computing (HPC)

    • Cluster; scheduler; job queue; batch job; MPI; OpenMP; CUDA/GPU computing; vectorization
      8 more points
      • Cluster
      • Scheduler
      • Job queue
      • Batch job
      • MPI
      • OpenMP
      • CUDA/GPU computing
      • Vectorization
    • Interconnect; distributed filesystem; checkpoint/restart; parallel I/O; profiling; scaling study; energy efficiency
      7 more points
      • Interconnect
      • Distributed filesystem
      • Checkpoint/restart
      • Parallel I/O
      • Profiling
      • Scaling study
      • Energy efficiency
  11. 11

    Reliability, cost and operations

    • SLI/SLO; redundancy; fault domains; graceful degradation; chaos engineering; disaster recovery
      6 more points
      • SLI/SLO
      • Redundancy
      • Fault domains
      • Graceful degradation
      • Chaos engineering
      • Disaster recovery
    • Observability; capacity planning; cloud spend; cost allocation; workload rightsizing; data transfer costs
      6 more points
      • Observability
      • Capacity planning
      • Cloud spend
      • Cost allocation
      • Workload rightsizing
      • Data transfer costs
  12. 12

    Advanced research

    • Distributed transactions; geo-replication; consensus protocols; Byzantine fault tolerance; confidential computing
      5 more points
      • Distributed transactions
      • Geo-replication
      • Consensus protocols
      • Byzantine fault tolerance
      • Confidential computing
    • Serverless cold starts; disaggregated memory; composable infrastructure; edge AI; carbon- aware placement
      5 more points
      • Serverless cold starts
      • Disaggregated memory
      • Composable infrastructure
      • Edge AI
      • Carbon- aware placement