Entrepreneurship and Product Development

BSD 2208 Cloud Application Development.

Cloud Computing vs Parallel Computing.

Traditionally software has been written for serial computations: To be run on a single computer having a single Central Processing Unit (CPU) A problem is broken into a discrete set of instructions Instructions are executed one after another Only one instruction can be executed at any moment in time problem.

Parallel Computing: In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem: To be run using multiple CPUs A problem is broken into discrete parts that can be solved concurrently Each part is further broken down to a series of instructions Instructions from each part execute simultaneously on different CPUs problem instructions.

Parallel Computers: Virtually all stand-alone computers today are parallel from a hardware perspective: Multiple functional units (floating point, integer, GPU, etc.) Multiple execution units / cores Multiple hardware threads Memory Controller Intel Core i7 CPU and its major components Shared L3 Cache : 1; i 14.

ParallelComputers: Networks connect multiple stand-alone computers (nodes) to create larger parallel computer clusters Each compute node is a multi-processor parallel computer in itself Multiple compute nodes are networked together with an InfiniBand network Special purpose nodes, also multi-processor, are used for other purposes NODE— memory core core core core OD NODE— memory OD memory NODe memory core core core core core core core core core core core core core core core core NOD memory core core core core.

Why Use HPC? Major reasons : .47 .4'„ ACCESSGRID Save time and/or rnoney: In theory, throwing more resources at a task will shorten its time to completion, with potential cost savings. Parallel clusters can be built from cheap, commodity components. Solve larger problems: Many problems are so large and/or complex that it is impractical or impossible to solve them on a single computer, especially given limited computer memory. Provide concurrency: A single compute resource can only do one thing at a time. Multiple computing resources can be doing many things simultaneously. Use of non-local resources: Using compute resources on a wide area network, or even the Internet when local compute resou rces are scarce..

HPC TERMINOLOGY.

Parallel Computer Memory Architectures: CPU CPU Memory CPU CPU Shared Memory: Multiple processors can operate independently, but share the same memory resources Changes in a memory location caused by one CPU are visible to all processors.

Parallel Computer Memory Architectures: CPU Advantages: cpu Memory CPU CPU Shared Memory: Multiple processors can operate independently, but share the sarne memory resources Changes in a memory location caused by one CPU are visible to all processors Global address space provides a user-friendly programming perspective to memory Fast and uniform data sharing due to proximity of memory to CPUs Disadvantages: Lack of scalability between memory and CPUs. Adding more CPUs increases traffic on the shared memory-CPU path Programmer responsibility for "correct" access to global memory.

Parallel Computer Memory Architectures: Distributed Memory: CPU CPU Memory Memory CPU CPU Memory Memory Requires a communication network to connect inter-processor memory Processors have their own local memory. Changes made by one CPU have no effect on others Requires communication to exchange data among processors.

Parallel Computer Memory Architectures: CPU cpu Memory Memory CPU CPU Mentory Memory Distributed Memory: Requires a communication network to connect inter-processor memory Processors have their own local memory. Changes made by one CPU have no effect on others Requires communication to exchange data among processors Advantages: Memory is scalable with the number of CPUs Each CPU can rapidly access its own memory without overhead incurred with trying to maintain global cache coherency Disadvantages: Programmer is responsible for many of the details associated with data communication between processors It is usually difficult to map existing data structures to this memory organization, based on global memory.

Parallel Computer Memory Architectures: Hybrid Distributed-Shared Mernory: The largest and fastest computers in the world today employ both shared and distributed memory architectures. Memory CPU Memory pu CPU Memory pu CPU PU CPU Memory PU CPU Memory CPU Memory CPU Memory CPU GPU CPU Memory CPU Shared memory component can be a shared memory machine and/or GPU Processors on a compute node share same memory space Requires communication to exchange data between compute nodes.

Parallel Computer Memory Architectures: Hybrid Distributed-Shared Mernory: The largest and fastest computers in the world today employ both shared and distributed memory architectures. Memory PU CPU Memory CPU PU CPU Memory PU CPU Memory CPU CPU Memory CPU Memory CPU Memory CPU Memory CPU Shared memory component can be a shared memory machine and/or GPU Processors on a compute node share same memory space Requires communication to exchange data between compute nodes Advantages and Disadvantages: Whatever is common to both shared and distributed memory architectures Increased scalability is an important advantage Increased programming complexity is a major disadvantage.

Parallel Programming Models: Parallel Programming Models exist as an abstraction above hardware and memory architectures Shared Memory (without threads) Shared Threads Models (Pthreads, OpenMP) Distributed Memory / Message Passing (MPI) Data Parallel Hybrid Single Program Multiple Data (SPMD) Multiple Program Multiple Data (MPMD).

END.