Parallel Computing
Introduction
Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time. There are several different forms of parallel computing: bit-level, instruction-level, data, and task parallelism. Parallelism has long been employed in high-performance computing, but it's gaining broader interest due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multi-core processors.
Parallel Computing Models
Parallel computing models can be broadly classified into data parallel and task parallel models.
Data Parallelism
Data parallelism emphasizes the distributed (parallel) nature of the data, as opposed to the processing (task parallelism). Most real programs fall somewhere on a continuum between task parallelism and data parallelism.
Task Parallelism
Task parallelism is the simultaneous execution of the same task (split up and specially adapted) on multiple processors in order to obtain results faster. This is the parallel computing form by which most vector- and array-processors work.
Architecture
Parallel computers can be roughly classified according to the level at which the hardware supports parallelism, with multi-core and multi-processor computers having multiple processing elements within a single machine, while clusters, MPPs, and grids use multiple computers to work on the same task.
Shared memory systems may be used in multi-processor systems. In this case, the system uses multiple processors to share a common memory.
Distributed Memory Architecture
In distributed memory systems, each processor has its own memory. A processor can directly address its own local memory and remote memory (memory local to another processor). It may have to have explicit software support, depending on the interconnect between processors of the system.
Programming Models
Parallel programming models exist as an abstraction above hardware and memory architectures. They are generally not associated with a particular type of machine, but rather they take advantage of certain types of patterns that can be applied to many different types of machines. The types of models include: shared memory model, thread model, message passing model, data parallel model, hybrid model, Flynn's taxonomy, and others.
In the shared memory programming model, tasks share a common address space, which they read and write to asynchronously.
Message Passing Model
The message passing model is typically used on distributed memory systems. These systems require a communication network to connect inter-processor memory.
Data Parallel Model
The data parallel model is a parallel design pattern. Data parallelism focuses on distributing the data across different parallel computing nodes.
Algorithms
Parallel algorithms are a subfield of computer science and are used in parallel computing. Parallel algorithms are designed to improve computation speed, where large problems are divided into smaller ones, which are then solved concurrently. There are many types of parallel algorithms, including parallel sorting algorithms, parallel searching algorithms, and parallel graph algorithms.
Applications
Parallel computing is used in a wide range of applications, including computational fluid dynamics, data mining, bioinformatics, quantum mechanics, and others. These applications require the processing of large amounts of data and complex calculations, which are well-suited for parallel computing.
See Also
