Menu
$begingroup$![Sequential Sequential](/uploads/1/2/5/6/125683470/581730074.jpg)
While Sequential programming involves a consecutive and ordered execution of processes one after another. 2 In other words with sequential programming, processes are run one after another in a succession fashion while in parallel computing, you have multiple processes execute at the same time. Electronic Data Processing. Electronic data processing or EDP is the modern technique to process data. The data is processed through a computer; Data and set of instructions are given to the computer as input, and the computer automatically processes the data according to the given set of instructions.
Let me clear first that I am not asking about parallel (data/task) or distributed computing architecture. A lot had been discussed here. I am just asking a plain theoretical question. What is core principal difference between Parallel and Distributed Algorithms? Below are my under standings:
In parallel algorithms (task parallelism), A big task is divided into two or more sub tasks and each sub task is executed by one processing element (PE) parallely. All sub tasks are depended on each other. What I further understand, a parallel algorithm can be coded using OpenMP which runs on multiprocessors as well as can be coded using MPI which runs on message passing systems. on the other hand, in distributed algorithm, every processing element or node executes the same copy of algorithm. They are totally independent and exchange messages to achieve the common goal. I think distributed algorithms are designed only for multi-computer systems.
It is requested to provide parallel and distributed algorithm for a same example problem for better understanding.
Community♦
user3606704user3606704
$endgroup$6 Answers
$begingroup$An algorithm is parallel if there are several processes (tasks, threads, processors) working on it at the same time. Often the tasks run in the same address space, and can communicate/reference results by others freely (low cost).
An algorithm is distributed if it is parallel and the tasks run on separate machines (separate address spaces), one task has no direct access to the work of the others. It has to request needed data, or just wait until it is sent to it.
Yes, it is a fuzzy distinction.
vonbrandvonbrand8,77233 gold badges3131 silver badges3939 bronze badges
$endgroup$$begingroup$One important quantitative distinction is that communication often costs more in distributed computing than in parallel computing.
An important qualitative distinction is that distributed algorithms often must deal with failure (e.g., one machine crashes, one machine starts misbehaving and sending spurious messages, or messages get lost or corrupted). In contrast, parallel algorithms typically assume there are no failures and do not have to deal with failures. Thus in the distributed algorithms literature we get algorithms for consensus, Byzantine fault tolerance, and more; those are issues that are specific to distributed systems and normally don't arise in parallel computing (or don't have much importance there).
D.W.♦D.W.104k1414 gold badges133133 silver badges305305 bronze badges
$endgroup$$begingroup$The terms can mean almost anything, but I will try to present here one way in which the terms 'parallel algorithms' and 'distributed algorithms' are understood. Here we interpret 'distributed algorithms' from the perspective of 'network computing' (think: algorithms that keep the Internet running).
I will use as a running example the problem of finding a proper 3-colouring of a directed path (linked list). I will first describe the problem from the perspective of 'traditional' algorithms — those are also known as centralised algorithms, to emphasise that they are not distributed, or sequential algorithms, to emphasise that they are not parallelised.
Centralised sequential algorithms
The model of computing is e.g. the familiar RAM model.
The input is a linked list that is stored in the main memory of the computer. There is a read-only array $x$ with $n$ elements; node number $x[i]$ is the successor of node number $i$.
The output will be also stored in the main memory of the computer. There is a write-only array $y$ with $n$ elements.
We need to find a proper colouring of the list with $3$ colours. That is, for each index $i$ we must choose a colour $y[i] in {1,2,3}$ such that $y[i] ne y[j]$ whenever node $j$ is the successor of node $i$.
There is a single processor that can directly access any part of the main memory. In one time unit, the processor can read from main memory, write to main memory, or perform elementary operations such as arithmetic or comparisons. The running time of the algorithm is defined to be the number of time units until the algorithms stops.
Clearly, the problem can be solved in time $O(n)$, and this is optimal. For the upper bound, follow the linked list and colour the nodes with e.g. colours $1,2,1,2,dotsc$. For the lower bound, observe that we need to write $Omega(n)$ elements of output.
Parallel algorithms
The only difference between parallel and sequential algorithms is that we will use the PRAM model instead of the RAM model. In the PRAM model we can consider any number of processors, but here a particularly interesting case is what happens if there are precisely $n$ processors.
While we will have multiple processors, there is still just one main memory. As before, the input is stored as a single array in the main memory, and the output will be written in a single array in the main memory.
Now in one time unit, each processor in parallel can read from main memory, write to main memory, or perform elementary operations such as arithmetic or comparisons. Some care is needed with memory accesses that may conflict. For the sake of concreteness, let us focus on the CREW PRAM model: the processors may freely read any part of the memory, but concurrent writes are forbidden.
Now in this setting it is not at all obvious what is the time complexity of $3$-colouring linked lists. Perhaps we could solve the problem in $O(1)$ time, as we have $n$ processors, and only $n$ units of input to read and $n$ units of output to write?
However, it turns out that the time complexity of this problem is precisely $Theta(log log^* n)$. So it can be solved in almost constant time, but not quite.
Distributed algorithms
Now things change radically. The model of computing is e.g. the LOCAL model, which has very little resemblance to RAM or PRAM.
There is no 'main memory'. There are no 'arrays'.
We are only given a computer network that consists of $n$ nodes. Each node is labelled with a unique identifier (say, a number from ${1,2,dotsc,n}$). Each node has two communication ports: one port that connects the node with its successor, and one port that connects it with its predecessor.
The same (unknown) computer network is both our input and the tool that we are supposed to use to solve the problem. Each node is a computational entity that has to output its own colour, and the colours have to form a proper colouring of the network (i.e., my colour has to be different from the colours of my neighbours).
Note that everything is distributed: no single entity holds the entire input, and no single entity needs to know the entire output.
All nodes run the same algorithm. In one time unit, all nodes in parallel can send messages to their neighbours, receive messages from their neighbours, or perform elementary operations. The running time of the algorithm is defined to be the number of time units until all nodes have stopped and produced their local outputs.
Again, it is not at all obvious what is the time complexity of $3$-colouring. It turns out that it is precisely $Theta(log^* n)$.
From this perspective:
- Research on parallel algorithms is primarily about understanding how to harness the computational power of a massively parallel computer. For practical applications, consider high-performance computing, number-crunching, multicore, GPU computing, OpenMP, MPI, grids, clouds, clusters, etc.
- Research on distributed algorithms is primarily about understanding which tasks can be solved efficiently in a distributed system. For practical applications, consider computer networks, communication networks, social networks, markets, biological systems, chemical systems, physical systems, etc.
For example:
- If you want to know how to multiply two huge matrices efficiently with modern computer hardware, it may be a good idea to first have a look at research related to 'parallel algorithms'.
- If you want to know if there is any hope people could form stable marriages in their real-world social network, by just exchanging information with those whom they know, it may be a good idea to first have a look at research related to 'distributed algorithms'.
Once again, I emphasise that this is just one way in which the terms are used. There are many other interpretations. However, this is perhaps the most interesting interpretation in the sense that e.g. PRAM and LOCAL are radically different models.
As many other answers show, another possible interpretation is to understand 'distributed algorithms' from the perspective of e.g. distributed high-performance computing (computer clusters, cloud computing, MPI, etc.). Then you could indeed say that distributed algorithms are not necessarily that different from e.g. I/O efficient parallel algorithms. At least if we put aside e.g. issues related to fault tolerance.
Incidentally, there is apparently some interest in the community to make the terminology slightly less confusing. People occasionally use the term distributed graph algorithms (cf. http://adga.hiit.fi/) or the term network computing to emphasise the perspective that I described here. However, there is not that much pressure to do that, as we can use formally precise terms such as 'LOCAL' and 'CONGEST' for distributed graph algorithms, 'PRAM' for parallel algorithms, and e.g. 'congested clique' and 'BSP' (bulk synchronous parallel) for various in-between cases.
References
- Cole & Vishkin (1986): http://dx.doi.org/10.1016/S0019-9958(86)80023-7
- Fich & Ramachandran (1990): http://dx.doi.org/10.1145/97444.97676
- Linial (1992): http://dx.doi.org/10.1137/0221015
![Sequential Sequential](/uploads/1/2/5/6/125683470/581730074.jpg)
34.1k77 gold badges9898 silver badges166166 bronze badges
Jukka SuomelaJukka Suomela
$endgroup$$begingroup$You should distinguish two concepts that are orthogonal:
- system architecture, which can be shared or distributed, meaning that the processing elements have direct access to common resources or on the opposite manage some local resources and can only access the resources of their fellows through them;
- programming model, which can be synchronous (SIMD) or asynchronous (MIMD), meaning that the processing elements execute the same instructions concurrently or independently of each other; one can further distinguish the case where the processing elements use the same code (SPMD) or different pieces (asymmetric processing).
Parallel is a general qualifier that can cover any form of multiprocessing. Distributed is more related to loosely coupled systems.
Multicore parallelism falls in the shared-memory MIMD category. GPGPU supports a mix of shared and distributed memory, operating in a kind of multi-SIMD mode. PC farms are clearly distributed MIMD.
Yves DaoustYves Daoust
$endgroup$$begingroup$There is no strict theoretical distinction between a parallel algorithm and a distributed one. The meaning of these terms is derived from their conventional use in computing.
A parallel algorithm is one that can be executed by a parallel computer. Here, parallelism refers only to the fact that the computer can execute multiple sequential programs simultaneously. (Strictly, even some sequential processors could be called parallel because they execute instructions simultaneously, but this is not the conventional usage of the term parallel computing.) The use of the term parallel algorithm also does not imply anything about the programming model or structure of the algorithm (i.e. whether it is shared memory, written in OpenMP or SIMD, MIMD etc.).
A distributed algorithm is one that can be executed by a distributed computer, which consists of multiple processors with separate memories. The processors are connected by a communication network allowing them to transfer data between themselves. Typically, distributed computing refers to supercomputers or grid processing systems, however, there are examples of distributed chip multi-processor architectures such as the MIT RAW and Kalray MPPA processors. As with parallel algorithms, the use of the term distributed algorithm also does not imply anything about the programming model or algorithm structure. Indeed, it is possible to implement shared memory with a distributed algorithm.
JamesJames
$endgroup$$begingroup$By no means distributed nodes are SIMD or homogenous.
IMO, distributed means that computers are separated is space. The further you separate computing nodes the more distributed your system is. Parallel computing is also distributed but it is not that obvious if it runs within single processor.
That is why you deal with node and transmission failures when regard distributed computing. If all your computation is parallel, it fail at once if your processor is down. It all goes down if something bad happens in that location. The distributed nodes can fail independly however and can experience communication failures when distributed over wide area in space. They can provide redundancy. You also distribute your redundant data to reduce the risk of loosing it.
Another answer also agrees with this point. It identifies multicomputers with distributed whereas says that multiprocessor is parallel. They both are parallel and distributed but there are more chances that you abandon the common memory at larger distances so you can speak about
distributed
here.This also accommodates the definition given in the most popular answer as soon as we understand that shared-mem threads are basically virtual multiprocessors whereas private-memory processes are virtual multicomputers. Here again, shared mem computing nodes are called 'parallel' computing nodes whereas more loosely coupled private mem coputing nodes are 'distributed'.
It seems that multiprocessor/multicomputer is a computer engeneering division whereas CS community prefer to speak in SW terms of threads and processes.
Distributed computation can even be sequential. Yes, client may make a blocking call to a web-server, which makes blocking calls to DB. There is not parallelism in this model but there is a distribution -- all components are located in different places.
Community♦
Valentin TihomirovValentin Tihomirov
$endgroup$Not the answer you're looking for? Browse other questions tagged algorithmsterminologydistributed-systemsparallel-computing or ask your own question.
Computer science is the integration of principles and applications of technologies that are required to provide access to information. This science revolves around studying the structure and expressions of algorithms, which are then used to solve complex problems. To excel at and understand computer science, it is important to understand industry terminology. By learning the basic terminology used in computer sciences, other, more complicated concepts will be easier to understand. Below is a list of popular computer science terminology.
Application: An application is a set of codes designed to allow specific tasks to happen. Microsoft Windows and Internet Explorer are common examples.
Application Server: Application Server is a specialized server that is based on client/server architecture. Its sole responsibility is to run specific applications within the network.
API: API refers to Application Programming Interface. It’s the platform used by a program to access different services on the computer system.
Array: An array is similar data saved on a computer system in a sequential form.
BIOS: BIOS stands for Basic Input/Output System. It gives the computer a platform to run the software using a floppy disk or a hard disk. BIOS is responsible for booting a PC.
Bit: Bit is Binary Digit. It refers to a digit number, either a 0 or a 1. The binary digit is used to represent computerized data.
Boolean: An expression, the value of which is either true or false.
Buffer: Buffer is a location, especially in RAM, for storage of temporary data to speed up an operation such as disk access or printing.
BUS: A bus is a set of wires that enables flow of data from one location of the computer to another.
Byte: Eight bits is equal to 1 byte.
CGI: CGI stands for Common Gateway Interface. It defines how an auxiliary program and a Web server would communicate.
Class: A group of objects having same operations and attributes is defined as a class.
Client: A client is a program that asks for information from other processes or programs. Outlook Express is a great example of a client.
CMOS: CMOS is an abbreviation for Complementary Metal-Oxide-Semiconductor. It is the battery powered chip that is situated on the Motherboard that retains system information such as date and time.
Data: Data refers to the information that is saved on a computer.
DOS: DOS is an acronym for Disc Operating System. It is a command line operating system launched by Bill Gates.
DTP: Desk Top Publisher (ing) is a term that describes a program that enables users to create, design, and print items such as business cards, birthday cards, letterheads, calendars, invitations, and so on.
Entity-relationship diagram: It’s a diagram that represents entities and how they are related to each other.
Environment: Environment refers to the interaction among all factors external to a physical platform. An environment is made of specific software, hardware, and network protocols that allow communication with the system.
Event Listener: It is an interface of JAVA responsible for handling events.
FAT: FAT is an acronym for File Allocation Table. It resembles a table of contents so that files can be located on a computer.
Fault: Hardware or software failure.
Front End: It is an interface through which a program can be accessed by common users.
Hardware: Hardware is a set of physical objects such as monitor, keyboard, mouse, and so on.
Icon: Icon is a small visual display of an application which can be activated by clicking on it.
IDE: It stands for Integrated Development Environment. IDE is a programming system that combines several tools of programming to provide an integrated platform for programming. For instance, Visual Basic provides an IDE.
Instance: It is an object described by its class.
Internet: Internet is a network that accommodates several computers to facilitate exchange and transfer of data.
Kernel: It is a program called when a computer system is started. Kernel is responsible for setting up system calls in order to manage hardware and system services, and allocate resources to applications.
LAN: LAN is an acronym for Local Area Network that spans small area. A LAN can be connected to another LAN to accommodate more computers.
Memory: Memory is the internal storage location where data and information is stored on a computer.
Modem: Modem is a term created from the beginning letters of two other words viz. MOdulation and DEModulation. The term implies changing of data from digital to analog and then back to digital.
Network: A Network is a group of computers connected to each other in order to send and receive data.
Operating System: An Operating System provides the software platform required for various applications to run on. Its responsibility is to manage memory storage and security of Data.
Packet: Sections in which message or data are divided to transfer it over a network.
Pixel: Pixel is formed by combining the two words viz. Picture Element. It represents one point within an image.
Port: Port is a connecting component mainly a hardware that enables two computers to allow data sharing physically. Examples are USB and HDMI.
Process: It’s a series of commands that changes data values.
Protocol: Protocol refers to a set of rules that are followed by two devices while interacting with each other.
Query: Query is a request made by a computer from a database residing in the same system or a remotely located system.
RAM: RAM is an acronym for Random Access Memory. It is a configuration of storage cells that hold data so that it can be processed by the central processing unit. RAM is a temporary storage location.
ROM: ROM is an acronym for Read-Only Memory. It is semiconductor-based storage system that saves information permanently.
Software: Software is a program (coding) that the computer reads. The system then carries out functions as directed by the code. Adobe Photoshop is software.
TCP/IP: TCP/IP is an acronym for Transmission Control Protocol/Internet Protocol. It’s a set of communication protocols used to connect host computers on the Internet.
URL: URL stands for Universal Resource Locator. It’s a way of accessing the Internet.
Virtual Memory: Virtual Memory is the unused memory on the hard disk used when certain applications require more RAM than is available on the machine.
Virus: Virus is a program that is loaded onto your computer without you knowing about it and it runs to hinder the normal functioning of the computer.
WWW: WWW stands for World Wide Web. It’s a term used to define the Internet.
WAN: WAN is an acronym for Wide Area Network. Such a network spans over an area larger than a LAN.
ZIP: ZIP is an acronym for Zone Information Protocol. ZIP application enables transfer of data using compression of files.
For additional information, please refer to the following sites: