Algorithms 2: A Quickstudy Laminated Reference Guide [1 ed.] 1423245636, 9781423245636

Table of contents :
Algorithms 2
ANALYSIS & OPTIMIZATION
ASYMPTOTIC NOTATION
COMPLEXITY TYPES
LOWER BOUND THEORY
ANALYSIS OF LOOPS
RECURRENCES
AMORTIZED ANALYSIS
SEARCH & SORT
PROCESS/IMPLEMENTATION TYPES

Citation preview

WORLD’S #1 ACADEMIC OUTLINE

ALGORITHMS 2 ANALYSIS & OPTIMIZATION algorithm analysis, we prove the context-free, and multiplicity; can be implemented Note: Execution speed or running time is a effectiveness, correctness, and completeness of an in any computer language or pseudocode and on any numerical measure (value) calculated via the algorithm by calculating its time and space metrics computer system. formula for time complexity. ▹ Space complexity: A.k.a. space efficiency; a (complexity), which are expressed as standardized ► Algorithm properties describe the effectiveness and complexity of an algorithm. measure of how much data space is needed by the notation via mathematical systems. algorithm to complete. ► Algorithms are measured by algorithm characteristics ▸ Algorithm effectiveness, correctness, and complete­ ness are realized (expressed) as a combination of Note: There are numerous overlapping terms as well as the specific algorithm’s properties. attributes that can be mathematically proven. across industries for algorithm complexity in both ► Algorithm characteristics (a.k.a. criteria) are ▹ Optimality: Yields the best solution(s). space and time contexts. interrelated and interdependent. ▹ Accuracy: Produces accurate results (solutions), ► Algorithm optimization is employed to: ▸ Input: One or more input values (or states) that sometimes within calculated error margins. ▸ Increase an algorithm’s effectiveness, correctness, may be of specific data types. ▹ Boundedness: Defines time and space bounds for and/or completeness ▸ Output: One or more output values (or states) that completion in the best, worst, and average cases. ▸ Decrease the time and/or space required for an may be of specific data types. algorithm ▸ Effectiveness: A.k.a. precision, definiteness, and ▸ Algorithm complexity, performance, and efficiency unambiguity; each statement is clear and basic are measured in terms of time and space, using ▸ Discover feasible solution spaces for algorithms enough to be carried out theoretically or on paper. standard notation and mathematical systems. and determine which solution is optimal ▸ Finiteness: Completes (terminates) in a finite ▹ Time complexity: A.k.a. time efficiency and ► Optimization methods include dynamic and linear number of steps. performance; a measure of how long the algorithm programming (see Process/Implementation ▸ Language-independent: A.k.a. language-agnostic, takes to complete. Types, pp. 5–6). ► Through

COMPLEXITY TYPES

ASYMPTOTIC NOTATION (BIG-O NOTATION) ► Algorithm analysis estimates algorithm

complexity in the asymptotic sense using big-O notation. Note: Asymptotic means approaching a value or closely fitting a curve, wherein an inherent limit is applied. ► Big-O is a mathematical notation invented by Donald Knuth that provides a measure of algorithm complexity, representing complexity as a function of input size. ► The notion is used to formulate (describe) time and space complexity for an algorithm, including bounds and growth rates. Note: The letter O stands for order of the function, meaning the growth rate. Functions (or algorithms) with the same growth rate are said to be of the same order. ► Algorithm complexity is defined as a numerical function in big-O notation whose input is the size (or number) of inputs to an algorithm and whose output is a bound on the execution time (or space). ▸ Using big-O notation, the time complexity function, T, is expressed as T(n) ϵ O(f(n)), where f is a specific numeric function. EX: T(n) ∊ O(log(n)); T(n) ∊ O(n2) ► This big-O complexity graph shows two time-complexity functions. The horizontal axis is the input size, and the vertical axis is the execution time. 2

n

10,000

analysis/notion to determine viability and feasibility of algorithms. Big-O omits less significant mathematical terms to achieve an upper bound. EX: An algorithm with the exact complexity of T(n) ϵ n2 × k, where n is the input size and k is a constant, would be considered T(n) ϵ O(n2), since the constant becomes insignificant as n grows larger than k. This is said to be “O of n-squared” or “O-n-squared.”

Big-Theta Θ(n)

► Big-Theta denotes an average-case

scenario for time or space complexity, most often expressed as a range. ► In the complexity example for big-O, n2 × k, different values for the constant k will yield different growth graphs, especially when k ≥ n. ▸ As shown in the big-Theta complexity graph, T(n) ∊ Θ(n2) means that the average execution time will remain between n2 × k1 and n2 × k2. This algorithm is said to be “theta of n-squared” or “theta-n-squared.” n2 × k2

40,000

20,000

n2

10,000

n2 × k1

0

5,000

2,500

3n + 4 20

40

60

80

100

Asymptotic Notation Types Big-O O(n)

► Big-O denotes the upper bound

or worst-case scenario for time or space complexity. EX: The upper bound on execution time for an algorithm with O(n2) time is the square of the input size.

(A.K.A. FUNCTION ORDERS & GROWTH RATES) The complexity types in this table apply to both time and space orders, although inefficient orders (e.g., cubic and beyond) are often infeasible in terms of space. There are several more specific time complexities used in computing, but they are all special cases and/or combinations of these time complexities. Complexity Name Running Time Description Constant time O(1) A constant number of steps that does not depend on input size EX: Determine odd or even for the input number. Logarithmic time O(log(n)) log(n) steps EX: Find an element in a sorted array with binary search. Linear time O(n) The same number of steps as the number of input elements EX: Find the largest number in an unsorted array. Linearithmic time O(n × log(n)) n × log(n) steps EX: Sort an array with merge sort. Quadratic time O(n2) n2 steps; a common polynomial complexity EX: Sort an array with bubble sort. Cubic time O(n3) n3 steps; a common polynomial complexity EX: Any triply nested loop Polynomial time O(nc) nc steps, where c > 1 EX: Find all the solutions to a multivariable equation. Factorial time O(n!) n! steps EX: Find all the permutations of a given string. Exponential time O(2n) 2n steps EX: Find all the subsets within a given set of elements.

The table below demonstrates the relative differences between the various complexity orders for a sampling of n values, where n is the number of elements in the input dataset. The algorithms of higher-order time complexity become infeasible and/or nonviable for large datasets. n

30,000

20

40

60

80

100

Big-Omega Ω(n)

► Big-Omega denotes the lower bound

7,500

0

► Big-O is the most commonly used

or best-case scenario for time or space complexity. It calculates the fastest possible execution speed or the least possible space usage. EX: An algorithm may have a worstcase scenario of T(n) ∊ O(n) and a best-case scenario of T(n) ∊ Ω(log(n)). This algorithm is said to be “omega of log-n” or “omega-log-n.” ► Big-Omega is used to define and verify lower bounds defined by big-Theta (optimization). ► The calculations inform eventual hardware and processing requirements once an algorithm is implemented and deployed. 1

1 10 100 1,000 10,000 100,000 1,000,000

O(2n) 2 1,024 1.27E + 30 1.07E + 301

O(n2) 1 100 10,000 1,000,000 100,000,000 10,000,000,000 1,000,000,000,000

O(n × log(n)) 0 10 200 3,000 40,000 500,000 6,000,000

This chart demonstrates the relative growth rates of a sampling of complexity orders over the dataset size.

Complexity Classes (P & NP)

► A Turing machine (TM) can abstractly run

30 25

Ο(n!)

O(n)

O(log(n))

1 10 100 1,000 10,000 100,000 1,000,000

0 1 2 3 4 5 6

Ο(2n ) Ο(n2)

Ο(n log (n)) O(n)

20 15

any algorithm. 10 ► A nondeterministic Turing machine 5 (NTM): Ο(log (n)) ▸ Is a variation of the TM (see the Ο(1) 0 Algorithms QuickStudy guide), wherein 5 10 15 20 for every state and symbol, there are a set of possible actions or values ▸ Employs a “choice” or “choose” function to select an action/value from the available set for each state transition ▹ The selection may be random, sequential, or any other linear function.

Complexity Types (continued )

ANALYSIS OF LOOPS

▸ May produce a different output and go through a different sequence of states in repeated executions, even given the same exact input ► Not all computational problems can be solved by an algorithm (e.g., Alan Turing proved that a general algorithm to solve the “halting problem” for all possible program-input pairs cannot exist). The complexity of computational problem spaces can be classified by whether a TM or NTM can exist to solve them, and then whether such a TM (or NTM) will run in deterministic time. This table describes the complexity classes in terms of TMs and NTMs, while the diagram shows the relationships of these classes in terms of sets of problems.

Complexity Classes

Loops are an essential part of almost all algorithms. ► To analyze loops, determine and separate constant-time statements/ functions from other statements/functions within the body of the algorithm that can take a variable amount of time. ► The complexity of loops is dependent on the iteration value and method (how the loop variable is manipulated in each iteration). This chart provides a starting guide to analyzing loop complexity. In the pseudocode examples, n represents data size, C is a constant greater than 1, and i and j are loop variables.



P Polynomial time A set of problems that can be solved by a deterministic Turing machine in polynomial time NP Nondeterministic polynomial time A set of problems that can be solved by a nondeterministic Turing machine in polynomial time NP-complete A set of problems A that satisfies two conditions: 1. A is of NP class (belongs to the NP set). 2. Any problem B in the NP set has a polynomial time reduction to A. The circular definition means that if one NP problem is solved, NP-hard so are all others. There are no known polynomial time algorithms for NP-complete NP-complete problems. NP-hard NP A set of problems that cannot be solved in polynomial time by any TM until P = NP There are no known polynomial time algorithms for P NP-hard problems.

Time Complexity

Complexity is constant when the algorithm does not contain a: • Loop • Recursion • Call to a nonconstant-time function Exception: A loop, function, or recursion that runs in constant time. The example to the right is O(1).

for (i=1; i 0; i=i-C) for (j=i; j