Gradient derivatives, Schemes and Mind Maps of Software Engineering

Download Gradient derivatives and more Schemes and Mind Maps Software Engineering in PDF only on Docsity!

Derivatives and Gradients

Jacobian, Hessian, Convex Set

& Function

K K Singh

• (^) Partial Derivatives. A partial derivative of a function with several variables is its derivative

with respect to one variables, while others held constant. Partial derivatives are used in

vector calculus and differential geometry

• A function z=f(x, y) has two derivatives: ∂z/∂x and ∂z/∂y.
• (^) These derivatives correspond to each of the independent variables and can be interpreted as instantaneous rates of change (that is, as slopes of a tangent line).
• Directional Derivatives: Let there is a surface defined by the equation
• (^) z=f(x, y). Given a point (a, b) in the domain of f,
• (^) Let we travel h distance in the direction of the unit vector u=(cosθ)i+(sinθ)j.
• (^) Then, the z-coordinate of the second point on the graph is given by z= f(a+hcosθ, b+hsinθ).
• (^) Then the directional derivative of f in the direction of u is given by
• (^) =
• (^) =
• Prove it (Apply L’ hospital)

**Examples

2.**

Gradient

• (^) Let z=f(x,y) be a function of x and y, such that their derivatives f x

and f

y

exist.

The vector ∇f(x,y) is called the gradient of f and is defined as

• (^) ∇ f(x,y) =f x

(x,y)i +f

y

(x,y) j.

• (^) The vector ∇f(x, y) is also written as “gradf.”
• (^) The derivative can be written as the dot product of the gradient and u :
• (^) D u

f(x,y)= ∇ f(x,y) u.⋅

• Examples

Example

• (^) The Jacobian matrix is a matrix composed of the first- order partial derivatives of a multivariable function.
• (^) Jacobian matrices will always have as many rows as vector components (f1, f2,.., fm), and the number of columns as match the number of variables (x1, x2, …, xn) of the function

Hessian matrix

• (^) The Hessian matrix , or simply Hessian , is an n×n square matrix composed of the second-order partial derivatives of a function of n variables.
• (^) the Hessian matrix is a symmetric matrix.
• (^) The Hessian matrix is the matrix with the second-order partial derivatives of a function. On the other hand, the matrix with the first-order partial derivatives of a function is the Jacobian matrix.
• (^) Example: Calculate the Hessian matrix at the point (1,0) of the following multivariable function:

Applications of the Hessian matrix

• Minimum, maximum, or saddle point
• (^) If the gradient of a function is zero at some point, that is f (x)=0,

then function f has a critical point at x. In this regard, we can determine whether

that critical point is a local minimum, a local maximum, or a saddle point using

the Hessian matrix:

• (^) If the Hessian matrix is positive definite (all the eigenvalues of the Hessian matrix

are positive), the critical point is a local minimum of the function.

• (^) If the Hessian matrix is negative definite (all the eigenvalues of the Hessian

matrix are negative), the critical point is a local maximum of the function.

• (^) If the Hessian matrix is indefinite (the Hessian matrix has positive and negative

eigenvalues), the critical point is a saddle point.

• Note that if an eigenvalue of the Hessian matrix is 0, we cannot know whether the

critical point is an extremum or a saddle point.

Convexity or concavity

• Another utility of the Hessian matrix is to know whether a function is concave or convex. And this can be determined by applying the following theorem.
• (^) Because Hessian matrix is square and symmetric symmetric (the original and the transpose are the same), they have a special property that their eigenvalues will always be real numbers
• (^) Let A Rn^ be an open set and f :A -> R a function whose second derivatives are continuous, its concavity or convexity is defined by the Hessian matrix:
• (^) Function f is convex on set A if, and only if, its Hessian matrix is positive semidefinite at all points on the set.
• (^) Function f is strictly convex on set A if, and only if, its Hessian matrix is positive definite at all points on the set.
• (^) Function f is concave on set A if, and only if, its Hessian matrix is negative semi-definite at all points on the set.
• (^) Function f is strictly concave on set A if, and only if, its Hessian matrix is negative definite at all points on the set.
• (^) The eigenvalues are real and positive values = Ax has the same direction as x.
• The eigenvalues are real and negative values = Ax has the opposite direction as x

Convex sets, convex Function

• (^) A convex set is a collection of points in which the line AB connecting any two points A, B in the set lies completely within the set.
• (^) In other words, A subset S of R n is considered to be convex if any linear (convex) combination θx1 + (1 − θ)x2, (0 ≤ θ ≤ 1) is also included in S for all pairs of x1, x2 ∈ S.
• (^) What is a Non-convex Set?
• (^) Non-convex sets are those that are not convex.
• (^) A non-convex polygon is occasionally referred to as a concave polygon, and also some sources use the phrase concave set to refer to a non-convex set.

Example 2: Prove that the set B = {(x 1 , x 2 , x 3 ) : 2x 1 − x 2

• x 3

≤ 4} ⊂ R

3 is a convex set. Solution: Assume that X = (x 1 , x 2 , x 3 ) and Y = (y 1 , y 2 , y 3 ) are the two points of B. From the given conditions, we can write 2x 1 − x 2

• x 3

2y 1 − y 2

• y 3

Now, assume that W = (w 1 , w 2 , w 3 ) is any point of [X, Y ] such that 0 ≤ θ ≤ 1, w 1 = θx 1 + (1 − θ)y 1 w 2 = θx 2 + (1 − θ)y 2 w 3 = θx 3 + (1 − θ)y 3 Using the above equations, we can write 2w 1 − w 2 + w 3 = θ(2x 1 − x 2 + x 3 ) + (1 − θ)(2y 1 − y 2 + y 3 ) ≤ 4θ + 4(1 − θ) = 4 Thus, W = (w 1 , w 2 , w 3 ) is a point of S.

Convex and concave function

• (^) Convex functions :Geometrically, a function on a subset of a vector space is

convex if the line segment joining any two points on its graph lies above the

graph.

• (^) Given points x and y in the domain, a typical point on the segment joining (x,

f(x)) and ( y, f(y)) is of the form ( (1 − α)x + αy,(1 − α)f(x) + αf(y)).