FinDi package

The findi contains public functions that a user will use to for optimization. They wrap functions from _numba_findi and _python_findi modules allowing increased performance and flexibility. The functions optimize the objective via Gradient Descent Algorithm variation that uses finite difference instead of infinitesimal differential for computing derivatives. This approach allows for the application of Gradient Descent on non-differentiable functions, functions without analytic form or any other function, as long as it can be evaluated. descent function performs regular finite difference gradient descent algorithm, while partial_descent function allow a version of finite difference gradient descent algorithm where only a random subset of gradients is used in each epoch. partially_partial_descent function performs partial_descent algorithm for the first partial_epochs number of epochs and descent for the rest of the epochs.Parallel computing for performance benefits is supported in all of these functions. If numba=True parallelization is done by Numba, otherwise it is done by joblib library. Objective functions with multiple outputs are supported (only the first one is taken as objective value to be minimized), as well as objective function metaparameters that are held constant throughout the epochs. Furthermore, values_out function is included for compactly exporting values of outputs, parameters and metaparameters for each epoch.

findi.descent(objective, initial, h, l, epochs, metaparameters=None, momentum=0, threads=1, numba=False)

Performs Gradient Descent Algorithm by using finite difference instead of infinitesimal differential. Allows for the implementation of gradient descent algorithm on variety of non-standard functions.

Parameters:

• objective (callable) – Objective function to be minimized. If numba=True, objective has to be a Numba function

• initial (int, float, list, nb.typed.List or ndarray) – Initial values of objective function parameters

• h (int, float, list, nb.typed.List or ndarray) – Small change(s) in x. Can be a sequence or a number in which case constant change is used

• l (int, float, list, nb.typed.List or ndarray) – Learning rate(s). Can be a sequence or a number in which case constant learning rate is used

• epochs (int) – Number of epochs

• metaparameters (list, nb.typed.List or ndarray, optional) – Metaparameter values used for the evaluation or tuning of the objective function. These aren’t adjusted in the process of gradient descent, defaults to None

• momentum (int or float, optional) – Hyperparameter that dampens oscillations. momentum=0 implies vanilla algorithm, defaults to 0

• threads (int, optional) – Number of CPU threads used by joblib for computation. Argument only used when numba=False, as in the other case Numba takes case of parallelization, defaults to 1

• numba (bool, optional) – Whether to use Numba’s just-in-time compiler for performance improvements. If numba=True the function provided in argument objective has to be a Numba function (i.e. it has to be decorated with one of the relevant Numba decorators such as @numba.njit). For more information refer to Numba documentation, defaults to False

Returns:

Objective function outputs and parameters for each epoch

Return type:

ndarray, ndarray

findi.partial_descent(objective, initial, h, l, epochs, parameters_used, metaparameters=None, momentum=0, threads=1, rng_seed=88, numba=False)

Performs Gradient Descent Algorithm by computing derivatives on only specified number of randomly selected parameters in each epoch and by using finite difference instead of infinitesimal differential. Allows for the implementation of gradient descent algorithm on variety of non-standard functions.

Parameters:

• objective (callable) – Objective function to be minimized. If numba=True, objective has to be a Numba function

• initial (int, float, list, nb.typed.List or ndarray) – Initial values of objective function parameters

• h (int, float, list, nb.typed.List or ndarray) – Small change(s) in x. Can be a sequence or a number in which case constant change is used

• l (int, float, list, nb.typed.List or ndarray) – Learning rate(s). Can be a sequence or a number in which case constant learning rate is used

• epochs (int) – Number of epochs

• parameters_used (int) – Number of parameters used in each epoch for computation of gradients

• metaparameters (list, nb.typed.List or ndarray, optional) – Metaparameter values used for the evaluation or tuning of the objective function. These aren’t adjusted in the process of gradient descent, defaults to None

• momentum (int or float, optional) – Hyperparameter that dampens oscillations. momentum=0 implies vanilla algorithm, defaults to 0

• threads (int, optional) – Number of CPU threads used by joblib for computation. Argument only used when numba=False, as in the other case Numba takes case of parallelization, defaults to 1

• rng_seed (int, optional) – Seed for the random number generator used for determining which parameters are used in each epoch for computation of gradients, defaults to 88

• numba (bool, optional) – Whether to use Numba’s just-in-time compiler for performance improvements. If numba=True the function provided in argument objective has to be a Numba function (i.e. it has to be decorated with one of the relevant Numba decorators such as @numba.njit). For more information refer to Numba documentation, defaults to False

Returns:

Objective function outputs and parameters for each epoch

Return type:

ndarray, ndarray

findi.partially_partial_descent(objective, initial, h, l, partial_epochs, total_epochs, parameters_used, metaparameters=None, momentum=0, threads=1, rng_seed=88, numba=False)

Performs Partial Gradient Descent Algorithm for the first partial_epochs epochs and regular Finite Difference Gradient Descent for the rest of the epochs (i.e. total_epochs-partial_epochs).

Parameters:

• objective (callable) – Objective function to be minimized. If numba=True, objective has to be a Numba function

• initial (int, float, list, nb.typed.List or ndarray) – Initial values of objective function parameters

• h (int, float, list, nb.typed.List or ndarray) – Small change(s) in x. Can be a sequence or a number in which case constant change is used

• l (int, float, list, nb.typed.List or ndarray) – Learning rate(s). Can be a sequence or a number in which case constant learning rate is used

• partial_epochs (int) – Number of epochs for Partial Gradient Descent

• total_epochs (int) – Total number of epochs including both for partial and regular algorithms. Implies that the number of epochs for the regular algorithm is given as total_epochs-partial_epochs

• parameters_used (int) – Number of parameters used in each epoch for computation of gradients

• metaparameters (list, nb.typed.List or ndarray, optional) – Metaparameter values used for the evaluation or tuning of the objective function. These aren’t adjusted in the process of gradient descent, defaults to None

• momentum (int or float, optional) – Hyperparameter that dampens oscillations. momentum=0 implies vanilla algorithm, defaults to 0

• threads (int, optional) – Number of CPU threads used by joblib for computation. Argument only used when numba=False, as in the other case Numba takes case of parallelization, defaults to 1

• rng_seed (int, optional) – Seed for the random number generator used for determining which parameters are used in each epoch for computation of gradients, defaults to 88

• numba (bool, optional) – Whether to use Numba’s just-in-time compiler for performance improvements. If numba=True the function provided in argument objective has to be a Numba function (i.e. it has to be decorated with one of the relevant Numba decorators such as @numba.njit). For more information refer to Numba documentation, defaults to False

Returns:

Objective function outputs and parameters for each epoch

Return type:

ndarray, ndarray

findi.values_out(outputs, parameters, metaparameters=None, columns=None)

Produces a Pandas DataFrame of objective function outputs, parameter values and metaparameter values for each epoch of the algorithm.

Parameters:

• outputs (list or ndarray) – Objective function outputs throughout epochs

• parameters (list or ndarray) – Objective function parameter values throughout epochs

• metaparameters (list, nb.typed.List or ndarray, optional) – Metaparameter values used for the evaluation or tuning of the objective function. These aren’t adjusted in the process of gradient descent, defaults to None

• columns (list or ndarray, optional) – Column names of outputs and parameters, defaults to None

Returns:

Dataframe of all the values of inputs and outputs of the objective function for each epoch

Return type:

pd.DataFrame