BFGS user should be able to pass parameters to the line search

[axch]

This should just be a matter of accepting a dict named line_search_kwargs or such and passing it through to the actual line search.

This was a feature request that came out of discussing #1034.

[axch]

Clarification: The purpose is for the user to be able to control the parameters of the actual line search algorithm, of which only linesearch.hager_zhang is currently implemented.

[NeelGhoshal]

Hello @axch, is this issue fixed? If not, I would like to work on it as my first open source contribution. It would be very helpful if you could guide me through the process.

[axch]

Thanks for your interest @NeelGhoshal. No, as far as I know the issue is not currently fixed, and we would welcome a contribution. As far as process goes, check out our contributing guidelines. Basically, you grab the development sources, run unit tests, make your change (adding relevant unit tests), make a PR when tests pass, and sign Google's Contributor License Agreement if you haven't already. Then someone on the team reviews your PR (for something like this I would expect it would mostly be for code style), you update the PR in response, and when that's done we merge it.

[NeelGhoshal]

I checked out the Hager Zhang line search algorithm function, so it currently contains these parameters: initial_step_size,
value_at_initial_step, value_at_zero, threshold_use_approximate_wolfe_condition=1e-06, shrinkage_param=0.66, expansion_param=5.0, sufficient_decrease_param=0.1 , curvature_param=0.9, max_iterations=50
What params would you want added/changed w.r.t. these?

[NeelGhoshal]

Hi @axch, could you please point out what I need to change in the line search code? Which parameters should I focus on adding to the current list of parameters

[axch]

PR #1070 was a good start (and see the comments thereupon). I don't think it's a good idea to list individual parameters of hager_zhang anywhere else, because if we implement another line search algorithm at some point, it will presumably have different parameters. I would suggest taking a dictionary argument at the toplevel minimize, passing it through to line_search_step, and then **-expanding it at the call to hager_zhang.

[VijaySamant4368]

Hi! I checked the latest stable versions locally and tfp.optimizer.bfgs_minimize still does not have a line_search_kwargs parameter.

Environment:

• Python 3.10.0
• TensorFlow 2.20.0
• TensorFlow Probability 0.25.0
• tf_keras 2.20.1
• Windows

The current signature of the function is:

(value_and_gradients_function, initial_position, tolerance=1e-08,
 x_tolerance=0, f_relative_tolerance=0, initial_inverse_hessian_estimate=None,
 scale_initial_inverse_hessian=True, max_iterations=50,
 parallel_iterations=1, stopping_condition=None, validate_args=True,
 max_line_search_iterations=50, f_absolute_tolerance=0, name=None)

Is this feature still desired? Should I revive PR #1070 or open a new PR?

[hasan-malik]

Opened #2023 to address this. It's based on @axch's review of PR #1070 and implements the additional test.