Functions in prim/mat/fun that could use custom reverse mode autodiff implementations

[bbbales2]

Description

The functions listed here currently do not have custom reverse mode autodiff implementations.

Some of these functions would be easy and some would be hard. I'll be piecemeal working on sections of this pull in the coming weeks, so you should post a message here if you want to work on any of these so we'll stay coordinated.

The main reason many functions are included in this list is that they create a lot of varis on the chaining stack.

A lot of not-included functions (rep_matrix, sort_asc, fill, etc.) were not included because they didn't have large computational components

• csr_matrix_times_vector -- Check discourse (at least https://discourse.mc-stan.org/t/sparse-matrix-roadmap/5493) before trying
• csr_to_dense_matrix -- Edit: Looking at this (April 2020), I don't think this is a good target. Regular autodiff probably fast enough
• cumulative_sum
• diag_pre_multiply
• diag_post_multiply
• gp_dot_prod_cov
• gp_exp_quad_cov
• gp_matern52_cov
• gp_periodic_cov
• inverse
• log_softmax
• make_nu
• mdivide_right
• mdivide_right_ldlt
• mdivide_right_spd
• mdivide_right_tri
• quad_form_diag
• cholesky_corr_constrain
• cholesky_factor_constrain
• corr_matrix_constrain
• cov_matrix_constrain
• cov_matrix_constrain_lkj

Based on this test (https://discourse.mc-stan.org/t/adj-jac-apply/5163/6), I'm not sure these would be sped up with custom autodiff, but they don't have it implemented:

• add
• add_diag
• divide
• elt_multiply
• elt_divide
• minus
• subtract

There are a couple other functions that could have custom autodiff but the math would not be easy mathematically:

• eigenvalues_sym
• eigenvectors_sym
• singular_values

Current Math Version

v2.18.0

[andrjohns]

@bbbales2 which of these have you implemented so far? I can work on some of the others

[bbbales2]

I haven't done any of them, but there are updates! I'll update the issue.

I did a test with the autodiff stuff and I think I should get some of them off the list cause the default implementations are going to be fast (https://discourse.mc-stan.org/t/adj-jac-apply/5163/6). That'd be like add, add_diag, elt_multiply, elt_divide, minus, subtract. These are the functions where I assumed that reducing the numbers of varis on the stack by using adj_jac_apply would help. I don't think that is true anymore.

The gp functions might have custom reverse modes now. That should be checked.

And the sparse stuff (csr and friends) looks weird but it is written this way to be super pedantic about errors (search on Discourse). There might be a way to write it faster, but that'd require figuring out the technicalities of why it is written the way it is first.

[andrjohns]

Excellent, I'll work through some of these in my spare time. I'll leave the sparse stuff for now since I'd have to get more familiar with the implementation, but I'll call dibs on:

• cumulative_sum
• diag_pre_multiply
• diag_post_multiply

[bbbales2]

Cool! I know it's probably a bit annoying, but you should build in little performance tests to convince yourself that adj_jac_apply is actually speeding things up. It's unfortunately fairly easy to write code that will be less efficient operation-wise than regular ol' autodiff :D.

[bbbales2]

I don't think the performance tests need to be part of the pull, btw. You can just report your results here and maybe share the basic code with gists with the reviewer or whatnot. Just double check cause the plain ol' autodiff is pretty quick even if it is pointer-chasing like crazy.

[andrjohns]

That's a good idea, will do. I've got some questions about how to implement adj_jac_apply, but I'll put them on discourse when I get stuck

[bob-carpenter]

OK---just make sure to put it in the developers category. I don't have time to read the user sections any more, but am still following the developers parts!