feature request to pass out the test function output (like @elapsed, @btime can do)

[ahbarnett]

Dear Lilith,
I am starting to use Chairmarks more - thanks for a great tool.
However the major reason for not adopting it is that I can't figure out how to access the output of the tested function. So the feature request is either to provide access to this output or explain how to do it in the tutorial.

Obviously for small examples like @b rand(1000) sum this value doesn't matter, and there will be a long list of values, one for each run, as I can see via @b rand(1000) sum println

However in scientific code devel it is very common to want to insert a timing/alloc measurement without affecting the overall code functioning. Currently with Chairmarks I have to duplicate the line I want to time, and discard the output in this extra line. This means uglier code and twice as long a wait.
Example, original code

A = myfunc(x)
B = something(A)

Say I want to time myfunc in context. Currently Chairmarks seems to require

A = myfunc(x)
@b myfunc(x)
B = something(A)

Say myfunc is slow (>10 seconds), then Charimarks will run it once. So now the overall code takes twice as long and is hard to maintain (since IRL myfunc(x) will be some long code line I do not want to duplicate). What would be much preferred is:

@b A = myfunc(x)
B = something(A)

This runs but does not change the value of A. But with other tools I can do it:

t = @elapsed A = myfunc(x)
B = something(A)

which is great (since A is correct and I get to store t easily for later use), but no allocation measurement done.

using BenchmarkTools
BenchmarkTools.DEFAULT_PARAMETERS.seconds=0.1
A = @btime myfunc(x)
B = something(A)

If multiple runs are done, just the last could be used (not sure if @btime does that). Thanks for considering this feature request! Best, Alex

[LilithHafner]

How to get what you asked for

You can explicitly store the value in the teardown function:

julia> x = Ref{Float64}()
Base.RefValue{Float64}(0.0)

julia> @b rand(100) sum x[] = _
12.166 ns

julia> x[]
52.6901305539253

which lets you store just the last result (as done above) or every result:

julia> x = Float64[]
Float64[]

julia> @b rand(100) sum push!(x, _)
12.182 ns

julia> x
6975-element Vector{Float64}:
 52.393088513621535
 50.07351749469598
  ⋮
 47.86031675247005
 47.66991582609712

The teardown function's role is documented in ?@be. It runs after each sample and it's runtime is not measured (just like setup, but after instead of before).

An alternative to consider

For non-invasive, low overhead benchmarking of long running codes, @time and @timed are great choices. If the code being benchmarked has side effects, beware that @b may run it repeatedly. And if the code being benchmarked is longer than the runtime budged for benchmarking (0.1s by default), then @b is no more accurate than @time. I explain this a bit more in th the explanations page of the docs.

A way to make this easier

Chairmarks currently does not store function outputs because they may be quite large and I don't want to keep lots of memory around unless the user explicitly asks for it. One option would be to add a result::Any field of sample and set it to nothing unless the user passes save_result=true or somesuch in which case it stores the output of the final step of the pipeline.

[ahbarnett]

Thanks, this is a useful summary (it would be wonderful if you can add your two examples to your tutorial). If a more complicated tuple is returned, I don't know how to preallocate/Ref/push it. But I'd forgotten @timed does include alloc/GC info. The save_result API extension is useful but you're right it's not high priority. Feel free to close or move Issue to Discussions. Cheers, Alex

[LilithHafner]

Coolbeans, I'm going to document using @timed in the tutorial or FAQ or somesuch and then I'll close this issue and invite you or others to re-open if @timed doesn't meet your needs.