BayesianExperiments.jl
This is the documentation of BayesianExperiments.jl
, a library for conducting Bayesian AB testing in Julia.
Current features include:
- Hypothesis testing with Bayes factor. Support the effect size model with Normal distribution prior and JZS prior.
- Bayesian decision making with conjugate prior models. Support expected loss and probability to beat all as the stopping rule.
- Flexible experiment design for both fixed horizon experiments and sequential test experiment.
- Efficient simulation tools to support power analysis and sensitivity analysis.
Installation
You can install a stable version of BayesianExperiments by running the command in the Julia REPL:
julia> ] add BayesianExperiments
Quick Start
Here's a simple example showing how to use the package:
using BayesianExperiments
# Generate sample data
n = 1000
dataA = rand(Bernoulli(0.15), n)
dataB = rand(Bernoulli(0.16), n)
# Define the models
modelA = ConjugateBernoulli(1, 1)
modelB = ConjugateBernoulli(1, 1)
# Choose the stopping rule that we will use for making decision
stoppingrule = ExpectedLossThresh(0.0002)
# Setup the experiment by specifying models and the stopping rule
experiment = ExperimentAB([modelA, modelB], stoppingrule)
# Calculate the statistics from our sample data
statsA = BetaStatistics(dataA)
statsB = BetaStatistics(dataB)
# Update the models in the experiment with the newly created statistics
update!(experiment, [statsA, statsB])
# Calculate the metric (expected loss in this case) of each model
winner_index, expected_losses = metrics(experiment)
Contributing
We welcome contributions to this project and discussion about its contents. Please open an issue or pull request on this repository to propose a change.