doddle-benchmark

Benchmarking doddle-model implementations.

All experiments ran multiple times (iterations) for all implementations and with fixed hyperparameters, selected in a way such that models yielded similar test set performance.

Linear Regression

• dataset with 150000 training examples and 27147 test examples (10 features)
• each experiment ran for 100 iterations
• scikit-learn code, doddle-model code

Implementation	RMSE	Training Time	Prediction Time
scikit-learn	3.0936	0.042s (+/- 0.014s)	0.002s (+/- 0.002s)
doddle-model	3.0936	0.053s (+/- 0.061s)	0.002s (+/- 0.004s)

Logistic Regression

• dataset with 80000 training examples and 20000 test examples (250 features)
• each experiment ran for 100 iterations
• scikit-learn code, doddle-model code

Implementation	Accuracy	Training Time	Prediction Time
scikit-learn	0.8389	2.789s (+/- 0.090s)	0.005s (+/- 0.006s)
doddle-model	0.8377	3.080s (+/- 0.665s)	0.025s (+/- 0.025s)

Softmax Classifier

• MNIST dataset with 60000 training examples and 10000 test examples (784 features)
• each experiment ran for 50 iterations
• scikit-learn code, doddle-model code

Implementation	Accuracy	Training Time	Prediction Time
scikit-learn	0.9234	21.243s (+/- 0.303s)	0.074s (+/- 0.018s)
doddle-model	0.9223	25.749s (+/- 1.813s)	0.042s (+/- 0.032s)

Setup

To run the tests locally you will need to publish a local snapshot version of the repository.

    git clone https://github.com/picnicml/doddle-model.git
    cd doddle-model
    sbt publishLocal

Ensure the published version matches the version contained within the project/Dependencies.scala file.