Is there a way how to deal with large datasets using pytest?

[sinsoff]

Hello everyone!

I am currently testing Machine Learning services. And I need to calculate different statistics like: success rate, false positive rate, etc. I have large datasets (500-10000 images) for it and etalons for each of them. Each service takes one image as input, so theoretically, i have to run the same test len(dataset) times.

So my questions are:
1. How to aggregate results of len(dataset) tests?
2. How to add tables, plots, etc to the Report?

[nicoddemus]

Hey @sinsoff,

Not exactly answering your questions, but you can take a look at https://github.com/ESSS/pytest-regressions and https://github.com/gabrielcnr/pytest-datadir as starting points. 👍

Hope this helps.

[prusse-martin]

I will share my try on something similar (or not?). Probably not a good solution for this problem =)
A few weeks ago I used pytest to gather profile results.
I went to create a session scoped fixture to gather the results and on teardown print the results.
(session scoped fixture is good for me because I was not using xdist or suffer any worker crash in my profile)

For bigger and more robust collection you will probably want to submit those obtained statics to some kind of database then use the pytest hook pytest_sessionfinish (I am not sure if xdist or worker crash causes this to be called multiple times) to aggregate the results and create your plots.