Value Error when trying the library with ma2019.csv dataset

[mikel-hernandezj]

Hi,

I just wanted to notify you that I am getting a Value Error when trying to execute the command for the data quality report.

I would also like to ask how I can use your library to generate a quality report for an own dataset, not only for the datasets that are provided in the library.

I am sending you the traceback of the value error:

python -m sdnist.report diverse_communities_data_excerpts/massachusetts/ma2019.csv TX
| SDNist: Deidentified Data Report Tool
|-- Creating Evaluation Report for Deidentified Data at path: diverse_communities_data_excerpts/massachusetts/ma2019.csv
|---- Loading Datasets
|------ Features (24): ['AGEP', 'DEAR', 'DENSITY', 'DEYE', 'DPHY', 'DREM', 'DVET', 'EDU', 'HISP', 'HOUSING_TYPE', 'INDP', 'INDP_CAT', 'MSP', 'NOC', 'NPF', 'OWN_RENT', 'PINCP', 'PINCP_DECILE', 'POVPIP', 'PUMA', 'PWGTP', 'RAC1P', 'SEX', 'WGTP']
|------ Deidentified Data Records Count: 7634
|------ Target Data Records Count: 9276
|---- Finished Loading Datasets | Time: 0.9s <<<<
|---- Computing Utility Scores
|------ Finished Univariates | Time: 20.0s <<<<
/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/scipy/stats/_stats_py.py:4424: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
warnings.warn(stats.ConstantInputWarning(msg))
|------ Finished Correlations | Time: 2.6s <<<<
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/c/Users/mhernandez/SYNTHEMA/local-devs/sdnist-metrics-library/venv/lib/python3.9/site-packages/sdnist/report/main.py", line 154, in
run(**input_cnf)
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/sdnist/report/main.py", line 46, in run
ui_data, report_data = utility_score(dataset, ui_data, report_data, log)
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/sdnist/report/score/utility/init.py", line 576, in utility_score
s.compute_score()
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/sdnist/metrics/kmarginal.py", line 72, in compute_score
return self._compute_score_grouped()
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/sdnist/metrics/kmarginal.py", line 127, in _compute_score_grouped
group_den_sum = group_t_den_sum.where(group_t_den_sum <= group_den_sum).fillna(group_den_sum)
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/pandas/core/ops/common.py", line 81, in new_method
return method(self, other)
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/pandas/core/arraylike.py", line 52, in le
return self._cmp_method(other, operator.le)
File "/.../sdnist-metrics-library/venv/lib/python3.9/site-packages/pandas/core/series.py", line 6091, in _cmp_method
raise ValueError("Can only compare identically-labeled Series objects")
ValueError: Can only compare identically-labeled Series objects

[garyhowarth]

It looks like you are running MA data against the TX scoring function. This will fail because the PUMAs for MA fall out of the valid range for TX. It worth putting a better error message on our side to explain that.

SDNist doesn't support arbitrary schemas at this time. We hope to support bringing your own dataset in the future.