changes relating to issue 237

[BlaiseKelly]

It seems to be related to the locate_file functions which are difficult to debug as there are so many nested functions.

However, the dl_stats19 function either downloads the data or identifies the file in the local directories. Either way the directory of the data is known by the end of this function. So it maybe simplifies things if this just returns the full path of the one file that is being requested (only one year or type is allowed to be requested at any one time). This also negates many of the read functions.

There was a bug in the locate_files functions which was preventing multiple files from being returned for 2020. This amendment changes the dl_stats19 function to return the file path of either the downloaded file or the file already in the local folder. This bypasses the locate_file and read_in functions.

In the same function changed the default file to be returned if ask is FALSE to be the second file returned and not the e-scooter.

I was also getting errors importing earlier files as it seemed to be searching for a file with the year, however it appears all years prior to 2017 are stored in a single file named "1979".

A few other minor changes which hopefully won't cause any other issues with other functions

[Robinlovelace]

+1 to that, although will it still work for non 1979 (all) years? I guess so but would need to test..

[Robinlovelace]

+1 to moving everything into a more accessible place.

[Robinlovelace]

OK tests failing. Will investigate..

[BlaiseKelly]

Am not so familiar with the package testing, but have moved get_url test to test-get and run the testthat.R script (didn't do that before) and it passes with only warnings (on windows).

[Robinlovelace]

Hey @BlaiseKelly I've checked out your branch and had a look but no luck, as you say lots of changes.

Would be able to resubmit a more focussed PR, or we can set-up a call to double team this and you can talk me through the changes.

There are definitely issues with the code, I think we could remove many lines and make the package better, especially now DfT datasets are simpler. However, this PR actually adds more lines than it removes due to rendered HTML. Really keen to get your changes in in any case, just a question of working out how.

[Robinlovelace]

Ps. these are the errors I get when running devtools::check() locally:

checking top-level files ... NOTE
  Non-standard files/directories found at top level:
    ‘README.qmd’ ‘README_files’

── Test failures ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── testthat ────

> library(testthat)
> library(stats19)
Data provided under OGL v3.0. Cite the source and link to:
www.nationalarchives.gov.uk/doc/open-government-licence/version/3/
> 
> test_check("stats19")
[ FAIL 2 | WARN 5 | SKIP 0 | PASS 34 ]

══ Warnings ════════════════════════════════════════════════════════════════════
── Warning ('test-get.R:16:3'): get_stats19 works ──────────────────────────────
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Backtrace:
    ▆
 1. ├─stats19::get_stats19(year = 2019, type = "collision") at test-get.R:16:3
 2. │ └─stats19:::read_ve_ca(path = file_path)
 3. │   └─stats19:::read_null(path)
 4. │     └─readr::read_csv(path, ...)
 5. │       └─vroom::vroom(...)
 6. │         └─vroom:::vroom_(...)
 7. └─cli (local) `<fn>`(`<chr>`, "vroom_parse_issue")
── Warning ('test-get.R:18:3'): get_stats19 works ──────────────────────────────
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Backtrace:
    ▆
 1. ├─stats19::get_stats19(year = 2019) at test-get.R:18:3
 2. │ └─stats19:::read_ve_ca(path = file_path)
 3. │   └─stats19:::read_null(path)
 4. │     └─readr::read_csv(path, ...)
 5. │       └─vroom::vroom(...)
 6. │         └─vroom:::vroom_(...)
 7. └─cli (local) `<fn>`(`<chr>`, "vroom_parse_issue")
── Warning ('test-get.R:26:3'): get_stats19 works ──────────────────────────────
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Backtrace:
    ▆
 1. ├─stats19::get_stats19(year = 2019, type = "collision") at test-get.R:26:3
 2. │ └─stats19:::read_ve_ca(path = file_path)
 3. │   └─stats19:::read_null(path)
 4. │     └─readr::read_csv(path, ...)
 5. │       └─vroom::vroom(...)
 6. │         └─vroom:::vroom_(...)
 7. └─cli (local) `<fn>`(`<chr>`, "vroom_parse_issue")
── Warning ('test-get.R:27:3'): get_stats19 works ──────────────────────────────
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Backtrace:
    ▆
 1. ├─stats19::get_stats19(year = 2019, type = "collision", output_format = "data.frame") at test-get.R:27:3
 2. │ └─stats19:::read_ve_ca(path = file_path)
 3. │   └─stats19:::read_null(path)
 4. │     └─readr::read_csv(path, ...)
 5. │       └─vroom::vroom(...)
 6. │         └─vroom:::vroom_(...)
 7. └─cli (local) `<fn>`(`<chr>`, "vroom_parse_issue")
── Warning ('test-get.R:28:3'): get_stats19 works ──────────────────────────────
One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)
Backtrace:
    ▆
 1. ├─stats19::get_stats19(year = 2019, type = "collision", output_format = "sf") at test-get.R:28:3
 2. │ └─stats19:::read_ve_ca(path = file_path)
 3. │   └─stats19:::read_null(path)
 4. │     └─readr::read_csv(path, ...)
 5. │       └─vroom::vroom(...)
 6. │         └─vroom:::vroom_(...)
 7. └─cli (local) `<fn>`(`<chr>`, "vroom_parse_issue")

══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-utils.R:6:3'): find_file_name works ──────────────────────────
length(find_file_name(type = "coll")) > 5 is not TRUE

`actual`:   FALSE
`expected`: TRUE 
── Failure ('test-utils.R:9:3'): find_file_name works ──────────────────────────
`find_file_name(years = 1973)` did not produce any messages.

[ FAIL 2 | WARN 5 | SKIP 0 | PASS 34 ]
Error: Test failures
Execution halted

2 errors ✖ | 0 warnings ✔ | 1 note ✖

[BlaiseKelly]

The README and html were added later to my branch. I didn't intend to push them to the main.
Will try to redo it without.