Merge pull request #12 from iag-geo/202405

202405

README.md

@@ -1,14 +1,14 @@
 # Concord
 
-A [CSV file](https://minus34.com/opendata/geoscape-202402/boundary_concordance.csv) and supporting scripts for converting data between Australian boundaries.
+A [CSV file](https://minus34.com/opendata/geoscape-202405/boundary_concordance.csv) and supporting scripts for converting data between Australian boundaries.
 
 It solves the problem of trying to merge 2 or more datasets based on different census or administrative boundaries such as statistical areas or postcodes.
 
 It does this by providing a list of **_concordances_** between pairs of boundaries.  _e.g. In the image below: 100% of postcode 3126 fits within the Boroondara LGA. However, only ~46% of postcode 3127 fits within that LGA._
 
 In this context, **_concordance_** describes what % of residential addresses in a "from" boundary fit within a "to" boundary.
 
-Download & import the ~50Mb [concordance file](https://minus34.com/opendata/geoscape-202402/boundary_concordance.csv) into your database or reporting tool to [get started](#get-started). A [script](/postgres-scripts/00_import_concordance_file.sql) for importing into Postgres is also provided.
+Download & import the ~50Mb [concordance file](https://minus34.com/opendata/geoscape-202405/boundary_concordance.csv) into your database or reporting tool to [get started](#get-started). A [script](/postgres-scripts/00_import_concordance_file.sql) for importing into Postgres is also provided.
 
 ### Example Use Cases
 
@@ -42,7 +42,7 @@ In the [score](/data/boundary_concordance_score.csv) file, the **_error_** measu
 
 The concordance file is generated by the following process:
 
-1. Tag all GNAF addresses with 2016 & 2021 ABS Census boundaries and geoscape 202402 Administrative boundaries
+1. Tag all GNAF addresses with 2016 & 2021 ABS Census boundaries and geoscape 202405 Administrative boundaries
 2. Remove all addresses in non-residential ABS Census 2021 meshblocks
 3. Aggregate all residential addresses by a set of _**from**_ boundary and _**to**_ boundary pairs (e.g. postcode to LGA)
 4. Determine the % overlap of residential addresses between both boundary types for all boundary pairs
@@ -63,7 +63,7 @@ There are 2 options to get the data:
 
 #### 1. Download and Import
 
-1. Download the [concordance file](https://minus34.com/opendata/geoscape-202402/boundary_concordance.csv)
+1. Download the [concordance file](https://minus34.com/opendata/geoscape-202405/boundary_concordance.csv)
 2. Import it into your database/reporting tool of choice. If using Postgres:
     1. Edit the file path, schema name & table owner in `00_import_concordance_file.sql` in the [postgres-scripts](/postgres-scripts) folder
     2. Run the SQL script to import the file  
@@ -75,9 +75,9 @@ This requires a knowledge of Python, Postgres & pg_restore.
 BTW - if the boundary combination you want isn't in the default concordance file - you need to edit the `settings.py` file before running `create_concordance_file.py`. If this is too hard - raise an [issue](https://github.com/iag-geo/concord/issues) and we may be able to generate it for you; noting you shouldn't convert data to a smaller boundary due to the increase in data errors.
 
 **Running the script only needs to be done for 3 reasons:**
-1. The boundary from/to combination you need isn't in the standard [concordances file](https://minus34.com/opendata/geoscape-202402/boundary_concordance.csv)
+1. The boundary from/to combination you need isn't in the standard [concordances file](https://minus34.com/opendata/geoscape-202405/boundary_concordance.csv)
 2. It's now the future and we've been too lazy to update the concordances file with the latest boundary data from the ABS and/or Geoscape
-3. You have a license of [Geoscape Buildings](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202402/boundary_concordance.csv/buildings/) or [Geoscape Land Parcels](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202402/boundary_concordance.csv/land-parcels/) and want to use the _planning zone_ data in those products to:
+3. You have a license of [Geoscape Buildings](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202405/boundary_concordance.csv/buildings/) or [Geoscape Land Parcels](https://geoscape.com.auhttps://minus34.com/opendata/geoscape-202405/boundary_concordance.csv/land-parcels/) and want to use the _planning zone_ data in those products to:
     1. Use a more accurate list of residential addresses to determine the data apportionment percentages (see **note** below); or
     2. Use a different set of addresses to apportion your data; e.g. industrial or commercial addresses
 
@@ -88,8 +88,8 @@ BTW - if the boundary combination you want isn't in the default concordance file
 Running the script requires the following open data, available as Postgres dump files, as well as the optional licensed Geoscape data mentioned above:
 1. ABS Census 2016 boundaries ([download](https://minus34.com/opendata/census-2016/census_2016_bdys.dmp))
 2. ABS Census 2021 boundaries ([download](https://minus34.com/opendata/census-2021/census_2021_bdys_gda94.dmp))
-3. GNAF from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202402/gnaf-202402.dmp))
-4. Geoscape Administrative Boundaries from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202402/admin-bdys-202402.dmp))
+3. GNAF from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202405/gnaf-202405.dmp))
+4. Geoscape Administrative Boundaries from gnaf-loader ([download](https://minus34.com/opendata/geoscape-202405/admin-bdys-202405.dmp))
 5. ABS Census 2016 data - used to generate error rates only ([download](https://minus34.com/opendata/census-2016/census_2016_data.dmp))
 
 #### Process
@@ -117,7 +117,7 @@ The behaviour of the Python script can be controlled by specifying various comma
 * `--pgpassword` password for accessing the Postgres server. This defaults to the `PGPASSWORD` environment variable if set, otherwise `password`.
 
 ##### Optional Arguments
-* `--geoscape-version` Geoscape version number in YYYYMM format. Defaults to current year and last release month. e.g. `202402`.
+* `--geoscape-version` Geoscape version number in YYYYMM format. Defaults to current year and last release month. e.g. `202405`.
 * `--gnaf-schema` input schema name to store final GNAF tables in. Also the **output schema** for the concordance table. Defaults to `gnaf_<geoscape_version>`.
 * `--admin-schema` input schema name to store final admin boundary tables in. Defaults to `admin_bdys_<geoscape_version>`.
 * `--output-table` name of both output concordance table and file. Defaults to `boundary_concordance`.
@@ -147,8 +147,8 @@ WITH pc_data AS (
           con.to_name AS lga_name,
           sum(pc.cases::float * con.address_percent / 100.0)::integer AS cases
    FROM testing.nsw_covid_cases_20220503_postcode AS pc
-           INNER JOIN gnaf_202402.boundary_concordance AS con ON pc.postcode = con.from_id
-   WHERE con.from_source = 'geoscape 202402'
+           INNER JOIN gnaf_202405.boundary_concordance AS con ON pc.postcode = con.from_id
+   WHERE con.from_source = 'geoscape 202405'
      AND con.from_bdy = 'postcode'
      AND con.to_source = 'abs 2016'
      AND con.to_bdy = 'lga'
@@ -167,9 +167,9 @@ FROM testing.nsw_covid_tests_20220503_lga AS lga
 
 ## Data Licenses
 
-Incorporates or developed using G-NAF © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202402/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the [Open Geo-coded National Address File (G-NAF) End User Licence Agreement](https:/https://minus34.com/opendata/geoscape-202402/boundary_concordance.csv.gov.auhttps://minus34.com/opendata/geoscape-202402/boundary_concordance.csvset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=).
+Incorporates or developed using G-NAF © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202405/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the [Open Geo-coded National Address File (G-NAF) End User Licence Agreement](https:/https://minus34.com/opendata/geoscape-202405/boundary_concordance.csv.gov.auhttps://minus34.com/opendata/geoscape-202405/boundary_concordance.csvset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=).
 
-Incorporates or developed using Administrative Boundaries © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202402/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
+Incorporates or developed using Administrative Boundaries © [Geoscape Australia](https://geoscape.com.au/legalhttps://minus34.com/opendata/geoscape-202405/boundary_concordance.csv-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
 
 Based on [Australian Bureau of Statistics](https://www.abs.gov.au/websitedbs/d3310114.nsf/Home/Attributing+ABS+Material) data, licensed by the Commonwealth of Australia under [Creative Commons Attribution 4.0 International licence (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/).
 

create_concordance_file.py

@@ -202,7 +202,7 @@ def add_asgs_concordances(pg_cur):
                                    count(*) as address_count,
                                    100.0 as address_percent
                             from census_2016_bdys.mb_2016_aust as mb
-                            inner join gnaf_202402.address_principals as gnaf on gnaf.mb_2016_code::text = mb.mb_code16
+                            inner join gnaf_202405.address_principals as gnaf on gnaf.mb_2016_code::text = mb.mb_code16
                             group by from_id,
                                      from_name,
                                      to_id,
@@ -250,7 +250,7 @@ def add_asgs_concordances(pg_cur):
                                    count(*) as address_count,
                                    100.0 as address_percent
                             from census_2021_bdys_gda94.mb_2021_aust_gda94 as mb
-                                     inner join gnaf_202402.address_principals as gnaf 
+                                     inner join gnaf_202405.address_principals as gnaf 
                                          on gnaf.mb_2021_code::text = mb.mb_code_2021
                             group by from_id,
                                      from_name,

data/boundary_concordance_score.csv

@@ -2,7 +2,7 @@ from_source,from_bdy,to_source,to_bdy,concordance_percent,error_percent
 abs 2016,gcc,abs 2021,gccsa,100,
 abs 2016,lga,abs 2016,gccsa,99,0.1
 abs 2016,lga,abs 2016,ra,92,1.2
-abs 2016,lga,abs 2016,sa3,73,4.8
+abs 2016,lga,abs 2016,sa3,73,4.9
 abs 2016,lga,abs 2016,ste,100,0.2
 abs 2016,poa,abs 2016,lga,93,1.4
 abs 2016,poa,abs 2016,sa2,62,8.9
@@ -13,11 +13,11 @@ abs 2016,sa1,abs 2016,sa4,100,0.0
 abs 2016,sa1,abs 2021,sa1,92,
 abs 2016,sa2,abs 2016,gcc,100,0.0
 abs 2016,sa2,abs 2016,lga,97,0.5
-abs 2016,sa2,abs 2016,poa,79,4.4
+abs 2016,sa2,abs 2016,poa,79,4.5
 abs 2016,sa2,abs 2016,sa3,100,0.0
 abs 2016,sa2,abs 2016,sa4,100,0.0
 abs 2016,sa2,abs 2021,sa2,92,
-abs 2016,sa2,geoscape 202402,postcode,78,
+abs 2016,sa2,geoscape 202405,postcode,78,
 abs 2016,sa3,abs 2016,gcc,100,0.0
 abs 2016,sa3,abs 2016,lga,83,3.0
 abs 2016,sa3,abs 2016,sa4,100,0.0
@@ -39,24 +39,24 @@ abs 2021,sa2,abs 2021,lga,98,
 abs 2021,sa2,abs 2021,poa,84,
 abs 2021,sa2,abs 2021,sa3,100,
 abs 2021,sa2,abs 2021,sa4,100,
-abs 2021,sa2,geoscape 202402,postcode,84,
+abs 2021,sa2,geoscape 202405,postcode,84,
 abs 2021,sa3,abs 2021,gccsa,100,
 abs 2021,sa3,abs 2021,lga,85,
 abs 2021,sa3,abs 2021,sa4,100,
 abs 2021,sa4,abs 2021,gccsa,100,
-geoscape 202402,lga,abs 2016,gccsa,99,
-geoscape 202402,lga,abs 2016,lga,100,
-geoscape 202402,lga,abs 2021,gccsa,100,
-geoscape 202402,lga,abs 2021,lga,100,
-geoscape 202402,locality,abs 2016,lga,98,
-geoscape 202402,locality,abs 2016,sa2,93,
-geoscape 202402,locality,abs 2016,sa3,99,
-geoscape 202402,locality,abs 2021,lga,98,
-geoscape 202402,locality,abs 2021,sa2,90,
-geoscape 202402,locality,abs 2021,sa3,99,
-geoscape 202402,locality,geoscape 202402,lga,98,
-geoscape 202402,postcode,abs 2016,lga,93,
-geoscape 202402,postcode,abs 2016,sa3,92,
-geoscape 202402,postcode,abs 2021,lga,94,
-geoscape 202402,postcode,abs 2021,sa3,93,
-geoscape 202402,postcode,geoscape 202402,lga,93,
+geoscape 202405,lga,abs 2016,gccsa,99,
+geoscape 202405,lga,abs 2016,lga,100,
+geoscape 202405,lga,abs 2021,gccsa,100,
+geoscape 202405,lga,abs 2021,lga,100,
+geoscape 202405,locality,abs 2016,lga,98,
+geoscape 202405,locality,abs 2016,sa2,93,
+geoscape 202405,locality,abs 2016,sa3,99,
+geoscape 202405,locality,abs 2021,lga,98,
+geoscape 202405,locality,abs 2021,sa2,90,
+geoscape 202405,locality,abs 2021,sa3,99,
+geoscape 202405,locality,geoscape 202405,lga,98,
+geoscape 202405,postcode,abs 2016,lga,93,
+geoscape 202405,postcode,abs 2016,sa3,92,
+geoscape 202405,postcode,abs 2021,lga,94,
+geoscape 202405,postcode,abs 2021,sa3,93,
+geoscape 202405,postcode,geoscape 202405,lga,93,

postgres-scripts/00_import_concordance_file.sql

@@ -1,7 +1,7 @@
 
 -- create table
-drop table if exists gnaf_202402.boundary_concordance;
-create table gnaf_202402.boundary_concordance
+drop table if exists gnaf_202405.boundary_concordance;
+create table gnaf_202405.boundary_concordance
 (
     from_source     text not null,
     from_bdy        text not null,
@@ -14,21 +14,21 @@ create table gnaf_202402.boundary_concordance
     address_count   integer,
     address_percent numeric(4, 1)
 );
-alter table gnaf_202402.boundary_concordance owner to postgres;
+alter table gnaf_202405.boundary_concordance owner to postgres;
 
 -- import CSV file -- 586,977 rows affected in 1 s 365 ms
-COPY gnaf_202402.boundary_concordance
+COPY gnaf_202405.boundary_concordance
     FROM '/Users/minus34/Downloads/boundary_concordance.csv'
     WITH (HEADER, DELIMITER ',', FORMAT CSV);
 
-analyse gnaf_202402.boundary_concordance;
+analyse gnaf_202405.boundary_concordance;
 
 -- add primary key (faster if done after import) -- completed in 8 s 496 ms
-alter table gnaf_202402.boundary_concordance add constraint boundary_concordance_pkey
+alter table gnaf_202405.boundary_concordance add constraint boundary_concordance_pkey
     primary key (from_source, from_bdy, from_id, to_source, to_bdy, to_id);
 
 -- add index on required fields for converting data
-create index boundary_concordance_combo_idx on gnaf_202402.boundary_concordance
+create index boundary_concordance_combo_idx on gnaf_202405.boundary_concordance
     using btree (from_source, from_bdy, to_source, to_bdy);
 
-alter table gnaf_202402.boundary_concordance cluster on boundary_concordance_combo_idx;
+alter table gnaf_202405.boundary_concordance cluster on boundary_concordance_combo_idx;

postgres-scripts/data-prep/02_create_residential_address_table.sql

@@ -27,9 +27,9 @@ select gnaf.gnaf_pid,
        gnaf.mb_2021_code,
        lower(mb21.mb_cat) as mb_category_2021,
        gnaf.geom
-from gnaf_202402.address_principals as gnaf
-     inner join admin_bdys_202402.abs_2016_mb as mb16 on mb16.mb_16code = gnaf.mb_2016_code
-     inner join admin_bdys_202402.abs_2021_mb as mb21 on mb21.mb21_code = gnaf.mb_2021_code
+from gnaf_202405.address_principals as gnaf
+     inner join admin_bdys_202405.abs_2016_mb as mb16 on mb16.mb_16code = gnaf.mb_2016_code
+     inner join admin_bdys_202405.abs_2021_mb as mb21 on mb21.mb21_code = gnaf.mb_2021_code
     left outer join blg on blg.gnaf_pid = gnaf.gnaf_pid
 ;
 analyse geoscape_202203.address_principals_buildings;

postgres-scripts/data-prep/03_copy_to_gda2020_schema.sql

@@ -1,34 +1,34 @@
 
-CREATE SCHEMA IF NOT EXISTS gnaf_202402_gda2020 AUTHORIZATION postgres;
+CREATE SCHEMA IF NOT EXISTS gnaf_202405_gda2020 AUTHORIZATION postgres;
 
 -- create tables
-drop table if exists gnaf_202402_gda2020.address_principal_census_2016_boundaries;
-create table gnaf_202402_gda2020.address_principal_census_2016_boundaries as
-select * from gnaf_202402.address_principal_census_2016_boundaries;
-analyse gnaf_202402_gda2020.address_principal_census_2016_boundaries;
-
-drop table if exists gnaf_202402_gda2020.address_principal_census_2021_boundaries;
-create table gnaf_202402_gda2020.address_principal_census_2021_boundaries as
-select * from gnaf_202402.address_principal_census_2021_boundaries;
-analyse gnaf_202402_gda2020.address_principal_census_2021_boundaries;
-
-drop table if exists gnaf_202402_gda2020.boundary_concordance_score;
-create table gnaf_202402_gda2020.boundary_concordance_score as
-select * from gnaf_202402.boundary_concordance_score;
-analyse gnaf_202402_gda2020.boundary_concordance_score;
-
-drop table if exists gnaf_202402_gda2020.boundary_concordance;
-create table gnaf_202402_gda2020.boundary_concordance as
-select * from gnaf_202402.boundary_concordance;
-analyse gnaf_202402_gda2020.boundary_concordance;
+drop table if exists gnaf_202405_gda2020.address_principal_census_2016_boundaries;
+create table gnaf_202405_gda2020.address_principal_census_2016_boundaries as
+select * from gnaf_202405.address_principal_census_2016_boundaries;
+analyse gnaf_202405_gda2020.address_principal_census_2016_boundaries;
+
+drop table if exists gnaf_202405_gda2020.address_principal_census_2021_boundaries;
+create table gnaf_202405_gda2020.address_principal_census_2021_boundaries as
+select * from gnaf_202405.address_principal_census_2021_boundaries;
+analyse gnaf_202405_gda2020.address_principal_census_2021_boundaries;
+
+drop table if exists gnaf_202405_gda2020.boundary_concordance_score;
+create table gnaf_202405_gda2020.boundary_concordance_score as
+select * from gnaf_202405.boundary_concordance_score;
+analyse gnaf_202405_gda2020.boundary_concordance_score;
+
+drop table if exists gnaf_202405_gda2020.boundary_concordance;
+create table gnaf_202405_gda2020.boundary_concordance as
+select * from gnaf_202405.boundary_concordance;
+analyse gnaf_202405_gda2020.boundary_concordance;
 
 -- add primary key (faster if done after import) -- completed in 8 s 496 ms
-alter table gnaf_202402_gda2020.boundary_concordance add constraint boundary_concordance_gda2020_pkey
+alter table gnaf_202405_gda2020.boundary_concordance add constraint boundary_concordance_gda2020_pkey
     primary key (from_source, from_bdy, from_id, to_source, to_bdy, to_id);
 
 -- add index on required fields for converting data
-create index boundary_concordance_gda2020_combo_idx on gnaf_202402_gda2020.boundary_concordance
+create index boundary_concordance_gda2020_combo_idx on gnaf_202405_gda2020.boundary_concordance
     using btree (from_source, from_bdy, to_source, to_bdy);
 
-alter table gnaf_202402_gda2020.boundary_concordance cluster on boundary_concordance_gda2020_combo_idx;
+alter table gnaf_202405_gda2020.boundary_concordance cluster on boundary_concordance_gda2020_combo_idx;
 

postgres-scripts/example-usage/02_join_pc_and_lga_data.sql

@@ -19,7 +19,7 @@
 -- select * from testing.nsw_covid_tests_20220503_lga;
 --
 -- -- concordance table
--- select * from gnaf_202402.boundary_concordance;
+-- select * from gnaf_202405.boundary_concordance;
 
 WITH pc_data AS (
     SELECT con.to_id AS lga_id,
@@ -28,8 +28,8 @@ WITH pc_data AS (
            sum(pc.cases) as qa_count,
            count(*) as postcode_count
     FROM testing.nsw_covid_cases_20220503_postcode AS pc
-    INNER JOIN gnaf_202402.boundary_concordance AS con ON pc.postcode = con.from_id
-    WHERE con.from_source = 'geoscape 202402'
+    INNER JOIN gnaf_202405.boundary_concordance AS con ON pc.postcode = con.from_id
+    WHERE con.from_source = 'geoscape 202405'
       AND con.from_bdy = 'postcode'
       AND con.to_source = 'abs 2016'
       AND con.to_bdy = 'lga'
@@ -49,8 +49,8 @@ INNER JOIN pc_data on pc_data.lga_id = lga.lga_code19;
 
 -- example of poor concordance due to number of partial postcodes that intersect with one LGA
 select *
-from gnaf_202402.boundary_concordance
-WHERE from_source = 'geoscape 202402'
+from gnaf_202405.boundary_concordance
+WHERE from_source = 'geoscape 202405'
   AND from_bdy = 'postcode'
   AND to_source = 'abs 2016'
   AND to_bdy = 'lga'