Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate scheduled ETLs to Civis #273

Open
hunterowens opened this issue May 7, 2020 · 1 comment
Open

Migrate scheduled ETLs to Civis #273

hunterowens opened this issue May 7, 2020 · 1 comment
Assignees
Labels
civis civis data science platform work

Comments

@hunterowens
Copy link
Contributor

hunterowens commented May 7, 2020

Currently, we have the following DAGs

├── homelessness
│   ├── dag_homeless_merge_into_common_schema.py
│   ├── racer_nightly_to_storage.py
│   ├── static_datasets
│   └── sync_rap_data.py
├── myla311
│   ├── dag_311_cd_outlier_detector.py
│   ├── dag_311_outlier_detector.py
│   ├── dag_311_retrieve_update.py
│   └── LA_City_Council_Districts.csv
├── public-health
│   ├── care-311-to-postgres.py
│   ├── Code55.py
│   ├── Code75.py
│   └── covid19
│       ├── get-help-to-esri.py
│       ├── jhu-county-to-esri.py
│       ├── jhu-to-esri.py
│       ├── README.md
│       ├── shelter_timeseries_current.csv
│       ├── sync-bed-availability-data.py
│       ├── sync-covid-testing-data.py
│       ├── sync-la-cases-data.py
└── transportation
    ├── bikeshare
    │   └── trips.py
    ├── dash
    │   └── trips.py
    ├── dockless
    │   ├── create_la_boundary.sql
    │   ├── dockless_elt.py
    │   ├── geom_parse.sql
    │   ├── indexes.sql
    │   ├── README.md
    │   ├── scooter-stat.py
    │   └── v_trips.sql
    ├── metro
    │   └── ridership.py
    └── waze
        ├── dag-waze-dataProcessor.py
        └── store_data_file_nologin.py

@hunterowens
Copy link
Contributor Author

Need to migrate

  1. homelessness/sync-rap-data.py
  2. MyLA311 can be deleted once we merge Import all of 311 in Redshift #254
  3. care-311-to-postgres should just be a view off of Import all of 311 in Redshift #254
  4. publichealth/code55.py / code75.py - possible to drop. @ian-r-rose to follow up with Oscar.
  5. publichealth/covid19 don't migrate
  6. transportation/bikeshare/trips.py @ian-r-rose to migrate
  7. transportation/dash/trip.py @ian-r-rose
  8. transportation/dockless @hunterowens to wait for @thekaveman to finish up library updates, then rebuild DB.
  9. metro/ridership.py @ian-r-rose to migrate
  10. waze is currently thousands of files split across two s3 buckets. Hunter to figure out long term how to host in Redshift, but fine to eliminate the transfer DAG for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
civis civis data science platform work
Projects
None yet
Development

No branches or pull requests

3 participants