Jan open data -- draft for now #1356

edasmalchi · 2025-01-22T21:02:14Z

Finished running the pipeline with some necessary tweaks but creating this as a draft to get some additional review before considering it done.

Speedmaps had a lot of missing data (and the site will revert back to December for now) but I don't know if this is due to:

My downloading vehicle positions the day after (on last thursday with an analysis date of last wednesday, is that just too soon?)
Changes I made here to fix various errors
Something else

I'll try redownloading vehicle positions and rerunning rt_segment_speeds later today, but @tiffanychu90 if anything I did here jumps out at you as a possible cause please let me know.

Otherwise I'll make a separate issue once I learn a little more.

Example of missing data for SFMTA, looked at a few others and the pattern seemed to be the same:

segments_new.time_of_day.value_counts() #  january

Early AM    686
Evening     572
PM Peak     569
AM Peak     539
Midday      483
Owl         259
Name: time_of_day, dtype: int64

speedmap_segs.time_of_day.value_counts() #  december

Early AM    5194
PM Peak     5189
Evening     4836
AM Peak     4645
Midday      4296
Owl         2044
Name: time_of_day, dtype: int64

tiffanychu90 · 2025-01-22T22:13:49Z

My thoughts my looking at the changes:

I think I saw vp_idx is commented out...I'm wondering how you're able to link back to the vp_usable_dwell table after? The nearest neighbor approach keys into that and uses it to link vp without carrying a mess of columns. And it takes varying forms (it can be an array in the intermediate steps, so leaving it off and trying to merge it in would not be equivalent)
The logs look suspiciously fast...even compared to prior months.
- Having run this for the last 2 years, I'd say that times do not vary much, and any time there's a glaring drop in times without big refactors, I would check each staged parquet to see where the difference occurs.
- I would compare the first 3 tables for Dec 2024 and Jan 2025 and look at the number of rows. They should be the same magnitude from month to month. 15M in vp_usable then 12M in vp_usable_dwell, and then vp_condensed_line should be the same number of rows as number of trip_instance_keys.
```
speeds_tables:
 raw_vp: vp
 usable_vp: vp_usable
 vp_dwell: vp_usable_dwell
 vp_condensed_line: condensed/vp_condensed
```

Specifically...these logs jump out to me:

The very very large decrease in times seems to indicate that there's some large reduction in an input df, so either that's happening in one of the scripts captured in vp_preprocessing.log or in the initial download (your initial guess). It can be in either place, but the other logs don't seem to face that large decrease, so it seems to rule out the downloading of vp.
Why would something that takes 30-40 min take 5 min now?

2024-12-18 10:42:34.461 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:189 - nearest neighbor for stop_segments 2024-12-11: 0:36:50.316084
2024-12-18 14:32:17.520 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:189 - nearest neighbor for rt_stop_times 2024-12-11: 0:40:52.717197
2024-12-18 14:54:17.879 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:189 - nearest neighbor for speedmap_segments 2024-12-11: 0:08:09.784862
2025-01-21 13:27:30.117 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:188 - nearest neighbor for stop_segments 2025-01-15: 0:05:06.574214
2025-01-21 17:11:06.855 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:188 - nearest neighbor for stop_segments 2025-01-15: 0:04:14.577150
2025-01-21 18:22:23.308 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:188 - nearest neighbor for rt_stop_times 2025-01-15: 0:39:52.283432
2025-01-21 18:47:14.614 | INFO     | nearest_vp_to_stop:nearest_neighbor_for_stop:188 - nearest neighbor for speedmap_segments 2025-01-15: 0:08:43.206096

Why would something that took 15 min be 1 min?

2024-12-17 17:43:47.305 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:279 - interpolate arrivals for stop_segments 2024-12-11:  2024-12-11: 0:14:09.409333
2024-12-18 10:52:54.623 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for stop_segments 2024-12-11:  2024-12-11: 0:10:20.061296
2024-12-18 14:42:51.535 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for rt_stop_times 2024-12-11:  2024-12-11: 0:10:33.922215
2024-12-18 14:56:54.836 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for speedmap_segments 2024-12-11:  2024-12-11: 0:02:36.918147
2025-01-21 17:12:27.738 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for stop_segments 2025-01-15:  2025-01-15: 0:01:20.848747
2025-01-21 18:35:30.275 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for rt_stop_times 2025-01-15:  2025-01-15: 0:13:06.847024
2025-01-21 18:49:29.708 | INFO     | interpolate_stop_arrival:interpolate_stop_arrivals:236 - interpolate arrivals for speedmap_segments 2025-01-15:  2025-01-15: 0:02:15.045884

edasmalchi · 2025-01-22T22:43:28Z

Thanks for the pointers!

My thoughts my looking at the changes:

I think I saw vp_idx is commented out...I'm wondering how you're able to link back to the vp_usable_dwell table after? The nearest neighbor approach keys into that and uses it to link vp without carrying a mess of columns. And it takes varying forms (it can be an array in the intermediate steps, so leaving it off and trying to merge it in would not be equivalent)

The logs look suspiciously fast...even compared to prior months.

Hm, I did comment out the sort_cols argument since it's not part of vp_transform.condense_point_geom_to_line.

data-analyses/gtfs_funnel/vp_keep_usable.py

Lines 157 to 162 in d59e058

    
               vp_condensed = vp_transform.condense_point_geom_to_line( 
        
                   vp_gdf, 
        
                   group_cols = ["trip_instance_key"], 
        
           #        sort_cols = ["trip_instance_key", "vp_idx"], not used? 
        
                   array_cols = ["vp_idx", "geometry"]         
        
               )

data-analyses/rt_segment_speeds/segment_speed_utils/vp_transform.py

Lines 19 to 24 in 43da8a2

    
           def condense_point_geom_to_line( 
        
               df: gpd.GeoDataFrame, 
        
               group_cols: list, 
        
               geom_col: str = "geometry", 
        
               array_cols: list = [] 
        
           ) -> gpd.GeoDataFrame:

vp_idx was in both sort_cols and array_cols, so it seemed to me like it would still be expanded here:

data-analyses/rt_segment_speeds/segment_speed_utils/vp_transform.py

Line 71 in 43da8a2

**{c: lambda x: list(x) for c in array_cols},

It's not clear to me what the behavior of sort_cols was supposed to be or if it should be restored somehow.

Yeah, makes sense to look at the logs. Some sort of log checking/comparison step might be cool to make it somewhat automatic (perhaps print something if there's a major deviation like this...).

Ahh, ok, specifically the sorting was to guarantee that for each trip, timestamps should be increasing (sorting on trip_instance_key-vp_idx). I think I remembered to put the sorting ahead of time, but left the arg there to also set in case it wasn't a sorted gdf.

edasmalchi · 2025-01-22T23:51:29Z

I would compare the first 3 tables for Dec 2024 and Jan 2025 and look at the number of rows. They should be the same magnitude from month to month. 15M in vp_usable then 12M in vp_usable_dwell, and then vp_condensed_line should be the same number of rows as number of trip_instance_keys.
speeds_tables:
 raw_vp: vp
 usable_vp: vp_usable
 vp_dwell: vp_usable_dwell
 vp_condensed_line: condensed/vp_condensed

These all look about the same to me so far:

https://github.com/cal-itp/data-analyses/blob/jan-open-data/rt_segment_speeds/45_diff_tables.ipynb

tiffanychu90 · 2025-01-23T00:31:56Z

I would compare the first 3 tables for Dec 2024 and Jan 2025 and look at the number of rows. They should be the same magnitude from month to month. 15M in vp_usable then 12M in vp_usable_dwell, and then vp_condensed_line should be the same number of rows as number of trip_instance_keys.
speeds_tables:
 raw_vp: vp
 usable_vp: vp_usable
 vp_dwell: vp_usable_dwell
 vp_condensed_line: condensed/vp_condensed
These all look about the same to me so far:

https://github.com/cal-itp/data-analyses/blob/jan-open-data/rt_segment_speeds/45_diff_tables.ipynb

Then I would move further down into rt_segment_speeds portion to start debugging.

Since the logs for the times in nearest_stop_to_vp and interpolate_stop_arrival look similar to previous months for rt_stop_times/speedmap_segments, I would actually look next at the average_speedmap_segments script and compare the number of rows, shapes, route-dir, operators, everything, to see where you're getting a huge drop-off. The average_speedmap_segments starts with a trip-stop grain, so if all your trips are represented, then it's definitely something in the averaging portion.

If it's not there, then I'd move backwards from there into other staged parquets to see.

This reverts commit a76a6f4.

edasmalchi · 2025-02-12T20:20:52Z

Started looking back into this today -- average_segment_speeds seems to be OK. Looking backwards, there is a big difference in the stage4/trip speeds files:

for stop_segments, 2,835,752 rows / 78,872 trips in Dec and 191,833 rows / 66,769 trips in Jan.

for speedmap_segments, 3,348,200 rows / 79,571 trips in Dec and 3,074,234 rows / 79,946 trips in Jan.

So I'm inclined to believe the trouble lies somewhere within these steps, and that they seem to be functioning about the same as December for speedmap segments but not for stop segments.

Since both types of segments are concatenated to create the maps (with speedmap segments being the additional proxy stops), it makes sense that a ~90% drop in stop_segments trip speeds would lead to lots of missing data on the maps.

gs://calitp-analytics-data/data-analyses/rt_segment_speeds/speeds_stop_segments_2024-12-11.parquet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2835752 entries, 0 to 2835751
dtypes: category(1), datetime64[ns](1), float64(8), int64(2), object(9)
memory usage: 457.0+ MB
None
unique trips: 78872

gs://calitp-analytics-data/data-analyses/rt_segment_speeds/speeds_stop_segments_2025-01-15.parquet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'pandas.core.frame.DataFrame'>
Int64Index: 191834 entries, 0 to 191833
dtypes: category(1), datetime64[ns](1), float64(8), int64(2), object(9)
memory usage: 30.9+ MB
None
unique trips: 66769

gs://calitp-analytics-data/data-analyses/rt_segment_speeds/speedmap/speeds_2024-12-11.parquet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'pandas.core.frame.DataFrame'>
Int64Index: 3348200 entries, 0 to 3348199
dtypes: category(1), datetime64[ns](1), float64(9), int64(2), object(10)
memory usage: 590.7+ MB
None
unique trips: 79571

gs://calitp-analytics-data/data-analyses/rt_segment_speeds/speedmap/speeds_2025-01-15.parquet
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<class 'pandas.core.frame.DataFrame'>
Int64Index: 3074234 entries, 0 to 3074233 
dtypes: category(1), datetime64[ns](1), float64(9), int64(2), object(10)
memory usage: 542.4+ MB
None
unique trips: 79946

edasmalchi · 2025-02-13T00:08:02Z

Narrowed it down to this merge

The two tables join fine on ["st_trip_instance_key", "stop_id"], but not with the addition of "stop_pair" so that's where the difference lies.

stop_times = pd.merge(
        stops_to_use,
        shape_stop_combinations,
        on = ["st_trip_instance_key", "stop_id", "stop_pair"],
        how = "inner"
    ).drop(
        columns = "st_trip_instance_key"
    ).drop_duplicates().reset_index(drop=True)

edasmalchi · 2025-02-13T00:11:31Z

@tiffanychu90 more info here: https://github.com/cal-itp/data-analyses/blob/49daeaedc58b7287d7ee53511a4aa3fe9a481de9/rt_segment_speeds/48_investigate_further.ipynb

I'm going to switch my focus to https://github.com/cal-itp/data-analyses/blob/main/gtfs_funnel/stop_times_with_direction.py

edasmalchi · 2025-02-13T01:14:09Z

gtfs_funnel/stop_times_with_direction.py

think I've got it, will test the speeds pipeline tomorrow AM

edasmalchi · 2025-02-13T19:08:42Z

Wasn't able to run the entire pipeline before heading out this week, tried running things from the top but got some new errors on the new image. @vevetron if you could make a "legacy power user" that would probably be handy, that way I could focus on this round of debugging and handle any upgrade-induced changes separately...

make preprocess_vp
make[2]: Entering directory '/home/jovyan/data-analyses/gtfs_funnel'
python vp_keep_usable.py
2025-02-13 10:08:33.367 | INFO     | __main__:pare_down_to_valid_trips:88 - pare down vp: 0:02:19.071646
2025-02-13 at 10:08:33 | INFO | pare down vp: 0:02:19.071646
make[2]: *** [Makefile:15: preprocess_vp] Killed
make[2]: Leaving directory '/home/jovyan/data-analyses/gtfs_funnel'
make[1]: *** [Makefile:43: funnel_gtfs_single_day] Error 2
make[1]: Leaving directory '/home/jovyan/data-analyses/gtfs_funnel'
make: *** [Makefile:49: all] Error 2
jovyan@jupyter-edasmalchi ~/data-analyses/gtfs_funnel (jan-open-data) $

vevetron · 2025-02-13T19:12:05Z

@edasmalchi Got it i'll make a legacy power user as well.

edasmalchi requested a review from tiffanychu90 January 22, 2025 21:03

evansiroky mentioned this pull request Feb 4, 2025

Bug: City of Montebello (Montebello Bus Lines) not showing up in GTFS Digest #1357

Open

5 tasks

edasmalchi added 15 commits February 12, 2025 17:17

start funnel

7248d91

add nb

794625c

rename function call to match vp_tranform

a4f34cf

Revert "rename function call to match vp_tranform"

e670a9e

This reverts commit a76a6f4.

rename function call to match vp_tranform

590b06c

fix geopandas import

f60b395

comment out unused arg

e3ba1af

explode with pd method instead of pipe to missing function

d524ee4

finish funnel, speedmaps show missing shapes

bb2cbc2

clarify speedmaps change

7376280

re-add stop_meters to export for stage2

9cb4a39

finish running pipeline with fixes, reset makefiles

fcf7a13

add nb comparing Dec Jan speed tables

88dbf11

add nb comparing Dec Jan speed tables

b4b2fb3

revert speedmaps

521af62

edasmalchi force-pushed the jan-open-data branch from 1238973 to 521af62 Compare February 12, 2025 17:24

further investigation, find stop_pair issue

49daeae

debug stop_times_with_direction

8434467

edasmalchi commented Feb 13, 2025

View reviewed changes

gtfs_funnel/stop_times_with_direction.py

Copy link

Member Author

edasmalchi Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think I've got it, will test the speeds pipeline tomorrow AM

vevetron mentioned this pull request Feb 13, 2025

Added a power legacy user cal-itp/data-infra#3692

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jan open data -- draft for now #1356

Jan open data -- draft for now #1356

edasmalchi commented Jan 22, 2025

tiffanychu90 commented Jan 22, 2025 •

edited

Loading

edasmalchi commented Jan 22, 2025 •

edited by tiffanychu90

Loading

edasmalchi commented Jan 22, 2025

tiffanychu90 commented Jan 23, 2025

edasmalchi commented Feb 12, 2025 •

edited

Loading

edasmalchi commented Feb 13, 2025 •

edited

Loading

edasmalchi commented Feb 13, 2025 •

edited

Loading

edasmalchi Feb 13, 2025

edasmalchi commented Feb 13, 2025

vevetron commented Feb 13, 2025

Jan open data -- draft for now #1356

Are you sure you want to change the base?

Jan open data -- draft for now #1356

Conversation

edasmalchi commented Jan 22, 2025

tiffanychu90 commented Jan 22, 2025 • edited Loading

edasmalchi commented Jan 22, 2025 • edited by tiffanychu90 Loading

edasmalchi commented Jan 22, 2025

tiffanychu90 commented Jan 23, 2025

edasmalchi commented Feb 12, 2025 • edited Loading

edasmalchi commented Feb 13, 2025 • edited Loading

edasmalchi commented Feb 13, 2025 • edited Loading

edasmalchi Feb 13, 2025

Choose a reason for hiding this comment

edasmalchi commented Feb 13, 2025

vevetron commented Feb 13, 2025

tiffanychu90 commented Jan 22, 2025 •

edited

Loading

edasmalchi commented Jan 22, 2025 •

edited by tiffanychu90

Loading

edasmalchi commented Feb 12, 2025 •

edited

Loading

edasmalchi commented Feb 13, 2025 •

edited

Loading

edasmalchi commented Feb 13, 2025 •

edited

Loading