Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add out of the box performance timing mechanism #163

Open
colmsnowplow opened this issue Jul 31, 2020 · 0 comments
Open

Add out of the box performance timing mechanism #163

colmsnowplow opened this issue Jul 31, 2020 · 0 comments

Comments

@colmsnowplow
Copy link
Contributor

colmsnowplow commented Jul 31, 2020

SQL-runner logs each step to stdout with timestamps, which is awesome. When doing gnarly debugging, or just longer-term logging of performance, it's potentially not ideal however, because if the query is in the Queue, or the connection is laggy, these timings will appear to indicate poor performance.

Feels like we can easily create a boilerplate way to log start and end timestamps to a table in the actual database for both a playbook and the steps of a playbook. For example, this is what I'm using for the model I'm currently working on (redshift):


-- At start of whole model:
CREATE TABLE IF NOT EXISTS {{.output_schema}}.base_timings (
  run_id TIMESTAMP,
  test_id VARCHAR(64),
  step_name VARCHAR(64),
  start_tstamp TIMESTAMP,
  end_tstamp TIMESTAMP
);

DROP TABLE IF EXISTS {{.scratch_schema}}.step_timing;

CREATE TABLE {{.scratch_schema}}.step_timing (
  test_id VARCHAR(64),
  step_name VARCHAR(64),
  start_tstamp TIMESTAMP,
  end_tstamp TIMESTAMP);

-- At every step:
INSERT INTO {{.scratch_schema}}.step_timing VALUES(
  '{{.entropy}}', '01-new-events-limits', GETDATE(), NULL
);


UPDATE {{.scratch_schema}}.step_timing SET end_tstamp = GETDATE() WHERE test_id = '{{.entropy}}' AND step_name = '01-new-events-limits';

-- At end of whole model:


INSERT INTO {{.output_schema}}.base_timings (SELECT GETDATE(), * FROM {{.scratch_schema}}.step_timing);

Could be a nice feature if this was available as a configuration to sql-runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant