Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Long time to generate web pages #4

Open
pbitzer opened this issue Aug 20, 2020 · 3 comments
Open

Long time to generate web pages #4

pbitzer opened this issue Aug 20, 2020 · 3 comments
Milestone
v0.4.0

Comments

@pbitzer
Copy link
Contributor

pbitzer commented Aug 20, 2020

Depending on the logging frequency and the number log files, it can take a long time (~10 minutes) for Sindri to generate the website.
@CAM-Gerlach
Copy link
Member

CAM-Gerlach commented Aug 20, 2020 •
edited
Loading

Mitigations to implement (from low-hanging fruit to more complex tuning)

  • Add customizable limit to look back when rebuilding the overall data, and the CSVs the first run
  • If not the first run, only rebuild CSVs that have been changed since last run
  • Only update plot data, archive table, etc every n runs (default 5, i.,e. every 5 minutes)
  • More efficiently filter old data filer earlier and more aggressively?

@CAM-Gerlach CAM-Gerlach added this to the v0.2.x milestone Aug 20, 2020
@CAM-Gerlach
Copy link
Member

Right now, for 0.2.9 we can at least implement the first point, its basically making what you already did into a customizable setting

pbitzer added a commit to pbitzer/sindri that referenced this issue Aug 25, 2020
@pbitzer
Only look back to the last 30 days for data on website generation. Pa…
ade35b0 
…rtially address project-mjolnir#4
pbitzer added a commit to pbitzer/sindri that referenced this issue Aug 25, 2020
@pbitzer
Decrease the website update interval time. Partially addresses projec…
4b75af5 
…t-mjolnir#4
@pbitzer pbitzer mentioned this issue Aug 25, 2020
Merged
CAM-Gerlach pushed a commit to pbitzer/sindri that referenced this issue Oct 22, 2020
@pbitzer @CAM-Gerlach
Only look back to the last 30 days for data on website generation. Pa…
b77ca38 
…rtially address project-mjolnir#4
CAM-Gerlach pushed a commit to pbitzer/sindri that referenced this issue Oct 22, 2020
@pbitzer @CAM-Gerlach
Decrease the website update interval time. Partially addresses projec…
bd5ec94 
…t-mjolnir#4
pbitzer added a commit to pbitzer/sindri that referenced this issue Sep 2, 2021
@pbitzer
Merge branch 'web_page_generation' of https://github.com/pbitzer/sindri…
e7fb9e7 
… into web_page_generation

* 'web_page_generation' of https://github.com/pbitzer/sindri:
  Decrease the website update interval time. Partially addresses project-mjolnir#4
  Only look back to the last 30 days for data on website generation. Partially address project-mjolnir#4
@CAM-Gerlach
Copy link
Member

Part of this was implemented with #9 ; moving generation to the server in v0.3.0 should improve things further.

@CAM-Gerlach CAM-Gerlach modified the milestones: v0.2.11, v0.3.0, v0.4.0 May 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.4.0
Development

No branches or pull requests
2 participants
@pbitzer @CAM-Gerlach