Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiline inserts, fix memory leak caused by query_id, fix stack trace when no live server #35

Merged
merged 8 commits into from
Nov 16, 2020

Conversation

splichy
Copy link
Contributor

@splichy splichy commented Nov 16, 2020

  • feature: support for multiline inserts (some CH drivers are sending FORMAT TabSeparated with multiple rows e.g. Java ExecuteBatch) so we have count number of rows in data
  • improvement: log in µs & log only insert statements - SELECTs are not that interesting and actual query isn't logged at all
  • improvement: enable debug with ENV var
  • fix: memory leak caused by query_id in query params - basicaly every query is unique -> new map[]Table for every query with query_id -> tables were never deleted. I have added two options: clean_interval - all tables which are not updated for clean_interval will be deleted; remove_query_id - will remove query_id=... from query params before create / insert to Table (in our load the mem usage was 800MB+ within ~3h, with remove_query_id is the mem usage stable ~45MB) this can probably solve High Memory Usage #23
  • fix: return 503 instead of stack trace when there is no live clickhouse endpoint

@nikepan
Copy link
Owner

nikepan commented Nov 16, 2020

Thank you for this big update!
can you fix travis fail? https://travis-ci.org/github/nikepan/clickhouse-bulk/jobs/717600328

@nikepan nikepan merged commit 2a6d515 into nikepan:master Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants