Skip to content

Commit

Permalink
zq update through "pcap index: Compress offsets that exceed threshold…
Browse files Browse the repository at this point in the history
…" by mattnibs

This is an auto-generated commit with a zq dependency update. The zq PR
brimdata/super#1096, authored by @mattnibs,
has been merged.

pcap index: Compress offsets that exceed threshold

Introduce ranger.Envelope.Merge that merges two Envelopes into a
single Envelope.

This fixes bug where indexing a large pcap causes the system to
oom panic.

When constructing the time index for a pcap, compress the array of
offset points to an Envelope when the size of the array reaches a
certain threshold. Subsequent compressions will be merged into the
section's Envelope keeping the memory footprint low.

The downside to this approach is for the indexes of large pcap files
the difference between adjacent X values starts out very wide then
narrows as one iterate through the Bins. This will result in larger
pcap scans (i.e. slow searches) for hits at the beginning of the file
and smaller scans (i.e. faster searches) towards the end. Consensus
was that the difference in search times probably won't be noticeable
enough to warrant introducing a fancier algorithm. Filed brimsec/zq#1095 to
revisit.

Closes brimdata/super#1039
  • Loading branch information
Brim Automation committed Aug 14, 2020
1 parent 2e370ab commit a6349a5
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@
"styled-components": "^5.1.1",
"valid-url": "^1.0.9",
"zealot": "./zealot",
"zq": "git+https://github.com/brimsec/zq.git#0a84938e970a453a25c4c3462d0ff86aa11aad12"
"zq": "git+https://github.com/brimsec/zq.git#c46c0b55abed4562130b108390c0ded9a7820022"
},
"optionalDependencies": {
"electron-installer-debian": "^3.0.0",
Expand Down

0 comments on commit a6349a5

Please sign in to comment.