Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
zq update through "pcap index: Compress offsets that exceed threshold…
…" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR brimdata/super#1096, authored by @mattnibs, has been merged. pcap index: Compress offsets that exceed threshold Introduce ranger.Envelope.Merge that merges two Envelopes into a single Envelope. This fixes bug where indexing a large pcap causes the system to oom panic. When constructing the time index for a pcap, compress the array of offset points to an Envelope when the size of the array reaches a certain threshold. Subsequent compressions will be merged into the section's Envelope keeping the memory footprint low. The downside to this approach is for the indexes of large pcap files the difference between adjacent X values starts out very wide then narrows as one iterate through the Bins. This will result in larger pcap scans (i.e. slow searches) for hits at the beginning of the file and smaller scans (i.e. faster searches) towards the end. Consensus was that the difference in search times probably won't be noticeable enough to warrant introducing a fancier algorithm. Filed brimsec/zq#1095 to revisit. Closes brimdata/super#1039
- Loading branch information