Add compact index support for private sources

[segiddins]

Description:

Ensured working via https://github.com/rubygems/gem_server_conformance

• Blocked by Update sqlite-jdbc to 3.46 jruby/activerecord-jdbc-adapter#1149
• 2 postgres integration specs failing on CI only

---

Tasks:

• Describe the problem / feature
• Write tests

I will abide by the code of conduct.

[indirect]

looking good!

I'm also interested to hear if this gave you any ideas about how we could improve the compact index gem for consumers, so their implementations could potentially be shorter/easier.

[technicalpickles]

Is there any extra configuration needed to enable this? Wanted to test locally, but see this when trying to bundle install against it:

HTTP GET http://localhost:9292/private/versions
HTTP 404 Not Found http://localhost:9292/private/versions
Bundler::Fetcher::FallbackError: Gem::Net::HTTPNotFound: http://localhost:9292/private/versions

[segiddins]

@technicalpickles shouldn't be, what's showing up in the server logs?

[technicalpickles]

Startup:

❯ bin/gemstash start                                                                                                                                                                      ─╯
Starting gemstash!
[2024-07-31 17:10:38 -0400] - INFO - [19490] Puma starting in cluster mode...
[2024-07-31 17:10:38 -0400] - INFO - [19490] * Puma version: 6.4.2 (ruby 3.3.2-p78) ("The Eagle of Durango")
[2024-07-31 17:10:38 -0400] - INFO - [19490] *  Min threads: 0
[2024-07-31 17:10:38 -0400] - INFO - [19490] *  Max threads: 16
[2024-07-31 17:10:38 -0400] - INFO - [19490] *  Environment: development
[2024-07-31 17:10:38 -0400] - INFO - [19490] *   Master PID: 19490
[2024-07-31 17:10:38 -0400] - INFO - [19490] *      Workers: 1
[2024-07-31 17:10:38 -0400] - INFO - [19490] *     Restarts: (✔) hot (✔) phased
[2024-07-31 17:10:38 -0400] - INFO - [19490] * Listening on http://0.0.0.0:9292
[2024-07-31 17:10:38 -0400] - INFO - [19490] Use Ctrl-C to stop
[2024-07-31 17:10:38 -0400] - INFO - [19490] ! WARNING: Detected running cluster mode with 1 worker.
[2024-07-31 17:10:38 -0400] - INFO - [19490] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
[2024-07-31 17:10:38 -0400] - INFO - [19490] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
[2024-07-31 17:10:38 -0400] - INFO - [19490] ! Set the `silence_single_worker_warning` option to silence this warning message.
/Users/josh.nichols/workspace/gemstash/lib/gemstash/web.rb:10: warning: Skipping set of ruby2_keywords flag for initialize (method accepts keywords or method does not accept argument splat)
[2024-07-31 17:10:38 -0400] - INFO - [19490] - Worker 0 (PID: 19492) booted in 0.36s, phase: 0

During bundle update bigdecimal --conservative:

[2024-07-31 17:10:47 -0400] - INFO - Rewriting '/private/versions' to '/versions'
[2024-07-31 17:10:47 -0400] - INFO - Rewriting '/private/api/v1/dependencies' to '/api/v1/dependencies'
[2024-07-31 17:10:47 -0400] - INFO - Rewriting '/private/api/v1/dependencies?gems=redacted-gem' to '/api/v1/dependencies?gems=redacted-gem'
[2024-07-31 17:10:47 -0400] - INFO - Querying dependencies: redacted-gem
[2024-07-31 17:10:47 -0400] - INFO - Rewriting '/private/api/v1/dependencies?gems=bigdecimal' to '/api/v1/dependencies?gems=bigdecimal'
[2024-07-31 17:10:47 -0400] - INFO - Querying dependencies: bigdecimal

[segiddins]

I'll revisit this once there's a version of sqlite for jruby that supports what I used here

[martinemde]

Hey @segiddins, the jdbc-sqlite3 seems like a long term blocker at this point. Is the performance greatly affected by not having string_agg? I'd be willing to fix it up to see if we can get this in if you don't have time, but I could use some context about that choice.

[segiddins]

It's the only way to get an array out of SQLite unfortunately. I wish we knew what jruby usage was like :/

[technicalpickles]

Would it feasible to disable compact index on jruby until the blocker is fixed? That way, non-jruby users would get the benefit. We could also keep a PR open to enable it on jruby, so we have a reminder and/or place to validate fixes.

[simi]

CI logs are not reachable anymore. Would you mind to rebase this and update deps? Seems there are some news at jruby/activerecord-jdbc-adapter#1149 (comment).

[martinemde]

When I was looking closely at JRuby compact index results, they were coming out ordered differently even with identical code (and even aggregated as a separate query with its own sort) so I suspect jdbc-sqlite3 is not behaving identically to cruby's adapters. The order might be because of time field precision differences since we sort by created_at. Just a hunch.