Fix for The bundle currently has logstash-xxx-yyy locked at N when doing offline plugin install

[guyboertje]

At present the locking feature in Bundler is preventing version changes resulting in the unhelpful message.

Installing file: /tmp/logstash-test/logstash-offline-plugins-6.1.1.zip
ERROR: An error occured when installing the: file:///tmp/logstash-test/logstash-offline-plugins-6.1.1.zip, to have more information about the error add a DEBUG=1 before running the command., message: You have requested:
  logstash-codec-plain = 3.0.6

The bundle currently has logstash-codec-plain locked at 3.0.5.
Try running `bundle update logstash-codec-plain`

See #8979

Why exclude build and development gems from the offline pack? I think that if these gems are somehow included in the released artefact then they should not be added to the offline pack, maybe?

[jsvd]

@guyboertje should I wait here for the tests before reviewing?

[guyboertje]

@jsvd I suppose so.

[guyboertje]

Two integration tests fail because the user_agent source yaml was changed in ua-parser/uap-core@6318cae

[guyboertje]

Added #10017

[jsvd]

Why exclude build and development gems from the offline pack? I think that if these gems are somehow included in the released artefact then they should not be added to the offline pack, maybe?

sorry, I'm trying to understand how excluding the gems from the build and development groups in the package solves this problem. In the original issue the report comes from bundling many plugins and getting an error on "logstash-codec-plain" being locked, but that one isn't in either of those groups in plugins like the http input.

[guyboertje]

@jsvd
My understanding is that in cases like the kinesis input some plugins have other plugins for tests and are a development dependency but these get included in the offline pack zip but they do not need to be transferred to the offline machine.

[jsvd]

I think I'm starting to understand, so the PR fixes the issue by doing two things:

1. Unlocks the gemfile by passing true instead of {} to the definition
2. Excludes :build and :development gems from the Gemfile during the upgrade

So 1. is pretty clear that it needs to happen: we need to unlock the Gemfile because we want to upgrade gems and those plugins may have runtime dependencies used by other plugins.

However, it seems that when we unlock the Gemfile it performs extra validation on it, and it understands that there are a bunch of gems listed in the Gemfile that don't ship with logstash artifacts:

logstash-6.1.1 % DEBUG=1 bin/logstash-plugin install file:///tmp/test_offline/online/logstash-6.1.1/logstash-offline-plugins-6.1.1.zip
DEBUG: exec /tmp/test_offline/offline/logstash-6.1.1/vendor/jruby/bin/jruby /tmp/test_offline/offline/logstash-6.1.1/lib/pluginmanager/main.rb install file:///tmp/test_offline/online/logstash-6.1.1/logstash-offline-plugins-6.1.1.zip
Local file: /tmp/test_offline/online/logstash-6.1.1/logstash-offline-plugins-6.1.1.zip
Installing with strategy: LogStash::PluginManager::PackInstaller::Local
Installing file: /tmp/test_offline/online/logstash-6.1.1/logstash-offline-plugins-6.1.1.zip
Pack uncompressed to /var/folders/d7/v4056t5d03dfck1vgkrblfzm0000gn/T/studtmp-765ffdbf0a0ac4f67182f2d2ab00dcc7fa34713061e8433387f8c80b415c
Installing, logstash-input-http, version: 3.2.2 file: /var/folders/d7/v4056t5d03dfck1vgkrblfzm0000gn/T/studtmp-765ffdbf0a0ac4f67182f2d2ab00dcc7fa34713061e8433387f8c80b415c/logstash/logstash-input-http-3.2.2-java.gem
Installing, logstash-codec-plain, version: 3.0.6 file: /var/folders/d7/v4056t5d03dfck1vgkrblfzm0000gn/T/studtmp-765ffdbf0a0ac4f67182f2d2ab00dcc7fa34713061e8433387f8c80b415c/logstash/dependencies/logstash-codec-plain-3.0.6.gem
Bundler::GemNotFound: Could not find gem 'ci_reporter_rspec (= 1.0.0) java' in any of the gem sources listed in your Gemfile or installed on this machine.
  block in verify_gemfile_dependencies_are_found! at /tmp/test_offline/offline/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/bundler-1.9.10/lib/bundler/resolver.rb:328

Filtering out those gems from the Gemfile before doing the update avoid falling into this situation.

So it seems that combination of these two actions does solve the issue. My question here is: instead of skipping those gems during update, what if we just made sure that a release ships with a cleaner Gemfile that doesn't include those groups?

[guyboertje]

I'm good with assuring that the Gemfile is build with runtime dependencies only.

Do you know what changes to make to achieve that assurance?

[lucabelluccini]

@jsvd this is blocked?

[logstashmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?