--fail-on-warnings does not fail on warnings

[jhoblitt]

$ bundle list metadata-json-lint
/home/jhoblitt/github/puppet-jenkins/.bundle/ruby/2.2.0/gems/metadata-json-lint-1.0.0
$ bundle exec metadata-json-lint --fail-on-warnings
Warning: Dependency camptocamp/systemd has an open ended dependency version requirement >= 0.3.0
$ echo $?
0

[rnelson0]

@jhoblitt Can you provide the metadata.json so we can add a sample test for this?

[rnelson0]

Currently open dependencies only fail on --strict_dependencies. Likewise, license warnings only raise to an error when --strict-licenses is used. Adding an || options[:fail_on_warnings] will certainly raise these warnings to errors. However, it also is essentially a backwards incompatible change as it changes the current default results, even if it doesn't change the default settings being used.

This will need some discussion. I see a few options:

• Raise all warnings to errors when fail-on-warnings is enabled. Backwards incompatible
• Change the license and dependency warnings to notices. Preserves existing defaults and functionality
• Leave things they way they are
• ???

[jhoblitt]

@rnelson0 the metadata.json with the warnings is https://github.com/jenkinsci/puppet-jenkins/blob/beaa841665e8cb265d3a8a285270028844c76922/metadata.json

I was a bit surprised that it had warnings that weren't bubbling up as fatal and obviously didn't expect that --fail-on-warnings doesn't actually make all warnings fatal. I suppose we could just add the strict_dependencies flag to puppetlabs_spec_helper, which would solve this for most users, but I feel that the semantics of --fail-on-warnings should be fixed with a major version bump.

[vinzent]

If providing the fail-on-warnings option i would really expect to fail on all warnings.

[cdenneen]

@jhoblitt can you see if #78 fixes this issue as well as #76