Fetcher approach is too narrow

[ryan-beisner]

There is a notion of "fetchers" in charm-tools, which might seem to allow for a wide range of approaches (source code repos). However, in moving one project from Github.org to Opendev.org, we end up with a No fetcher or url error.

Perhaps there should be a generic https fetcher?

https://bugs.launchpad.net/charm-barbican/+bug/1846263

FETCHERS = [
    BzrFetcher,
    BzrMergeProposalFetcher,
    GithubFetcher,
    BitbucketFetcher,
    LocalFetcher,
    CharmstoreDownloader,
    BundleDownloader,
    LaunchpadGitFetcher,
    GitFetcher,
]

charmtools.build.fetchers: Copying /home/beisner/.cache/charm/17767/interface/tmp9c_1z0hu to /home/beisner/.cache/charm/17767/interface/openstack-ha                                     
charmtools.build.fetchers: Cleaning up /home/beisner/.cache/charm/17767/interface/tmp9c_1z0hu                                                                                            
charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/interfaces/barbican-hsm.json                                                                         
charmtools.build.fetchers: Found repo: https://opendev.org/x/charm-interface-barbican-hsm
build: Traceback (most recent call last):
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 941, in main                                           
    build()
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 649, in __call__                                       
    self.generate()
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 592, in generate                                       
    layers = self.fetch()
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 273, in fetch                                          
    return self.fetch_deps(self.top_layer)
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 278, in fetch_deps                                     
    self.fetch_dep(layer, results)
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 307, in fetch_dep                                      
    results["interfaces"].append(iface.fetch())
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/builder.py", line 100, in fetch                                          
    self.directory = path(fetcher.fetch(self.target_repo))
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/fetchers.py", line 150, in fetch                                         
    f, target = self._get_repo_fetcher_and_target(self.repo, dir_)
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/build/fetchers.py", line 143, in _get_repo_fetcher_and_target                  
    f = get_fetcher(repo)
  File "/home/beisner/git/release-tools.round6/barbican/.tox/build/lib/python3.7/site-packages/charmtools/fetchers.py", line 375, in get_fetcher                                         
    raise FetchError('No fetcher for url: %s' % url)
charmtools.fetchers.FetchError: No fetcher for url: https://opendev.org/x/charm-interface-barbican-hsm                                                                                   

build: No fetcher for url: https://opendev.org/x/charm-interface-barbican-hsm
ERROR: InvocationError for command '/home/beisner/git/release-tools.round6/barbican/.tox/build/bin/charm-build --log-level DEBUG -o /home/beisner/git/release-tools.round6/barbican/build src' (exited with code 1)

[johnsca]

As mentioned in the PR, at the very least, the existing git fetcher classes should be refactored to inherit from GitFetcher to clean up duplicated code. Perhaps we could also do something more sophisticated with the URL matcher; it would be good for a charm to be able to specify that a particular URL is a git repo via some manner of config rather than having to extend the code.