Running "charm build" on a limited-IPv6 system is very slow due to connection failures

[barryprice]

Checklist

• Confirmed this is an issue with charm-tools, not charmstore-client
• Provide versions of tools used
• Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm and charm-tools
publisher: Canonical✓
store-url: https://snapcraft.io/charm
contact:   https://discourse.juju.is/c/charming
license:   unset
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     latest/stable
refresh-date: 2021-06-25
channels:
  latest/stable:    2.8.2               2021-02-01 (609) 119MB classic
  latest/candidate: 2.8.2               2021-02-01 (609) 119MB classic
  latest/beta:      2.8.2               2021-02-01 (609) 119MB classic
  latest/edge:      2.8.3+git-1-736b1ad 2021-02-23 (620) 119MB classic
installed:          2.8.2                          (609) 119MB classic

I am using: Ubuntu 22.04 on amd64

Issue/Feature

I expect/expected the following

Running charm build on a system with full IPv4 but limited IPv6 connectivity results in a very long wait with no output, even with --debug set on the command line.

Edited to add: I have no native ISP support for IPv6, but my work VPN gives me limited IPv6 connectivity to work networks - but not to the wider IPv6 internet. The problem only appears when connected to that VPN. So this is much more of an edge case than I'd realised when first filing this.

What I got

$ charm --debug build
build: The lockfile /home/ubuntu/charms/ntp-charm/build.lock was not found; building using latest versions.
build: Destination charm directory: /home/ubuntu/charms/builds/ntp
[...very long wait - 15 mins so far, at time of writing...]

Using strace shows what's actually happening - the tooling is attempting to connect to juju.github.io, but is trying each of its four AAAA records in turn before falling back to its A records, with what seems to be a very long timeout.

As a first step, I think it's reasonable for --debug to output what site the tooling is trying to reach.

Ideally, it would be smart enough to figure out early whether a working IPv6 link to the target is available, and prefer IPv4 if not.

[ajkavanagh]

I'm curious, but what happens if you just curl https://juju.github.io/layer-index/ on the same box?

charm build uses requests which just uses the underlying python libs (urlib3, etc) to fetch the layer indexes; it's not setting up preferences for fetching via ipv6 or ipv4; the DNS to the box is saying ipv6 AAAA records exist and (I'm guessing ipv6 is enabled on the build-box) so requests/urllib3 tries to use them first.

It may be that ipv6 is legitimately enabled, but not routable, and thus a -4 type option might be required to force the networking stack for charm-tools to only use ipv4.

[barryprice]

I do have a limited IPv6 VPN setup but it's only to route specific work-related networks, so I can't disable it globally unfortunately. I'd ideally like to have native IPv6 but my ISP just doesn't offer it yet.

Running curl is much faster, I'd speculate that it somehow figures out very quickly that IPv6 won't fly here:

$ time curl -o /dev/null https://juju.github.io/layer-index/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 99557  100 99557    0     0   226k      0 --:--:-- --:--:-- --:--:--  226k

real	0m0.442s
user	0m0.057s
sys	0m0.005s

[ajkavanagh]

Oh, I feared as much; doing a quick web search yields things like: https://discuss.dizzycoding.com/force-requests-to-use-ipv4-ipv6/

i.e. it's a problem.

Possible solutions are a -4 flag to force ipv4 or reducing the timeout on the request when waiting for a page to appear; I guess what is a bit weird is that the request to the ipv6 address on the box hangs for so long; it's like the packet out is dropped rather than the stack reporting back no route; I wonder if it's a bug in the python libs?

[ajkavanagh]

Relevant urllib3 bug possibly: urllib3/urllib3#2055

And, almost certainly: urllib3/urllib3#2030

And, finally, this would actually solve it if it were implmented: Implement Happy Eyeballs to support IPv4/IPv6 combination: urllib3/urllib3#797

Which begs the question: what to do?

[barryprice]

Yeah it's not a straightforward one for sure.

I do think displaying e.g. "connecting to x.y.z..." output lines to charm when --debug is set would make the reason for any network-related delays/errors clearer.

A -4 flag might be worthwhile until those upstream bugs are resolved - when it's set, the tooling could do something like:

requests.get("https://juju.github.io/layer-index/", family=socket.AF_INET) to force IPv4 (it woudl be AF_INET6 to force IPv6).

[barryprice]

For the record, did a re-build today and remembered to wrap it in time:

barryprice@x1c9:~/charms/ntp-charm{lp1982517}$ time make build
Running flake8
lint installed: flake8==4.0.1,mccabe==0.6.1,pycodestyle==2.8.0,pyflakes==2.4.0
lint run-test-pre: PYTHONHASHSEED='2579139414'
lint run-test: commands[0] | flake8
_____________________________________________________________ summary _____________________________________________________________
  lint: commands succeeded
  congratulations :)
charm build
build: The lockfile /home/barryprice/Documents/Canonical/charms/ntp-charm/build.lock was not found; building using latest versions.
build: Destination charm directory: /home/barryprice/charms/builds/ntp
build: Processing layer: layer:options
build: Processing layer: layer:basic
build: Processing layer: layer:nagios
build: Processing layer: layer:ntpmon
build: Processing layer: ntp (from .)
build: Processing interface: juju-info
build: Processing interface: nrpe-external-master
build: 
build: ---------------------------------------
build:               Build Report
build: ---------------------------------------
build:  + .coverage
build:    templates/chrony.conf
build:    version
proof: I: `display-name` not provided, add for custom naming in the UI

real	57m21.509s
user	0m16.104s
sys	0m1.707s
barryprice@x1c9:~/charms/ntp-charm{lp1982517}$

Then on a hunch, deactivated my work VPN (which gives me a limited-use IPv6 address), and retried - so in this state, I have a link-local IPv6 address only:

barryprice@x1c9:~/charms/ntp-charm{lp1982517}$ time make build
Running flake8
lint installed: flake8==4.0.1,mccabe==0.6.1,pycodestyle==2.8.0,pyflakes==2.4.0
lint run-test-pre: PYTHONHASHSEED='1056145244'
lint run-test: commands[0] | flake8
_____________________________________________________________ summary _____________________________________________________________
  lint: commands succeeded
  congratulations :)
charm build
build: The lockfile /home/barryprice/Documents/Canonical/charms/ntp-charm/build.lock was not found; building using latest versions.
build: Destination charm directory: /home/barryprice/charms/builds/ntp
build: Processing layer: layer:options
build: Processing layer: layer:basic
build: Processing layer: layer:nagios
build: Processing layer: layer:ntpmon
build: Processing layer: ntp (from .)
build: Processing interface: juju-info
build: Processing interface: nrpe-external-master
build: 
build: ---------------------------------------
build:               Build Report
build: ---------------------------------------
build: New build; all files were modified.
proof: I: `display-name` not provided, add for custom naming in the UI

real	0m40.423s
user	0m15.135s
sys	0m1.497s
barryprice@x1c9:~/charms/ntp-charm{lp1982517}$

So I have a slightly inconvenient but absolutely worthwhile workaround for now.

[hloeung]

#609 might also be related as well - adding timeouts to the requests calls may reduce the overall time you're seeing (that is with attempts to fetch assets via IPv6 timing out in a shorter period of time).