fix(wsl): Special handling Landscape client config tags #5460

CarlosNihelton · 2024-06-28T15:07:01Z

Proposed Commit Message

fix(wsl): Special handling Landscape client config tags

UP4W business logic is so that its data overrides user at a key (module) 
level.
That means the entire Landscape config is overriden if both agent data
and user data contains config for that module.
Yet, for better usability, computer tags must be assignable per instance.
That's not possible with agent.yaml, because it's meant to be global.
Its config data affects all Ubuntu WSL instances.

Thus this aims to make a special case for landscape.client.tags,
if present in user provided data (either Landscape or local user -
  whatever is picked up before merging with agent.yaml)
its value overwrites any tags set by agent.yaml.

Only landscape.client.tags are treated specially.
The pre-existing merge rules still apply for any other value present in
both agent.yaml and user provided data.

Fixes UDENG-2464

Additional Context

Test Steps

Fixes UDENG-2464

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

UP4W business logic is so that its data overrides user at a key (module) level. That means the entire Landscape config is overriden if both agent data and user data contains config for that module. Yet, for better usability, computer tags must be assignable per instance. That's not possible with agent.yaml, because it's meant to be global. Its config data affects all Ubuntu WSL instances. Thus this aims to make a special case for landscape.client.tags, if present in user provided data (either Landscape or local user - whatever is picked up before merging with agent.yaml) its value overwrites any tags set by agent.yaml. Only landscape.client.tags are treated specially. The pre-existing merge rules still apply for any other value present in both agent.yaml and user provided data.

Seems a cosmetic "fix", but consistency is good. `ubuntu_advantage` is deprecated, so we'd better not risk our test's future-proofing.

tests/unittests/sources/test_wsl.py

cloudinit/sources/DataSourceWSL.py

blackboxsw · 2024-07-04T12:56:09Z

cloudinit/sources/DataSourceWSL.py

        if user_data:
            merged = user_data
+            user_tags = (
+                merged.get("landscape", {}).get("client", {}).get("tags")


Our agent data overriding user-data policy in WSL leads us down this path where we may start defining multiple exceptions to the "rule". If we end up coming up with a second case were other user-data keys are preferred over agent data keys, we may want to define a separate userdata_override_paths dict or list or a separate function that extracts a dict of userdata_overrides from the original data so we have one place that defines these expected userdata paths.

Given that every computer needs a different landscape.client.computer_title would we expect to override this with user-data too?

Nope, Ubuntu Pro for WSL handles controls the computer title assignment (via the wsl-pro-service component running inside the WSL instances, but that work is transparent to cloud-init).

Great thanks!

doc/rtd/reference/datasources/wsl.rst

Co-authored-by: Chad Smith <[email protected]>

blackboxsw · 2024-07-18T20:32:48Z

cloudinit/sources/DataSourceWSL.py

+                LOG.debug(
+                    " Landscape client conf updated with user-data"
+                    " landscape.client.tags: %s",
+                    user_tags,
+                )
+                if merged.get("landscape", {}).get("client"):
+                    merged["landscape"]["client"]["tags"] = user_tags


Let's trim the leading whitespace of the log and move this debug statement into the scope of where we are actually merging landscape.client.tags.

Suggested change

LOG.debug(

" Landscape client conf updated with user-data"

" landscape.client.tags: %s",

user_tags,

)

if merged.get("landscape", {}).get("client"):

merged["landscape"]["client"]["tags"] = user_tags

if merged.get("landscape", {}).get("client"):

LOG.debug(

"Landscape client conf updated with user-data"

" landscape.client.tags: %s",

user_tags,

)

merged["landscape"]["client"]["tags"] = user_tags

blackboxsw · 2024-07-18T21:51:23Z

tests/unittests/sources/test_wsl.py

+            """#cloud-config
+landscape:
+  client:
+    account_name: user


This is invalid JSON schema for landscape as it also requires a computer_title. maybe good to have that represented.

I'm pushing updates with more realistic Landscape configs. Yet, we don't need to represent the computer title in the UP4W-related cases, as that piece of data is handled internally.

blackboxsw · 2024-07-18T21:52:21Z

cloudinit/sources/DataSourceWSL.py

        if user_data:
            merged = user_data
+            user_tags = (
+                merged.get("landscape", {}).get("client", {}).get("tags")


Given that every computer needs a different landscape.client.computer_title would we expect to override this with user-data too?

tests/unittests/sources/test_wsl.py

blackboxsw

Beyond the outstanding question on whether we need user-data overrides for landscape.client.computer_name as well I ran this test on Azure Windows Server and can verify correct behavior in various permutations of agent.yaml w/ and w/out landscape.client config and user-data w/ and w/out landscape config tags.

Just minor nits on when to log that merge data line and breaking up the last unit test into two separate unit tests

And trim spaces

Adding common fields we expect to see in real world usage. Computer title is certainly expected on local user setups, but in the case of UP4W it's handled internally, transparently from cloud-init.

Splitting the longer tests in more specific test cases. Split "missing tags" from "missing landscape.client subkey" tests. Remove some no-longer relevant assertions.

blackboxsw

minor test nits and mypy fix being pushed Will land it after CI passes.
Thanks @CarlosNihelton!

blackboxsw · 2024-07-19T19:10:21Z

cloudinit/sources/DataSourceWSL.py

+        # We've already checked, this is just to please mypy.
+        assert user_data is not None
+        user_tags = (
+            user_data.get("landscape", {}).get("client", {}).get("tags")
+        )


heh. I saw this and originally agreed, but this is a real issue mypy has for use. at this point agent_data OR user_data could be non-None, so there is still a possibility that user_data could be None here, which would raise an AssertionError and break cloud-init. Since we are actually setting this 3 lines below after our if user_data check, let's just initialize user_tags to None here and set it if present a couple lines below.

I'll apply this diff and approve after CI.

mypy: init user_tags to empty string, only set it when present in userdata diff --git a/cloudinit/sources/DataSourceWSL.py b/cloudinit/sources/DataSourceWSL.py index bd71cd431..7a75ff4e6 100644 --- a/cloudinit/sources/DataSourceWSL.py +++ b/cloudinit/sources/DataSourceWSL.py @@ -328,16 +328,12 @@ class DataSourceWSL(sources.DataSource): # provides them instead. # That's the reason for not using util.mergemanydict(). merged: dict = {} + user_tags: str = "" overridden_keys: typing.List[str] = [] - # We've already checked, this is just to please mypy. - assert user_data is not None - user_tags = ( - user_data.get("landscape", {}).get("client", {}).get("tags") - ) if user_data: merged = user_data user_tags = ( - merged.get("landscape", {}).get("client", {}).get("tags") + merged.get("landscape", {}).get("client", {}).get("tags", "") ) if agent_data: if user_data:

tests/unittests/sources/test_wsl.py

blackboxsw · 2024-07-19T19:34:27Z

tests/unittests/sources/test_wsl.py

+        assert (
+            "tag_aiml" in userdata and "tag_dev" in userdata
+        ), "User-data should override agent data's Landscape computer tags"


Let's strictly assert we don't see wsl as well from agent.yaml. Also note landscape client config schema disallows spaces between tags. so our example should be tags: x,y

Suggested change

assert (

"tag_aiml" in userdata and "tag_dev" in userdata

), "User-data should override agent data's Landscape computer tags"

assert (

"tags: tag_aiml,tag_dev" in userdata

), "User-data should override agent data's Landscape computer tags"

assert "wsl" not in userdata

CarlosNihelton · 2024-07-19T20:11:34Z

minor test nits and mypy fix being pushed Will land it after CI passes.
Thanks @CarlosNihelton!

Thanks for the collaboration as usual!

…rdata - tests: lanscape.client.tags schema disallows spaces between tags. Update tests to assert comma-delimited values and the exclusion of the wsl tag from agent.yaml

) UP4W business logic is so that its data overrides user at a key (module) level. That means the entire Landscape config is overriden if both agent data and user data contains config for that module. Yet, for better usability, computer tags must be assignable per instance. That's not possible with agent.yaml, because it's meant to be global. Its config data affects all Ubuntu WSL instances. Thus this aims to make a special case for landscape.client.tags, if present in user provided data (either Landscape or local user - whatever is picked up before merging with agent.yaml) its value overwrites any tags set by agent.yaml. Only landscape.client.tags are treated specially. The pre-existing merge rules still apply for any other value present in both agent.yaml and user provided data. Fixes UDENG-2464

UP4W business logic is so that its data overrides user at a key (module) level. That means the entire Landscape config is overriden if both agent data and user data contains config for that module. Yet, for better usability, computer tags must be assignable per instance. That's not possible with agent.yaml, because it's meant to be global. Its config data affects all Ubuntu WSL instances. Thus this aims to make a special case for landscape.client.tags, if present in user provided data (either Landscape or local user - whatever is picked up before merging with agent.yaml) its value overwrites any tags set by agent.yaml. Only landscape.client.tags are treated specially. The pre-existing merge rules still apply for any other value present in both agent.yaml and user provided data. Fixes UDENG-2464

CarlosNihelton added 3 commits June 28, 2024 12:08

[wsl] s/ubuntu_advantage/ubuntu_pro

e9906e8

Seems a cosmetic "fix", but consistency is good. `ubuntu_advantage` is deprecated, so we'd better not risk our test's future-proofing.

[wsl] Update docs about Landscape tags

975ffbe

CarlosNihelton force-pushed the wsl-landscape-tags branch from ff2e20b to 975ffbe Compare June 28, 2024 15:11

CarlosNihelton marked this pull request as ready for review June 28, 2024 15:27

blackboxsw self-assigned this Jul 2, 2024

blackboxsw requested changes Jul 4, 2024

View reviewed changes

blackboxsw reviewed Jul 4, 2024

View reviewed changes

doc/rtd/reference/datasources/wsl.rst Outdated Show resolved Hide resolved

CarlosNihelton force-pushed the wsl-landscape-tags branch from 1accda5 to 34f203d Compare July 11, 2024 02:54

CarlosNihelton and others added 3 commits July 11, 2024 00:04

Apply readability suggestions from code review

e292034

Co-authored-by: Chad Smith <[email protected]>

Splits the long test in smaller more scoped units

5e626d7

Please mypy about user_data possibly being None

51a4b73

CarlosNihelton force-pushed the wsl-landscape-tags branch from 34f203d to 51a4b73 Compare July 11, 2024 03:06

CarlosNihelton requested a review from blackboxsw July 11, 2024 03:22

blackboxsw reviewed Jul 18, 2024

View reviewed changes

blackboxsw reviewed Jul 19, 2024

View reviewed changes

CarlosNihelton added 3 commits July 19, 2024 10:25

Move to the right scope log about merging tags

51c9ea0

And trim spaces

Makes Landscape configs in tests more realistic

ced6cc5

Adding common fields we expect to see in real world usage. Computer title is certainly expected on local user setups, but in the case of UP4W it's handled internally, transparently from cloud-init.

More focused datasource tests

7c1a3a5

Splitting the longer tests in more specific test cases. Split "missing tags" from "missing landscape.client subkey" tests. Remove some no-longer relevant assertions.

blackboxsw approved these changes Jul 19, 2024

View reviewed changes

mypy: init user_tags to empty string, only set it when present in use…

7c95527

…rdata - tests: lanscape.client.tags schema disallows spaces between tags. Update tests to assert comma-delimited values and the exclusion of the wsl tag from agent.yaml

blackboxsw force-pushed the wsl-landscape-tags branch from ec2d07c to 7c95527 Compare July 19, 2024 21:57

blackboxsw merged commit 5532b4a into canonical:main Jul 19, 2024
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(wsl): Special handling Landscape client config tags #5460

fix(wsl): Special handling Landscape client config tags #5460

CarlosNihelton commented Jun 28, 2024 •

edited

Loading

blackboxsw Jul 4, 2024

blackboxsw Jul 18, 2024

CarlosNihelton Jul 19, 2024 •

edited

Loading

blackboxsw Jul 19, 2024

blackboxsw Jul 18, 2024

blackboxsw Jul 18, 2024

CarlosNihelton Jul 19, 2024

blackboxsw Jul 18, 2024

blackboxsw left a comment

blackboxsw left a comment

blackboxsw Jul 19, 2024

blackboxsw Jul 19, 2024

blackboxsw Jul 19, 2024

CarlosNihelton commented Jul 19, 2024

fix(wsl): Special handling Landscape client config tags #5460

fix(wsl): Special handling Landscape client config tags #5460

Conversation

CarlosNihelton commented Jun 28, 2024 • edited Loading

Proposed Commit Message

Additional Context

Test Steps

Merge type

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarlosNihelton Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blackboxsw left a comment

Choose a reason for hiding this comment

blackboxsw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarlosNihelton commented Jul 19, 2024

CarlosNihelton commented Jun 28, 2024 •

edited

Loading

CarlosNihelton Jul 19, 2024 •

edited

Loading