What if I moved my website's AppService to another Resource Group AFTER having added a certificate on it?

[mirandaasm]

Symptons: AppService certificate is not being renewed.

Failure: UpdateSiteBinding fails to update bindings corretly. However, new certificates are being issued during the Renewal proccess. They are just not bound to the expected AppService.

When it started to happen: after moving the problematic AppService to a new Resource Group. Yes, we did this stupid move. :(

So, it seems that the Durable Tasks used by the bot are still using some Site data from before the move. We have a few options, but we don't know how to proceed with any of them, except for the last one:

1. "Kill" the bot data for this AppService instance, so we can "Add Certificate" from scratch to this AppService, considering its current (new) Resource Group, as if it was a new one. Not sure if this is possible. If so, how could we do it?

2. "Refresh" the bot data for this AppService instance, so we can tell the bot about this "Resource Group" change. It seems even more complicated to me, but who knows? Is it possible? How?

3. Not what we wanted but, we can always do: "Backup AppService => Delete Problematic AppService => Create New AppService => Restore Backup => Add Certificate to the new AppService". We know this will work, we just wanted to avoid this path. :)

So, can you point us in any directions about the options (1) or (2) above? If not, that's ok. We will take the "long way" from option (3).

Thanks in advance.

[shibayan]

Can you share UpdateSiteBinding error information? You may mask the resource group name and so on.

[mirandaasm]

Sure thing man.

Actually, digging deeper into the error information, it seems there's nothing to do with the "Resource Group" change. It seems something related with a custom User-Assigned Identity on the AppService itself. Either way, I'm not sure why...

What I can say is: the custom User-Assigned Identity on the AppService is not really required by the AppService, it's not even being used at all. So I could safely remove it from the problematic AppService. Now I'm waiting the next bot execution to see if removing this custom User-Assigned Identity from the AppService will fix the problem.

I'm also sharing with you the UpdateSiteBinding error information which hinted me about the existence of this weirdo custom User-Assigned Identity on my AppSerivce (I didn't even know it was there, lol).

Thanks in advance man, great work on the bot BTW! 😄 👍

---

Exception while executing function: RenewCertificates_SubOrchestrator Orchestrator function 'RenewCertificates_SubOrchestrator' failed: The activity function 'UpdateSiteBinding' failed: "Failed to deserialize exception from TaskActivity: {"$type":"Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException, Microsoft.Azure.Management.Websites","Request":{"$type":"Microsoft.Rest.HttpRequestMessageWrapper, Microsoft.Rest.ClientRuntime","Method":{"$type":"System.Net.Http.HttpMethod, System.Net.Http","Method":"PUT"},"RequestUri":"https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/RG-xxxxxxxxxxxxx/providers/Microsoft.Web/sites/my-app-svc?api-version=2020-06-01","Properties":{"$type":"System.Collections.Generic.Dictionary``2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]], System.Private.CoreLib"},"Content":"{\r\n \"properties\": {\r\n \"enabled\": true,\r\n \"hostNameSslStates\": [\r\n {\r\n \"name\": \"adomain.com.br\",\r\n \"sslState\": \"SniEnabled\",\r\n \"thumbprint\": \"A0D15XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3210E\",\r\n \"toUpdate\": true,\r\n \"hostType\": \"Standard\"\r\n },\r\n {\r\n \"name\": \"my-app-svc.azurewebsites.net\",\r\n \"sslState\": \"Disabled\",\r\n \"hostType\": \"Standard\"\r\n },\r\n {\r\n \"name\": \"anotherdomain.com.br\",\r\n \"sslState\": \"SniEnabled\",\r\n \"thumbprint\": \"A0D15XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3210E\",\r\n \"toUpdate\": true,\r\n \"hostType\": \"Standard\"\r\n },\r\n {\r\n \"name\": \"www.adomain.com.br\",\r\n \"sslState\": \"SniEnabled\",\r\n \"thumbprint\": \"A0D15XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3210E\",\r\n \"toUpdate\": true,\r\n \"hostType\": \"Standard\"\r\n },\r\n {\r\n \"name\": \"www.anotherdomain.com.br\",\r\n \"sslState\": \"SniEnabled\",\r\n \"thumbprint\": \"A0D15XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX3210E\",\r\n \"toUpdate\": true,\r\n \"hostType\": \"Standard\"\r\n },\r\n {\r\n \"name\": \"my-app-svc.scm.azurewebsites.net\",\r\n \"sslState\": \"Disabled\",\r\n \"hostType\": \"Repository\"\r\n }\r\n ],\r\n \"serverFarmId\": \"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/DiffResGrp/providers/Microsoft.Web/serverfarms/My-Awesome-ServicePlan\",\r\n \"reserved\": false,\r\n \"isXenon\": false,\r\n \"hyperV\": false,\r\n \"siteConfig\": {},\r\n \"scmSiteAlsoStopped\": false,\r\n \"clientAffinityEnabled\": false,\r\n \"clientCertEnabled\": false,\r\n \"clientCertMode\": \"Required\",\r\n \"hostNamesDisabled\": false,\r\n \"customDomainVerificationId\": \"02789XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXD1588\",\r\n \"containerSize\": 0,\r\n \"dailyMemoryTimeQuota\": 0,\r\n \"httpsOnly\": true,\r\n \"redundancyMode\": \"None\"\r\n },\r\n \"identity\": {\r\n \"type\": \"SystemAssigned, UserAssigned\",\r\n \"userAssignedIdentities\": {\r\n \"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/Some-Random-ResGrp/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-userassigned-identity\": {}\r\n }\r\n },\r\n \"kind\": \"app\",\r\n \"location\": \"Brazil South\",\r\n \"tags\": {\r\n \"Business Unit\": \"AAAAAAAAAAA\",\r\n \"Customer\": \"BBBBBBB\",\r\n \"Environment\": \"CCCCCCCCCC\",\r\n \"hidden-related:diagnostics/changeAnalysisScanEnabled\": \"true\"\r\n }\r\n}","Headers":{"$type":"System.Collections.Generic.Dictionary``2[[System.String, System.Private.CoreLib],[System.Collections.Generic.IEnumerable``1[[System.String, System.Private.CoreLib]], System.Private.CoreLib]], System.Private.CoreLib","x-ms-client-request-id":["75ee4cb5-1630-41ce-8b8f-028c91f77443"],"Accept-Language":["en-US"],"Authorization":["Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"],"User-Agent":["FxVersion/4.700.21.11102","OSName/Windows","OSVersion/Microsoft.Windows.10.0.14393","Microsoft.Azure.Management.WebSites.WebSiteManagementClient/3.1.0"],"Request-Context":["appId=cid-v1:115731f0-83eb-478b-8418-f74164d5cb10"],"Request-Id":["|a321db80d3a89e46bd9f543132c4d232.7411d47435c30748."],"traceparent":["00-a321db80d3a89e46bd9f543132c4d232-7411d47435c30748-00"],"Content-Type":["application/json; charset=utf-8"],"Content-Length":["2398"]}},"Response":{"$type":"Microsoft.Rest.HttpResponseMessageWrapper, Microsoft.Rest.ClientRuntime","StatusCode":403,"ReasonPhrase":"Forbidden","Content":"{\"error\":{\"code\":\"LinkedAuthorizationFailed\",\"message\":\"The client 'e0ezzzzz-zzzz-zzzz-zzzz-zzzzzzzzzdd2' with object id 'e0ezzzzz-zzzz-zzzz-zzzz-zzzzzzzzzdd2' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/RG-xxxxxxxxxxxxx/providers/Microsoft.Web/sites/my-app-svc'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/Some-Random-ResGrp/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-userassigned-identity' or the linked scope(s) are invalid.\"}}","Headers":{"$type":"System.Collections.Generic.Dictionary``2[[System.String, System.Private.CoreLib],[System.Collections.Generic.IEnumerable``1[[System.String, System.Private.CoreLib]], System.Private.CoreLib]], System.Private.CoreLib","Cache-Control":["no-cache"],"Pragma":["no-cache"],"x-ms-failure-cause":["gateway"],"x-ms-request-id":["22fa76b8-a78c-4383-8613-a0b56215a028"],"x-ms-correlation-request-id":["22fa76b8-a78c-4383-8613-a0b56215a028"],"x-ms-routing-request-id":["BRAZILSOUTH:20210416T000113Z:22fa76b8-a78c-4383-8613-a0b56215a028"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains"],"X-Content-Type-Options":["nosniff"],"Date":["Fri, 16 Apr 2021 00:01:12 GMT"],"Connection":["close"],"Content-Type":["application/json; charset=utf-8"],"Expires":["-1"],"Content-Length":["708"]}},"Body":{"$type":"Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponse, Microsoft.Azure.Management.Websites","error":{"$type":"Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseError, Microsoft.Azure.Management.Websites","code":"LinkedAuthorizationFailed","message":"The client 'e0ezzzzz-zzzz-zzzz-zzzz-zzzzzzzzzdd2' with object id 'e0ezzzzz-zzzz-zzzz-zzzz-zzzzzzzzzdd2' has permission to perform action 'Microsoft.Web/sites/write' on scope '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/RG-xxxxxxxxxxxxx/providers/Microsoft.Web/sites/my-app-svc'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/Some-Random-ResGrp/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-userassigned-identity' or the linked scope(s) are invalid.","target":null,"details":null,"innererror":null}},"StackTrace":" at Microsoft.Azure.Management.WebSites.WebAppsOperations.BeginCreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String name, Site siteEnvelope, Dictionary``2 customHeaders, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Management.WebSites.WebAppsOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String name, Site siteEnvelope, Dictionary``2 customHeaders, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.CreateOrUpdateAsync(IWebAppsOperations operations, String resourceGroupName, String name, Site siteEnvelope, CancellationToken cancellationToken)\r\n at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker``2.InvokeAsync(TReflected instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs:line 20\r\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker``2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\r\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 555\r\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 501\r\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 279","Message":"Operation returned an invalid status code 'Forbidden'","Data":{"$type":"System.Collections.ListDictionaryInternal, System.Private.CoreLib"},"InnerException":null,"HelpLink":null,"Source":"Microsoft.Azure.Management.Websites","HResult":-2146233088}". See the function execution logs for additional details.

[shibayan]

Perhaps the IAM configuration needed to be modified due to moving the resource group.

The resource group that contains the Web App needs to have the Website Contributor permission, and the resource group that contains the App Service Plan needs to have the Web Plan Contributor permission.

If the App Service Plan associated with the App Service exists in a separate resource group, you should assign a Website Contributor to the resource group where the App Service exists, and a Web Plan Contributor to the resource group where the App Service Plan exists.

Be careful if the Web App and App Service Plan are separate resource groups.

[mirandaasm]

Yes, they are different Resource Groups, indeed.

But since we have an exclusive Subscription for such App Services, and the bot should be able to manage every and any apps on this Subscription, we've set the required IAM configuration at the Subscription level (both Website Contributor and Web Plan Contributor), which should be inherited by all of the Resource Groups inside this Subscription.

We can confirm that this setup actually works, as we don't need to configure IAM for every new Resource Group we create inside this pre-configured Subscription, and still, all the certificates are being correctly generated and renewed for multiple App Services, on multiple different Resource Groups, including the ones we've created recently.

Only a handfull of App Services are failing to renew its certificates, and we thought it was related to the "Resource Group move" we did in the past. But looking closer to the error stack trace I shared above, it looks like there's nothing to do with this so called Resource Group move, but with a custom User-Assigned Identity which is assigned to the App Service. Every failing App Service has this custom User-Assigned Identity on it, but none of them actually use or require it. And, funny enough, the majority of our App Services (which are actually working fine), doesn't have this custom User-Assigned Identity on them.

Anyway, like I said in my previous reply, I've removed this custom User-Assigned Identity from the problematic App Service. The bot should try to renew the problematic App Service certificate again by tomorrow. We're waiting to see if removing this custom User-Assigned Identity from the App will fix the problem.

---

[leppie]

I have been struggling with a similar issue for almost a year now.

Thanks to @mirandaasm pointing out the specific place I needed to look for the error.

Ended up being related to my app services that have a VNET subnet assigned to them.

Solution was to add Network Contributor role to the function app, and everything is working again now!