[Feature]: Add retry logic to all Az cmdlets

[o-l-a-v]

Description of the new feature

When having PowerShell scripts running every 10 minutes 24x7 you get a lot of strange errors. Sometimes the backend API endpoint times out and does not respond, but will respond if you try again.

• Some resource providers seems less stable than others. In my experience Microsoft.OperationalInsights is more problematic than others, as in API (and thus Az cmdlets) more often fails.
• I've had this happen to something as simple as Get-AzResource -ResourceId

In important scripts I've had to create my own retry logic, but it would be even better if Az cmdlets got some built-in retry logic instead. Like what Invoke-RestMethod currently have with parameters -MaximumRetryCount and -RetryIntervalSec. Or set some global settings for current session that all Az cmdlets will adhere to.

If it can't be done with all Az cmdlets, at least add retry logic to Invoke-AzRestMethod?

Proposed implementation details (optional)

No response

[isra-fel]

Thanks for reporting.
Currently we have built-in retry for 429 Too Many Requests, but not for other error codes. We will discuss this internally.

[gabbsmo]

Another case is when a resource is not found. For example, create a SQL Server and a database directly after. The database provisioning because the server is not found, yet. If there was automatic retry for this case scripting would be a lot more straight forward.

[o-l-a-v]

17th of May we had three automation account runbook jobs crashing on Connect-AzAccount -Identity with $_.Exception.Message = A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (management.azure.com:443)". Even with our own retry logic trying it twice before crashing the script.

I don't know what can be done from module perspective to make this more resillient, it's looks to me like a backend issue. But if something can be done it'd be much appreciated.

For now we've updated the logic to try 5 times before crashing the script.