Set CacheOptions in ctrl.Manager to only cache secrets with certain labels to prevent caching all secrets

[randmonkey]

Problem Statement

As discussed in #1250 (comment), we need to configure CacheOption in Manager to configure the constraints of cached objects. We want to limit the caching of Secret because there are too many of them in the cluster, and most of them are not relevant.

The constraint can reduce the CPU/memory consumption of the controllers as well as Kubernetes API server load. But it needs users to set certain labels on the Secret if they want to use them somewhere (like certificates or configuration). We need to discuss whether should we add a GLOBAL constraint to watched Secrets. It will be a breaking change if we are already watching Secrets somewhere.

Proposed Solution

Consider making labels for all Secrets (and ConfigMaps) to be reconciled by KGO required (it could be a konghq.com/credential label existence requirement). It can be configured in cache.Options on the controller-runtime Manager level. Most likely this change would be breaking as we may not require all Secrets to be labeled as of now (but this is to be verified).

A non-breaking alternative would be to add a default off flag (e.g. --watch-only-labeled-secrets) that would change this config when someone specifically asks for it (e.g. because they experience performance issues).

Acceptance Criteria

• KGO only watches/caches labeled Secrets