PHP 8.4 | Add tokenization of asymmetric visibility

[DanielEScherzer]

Description

Set up tokenization of asymmetric visibility modifiers:

• Add tokens T_PUBLIC_SET, T_PROTECTED_SET, and T_PRIVATE_SET if not already defined
• Consider them scope modifiers (PHP_CodeSniffer\Util\Tokens::$scopeModifiers)
• Mark them as context sensitive (PHP_CodeSniffer\Util\Tokens::$contextSensitiveKeywords)
• Store their known lengths (PHP_CodeSniffer\Tokenizers\PHP::$knownLengths)
• Backfill tokenization for older versions of PHP

Suggested changelog entry

PHP 8.4 | Support tokenization of asymmetric visibility modifiers

Related issues/external references

Related to #851

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
  • This change is only breaking for integrators, not for external standards or end-users.
• Documentation improvement

PR checklist

• I have checked there is no other PR open for the same change.
• I have read the Contribution Guidelines.
• I grant the project the right to include and distribute the code under the BSD-3-Clause license (and I have the right to grant these rights).
• I have added tests to cover my changes.
• I have verified that the code complies with the projects coding standards.
• [Required for new sniffs] I have added XML documentation for the sniff.

[jrfnl]

@DanielEScherzer Awesome work on this !

I have left a few comments and questions in-line, but generally speaking, this is looking great!

One additional question: how did you determine where in the (huge) PHP::tokenize() method, the polyfill should go ?
It is currently between the tokenization blocks for T_ENUM_CASE and PHP 8.0 namespaced names handling.
Is there a particular reason for that ?

[jrfnl]

Suggested change

	&& $tokens[($stackPtr + 2)][0] === T_STRING
	&& is_array($tokens[($stackPtr + 2)]) === true
	&& $tokens[($stackPtr + 2)][0] === T_STRING

[jrfnl]

Not sure this is correct... Should these "keywords" ever be changed into T_STRING depending on their context.... ?

[jrfnl]

Looks like this is missing some tests to safeguard that invalid code is not affected by this polyfill.

Some suggestions, but feel free to get creative:

// Not valid as asym visibility in PHP 8.4.
public(unset) string $pubx;
public ( set ) string $pubx;
protected/*comment*/ ( set ) string $pubx;
Private{set} string $pubx;
public() string $pubx;
function public(Set $setter) {}

// Live coding test. Needs to be in a file of its own or at the end of the file without a new line after it (and this should be annotated as such to prevent someone adding that new line).
public(set

[jrfnl]

Have you done any tests how this change will affect pre-existing sniffs which use this token array ?

And if it affects pre-existing sniffs, should those be updated in the same PR to not leave those sniffs in a broken state ?