Drop reimplementation of Ripper lex state

[nevans]

This code was for ruby 2.4 compatibility, but rdoc dropped support for ruby 2.4 about three years ago, in f480b97. This code was almost half of the lines of code in rdoc/parser/ripper_state_lex.

[st0012]

This is great, and I think we can go a bit further to remove other unused stuff:

diff --git a/lib/rdoc/parser/ripper_state_lex.rb b/lib/rdoc/parser/ripper_state_lex.rb
index 928f2011..80de0d0e 100644
--- a/lib/rdoc/parser/ripper_state_lex.rb
+++ b/lib/rdoc/parser/ripper_state_lex.rb
@@ -9,24 +9,11 @@ class RDoc::Parser::RipperStateLex
 
   Token = Struct.new(:line_no, :char_no, :kind, :text, :state)
 
-  EXPR_NONE = 0
-  EXPR_BEG = 1
   EXPR_END = 2
-  EXPR_ENDARG = 4
   EXPR_ENDFN = 8
   EXPR_ARG = 16
-  EXPR_CMDARG = 32
-  EXPR_MID = 64
   EXPR_FNAME = 128
-  EXPR_DOT = 256
-  EXPR_CLASS = 512
   EXPR_LABEL = 1024
-  EXPR_LABELED = 2048
-  EXPR_FITEM = 4096
-  EXPR_VALUE = EXPR_BEG
-  EXPR_BEG_ANY  =  (EXPR_BEG | EXPR_MID | EXPR_CLASS)
-  EXPR_ARG_ANY  =  (EXPR_ARG | EXPR_CMDARG)
-  EXPR_END_ANY  =  (EXPR_END | EXPR_ENDARG | EXPR_ENDFN)
 
   class InnerStateLex < Ripper::Filter
     def initialize(code)
@@ -53,7 +40,7 @@ class RDoc::Parser::RipperStateLex
     when :on_backtick then
       if (tk[:state] & (EXPR_FNAME | EXPR_ENDFN)) != 0
         tk[:kind] = :on_ident
-        tk[:state] = Ripper::Lexer.const_defined?(:State) ? Ripper::Lexer::State.new(EXPR_ARG) : EXPR_ARG
+        tk[:state] = Ripper::Lexer::State.new(EXPR_ARG)
       else
         tk = get_string_tk(tk)
       end
@@ -266,7 +253,7 @@ class RDoc::Parser::RipperStateLex
   private def get_op_tk(tk)
     redefinable_operators = %w[! != !~ % & * ** + +@ - -@ / < << <= <=> == === =~ > >= >> [] []= ^ ` | ~]
     if redefinable_operators.include?(tk[:text]) and tk[:state] == EXPR_ARG then
-      tk[:state] = Ripper::Lexer.const_defined?(:State) ? Ripper::Lexer::State.new(EXPR_ARG) : EXPR_ARG
+      tk[:state] = Ripper::Lexer::State.new(EXPR_ARG)
       tk[:kind] = :on_ident
     elsif tk[:text] =~ /^[-+]$/ then
       tk_ahead = get_squashed_tk

Ripper::Lexer::State exists in Ruby 2.5 already, and the current RDoc requires Ruby 2.6 so it's safe to assume it's defined.

[nevans]

Ah, good point. Yeah. I haven't previously done more than tinker with Ripper and I'm only just starting to familiarize myself with how this part of rdoc is working, so I hadn't looked at it deeply enough to find those. I'll add your patch to the PR when I get a moment.

[nevans]

I was actually only in this code to see if I could figure out how much work #885 would be... and I was reading through this old code for quite a while before I got to the unless RIPPER_HAS_LEX_STATE at the bottom... I was thinking to myself, wow, is rdoc re-implementing ruby's entire lex state algorithm inside its parser?! Only when I got to the bottom of that class did I realize, yes, that's exactly what it was doing (as a polyfill). Thus this PR, to save future code readers some time. 😉

[martinemde]

The code looks right to me from a non-maintainer's perspective. I support merging this as long as the maintainers also approve and checks pass. Maybe we should merge the Ruby-core test workflow #1115 first so the rest of these can be revalidated under that environment.

[simi]

LGTM!

[st0012]

Thanks for the cleanup!