Multiple regex-strings in DocumentCleaner?

[danielsaaf]

Hi!

I didn't know where i should post this, but will start as a general question (if you think this is more suitable somewhere else, please let me know!).

I am using the DocumentCleaner (https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_cleaner.py) and have a few different regex i would like to apply to clean up my documents.
However, the DocumentCleaner is only supporting a single string as input to the regex variable.
(Compare with remove_substrings which allows a list of strings as input.)

class DocumentCleaner:

    def __init__(  # pylint: disable=too-many-positional-arguments
        self,
        remove_empty_lines: bool = True,
        remove_extra_whitespaces: bool = True,
        remove_repeated_substrings: bool = False,
        keep_id: bool = False,
        remove_substrings: Optional[List[str]] = None,         <- String
        remove_regex: Optional[str] = None,                          <- Regex
        unicode_normalization: Optional[Literal["NFC", "NFKC", "NFD", "NFKD"]] = None,
        ascii_only: bool = False,
    ):

Is there anything i am missing? To me this looks inconsistent, but guess that someone might have thought about this..

[julian-risch]

Hi @danielsaaf Thanks for your question! We have an issue scheduled for our current sprint to extend the DocumentCleaner so this is helpful feedback! For the remove_regex parameter, I would say that if you have a few different regular expressions that you would like to apply, you can always combine them into a single regex with OR condition |.

[danielsaaf]

Thank you for your response!

Ah, cool, looks like some improvements are on it's way!

Didn't know about the OR condition in regex - guess that that might be a doable solution. (I you stay with the current interface, i guess that one way of make life easier would be to mention this in the documents or so.. (for regex-newbies like me ;;)