Modify SPI flash driver to be compatible with stacks in PSRAM (IDFGH-14813)

[jonsmirl]

Is your feature request related to a problem?

This originally came up on esp-matter. espressif/esp-matter#1249

It would really be useful if the spi_flash driver had a menu config option telling it to switch onto it's own dedicated stack in internal RAM before doing anything.

Use case is a system running Lua. Lua uses a LOT of stack while running, too much to be in internal memory for larger Lua programs. Lua allocates all of its local variables on the stack. To get a lot of room it is easy to give Lua a PSRAM based stack, but then none of the flash calls work any more - like the file system. This is not so simple to fix in the application. Everything which touches flash has to be fixed not to use PSRAM stacks. All calls to NVM, file systems, OTA, etc... Much easier if the spi_flash driver just switched onto it's own stack when called.

Describe the solution you'd like.

Menuconfig option in the spi_flash driver telling it to use a private stack for internal operations. Alternatively, check the stack pointer in the spi_flash component and if it is in PSRAM switch stacks automatically.

Describe alternatives you've considered.

Fixing this problem at every call site in the user app. This is a lot of work and error prone.

Additional context.

No response

[BitsForPeople]

Sounds like you'd want to create a dedicated "flash-write" FreeRTOS task, with stack in interal RAM, to which you can send 'commands' from other tasks.

[jonsmirl]

I know that technique, but now consider Lua (Lua is a scripting engine). I have to search all over the Lua engine, locate everywhere it can use flash (like all file system calls, NVM, OTA, etc) and then make another task and figure out how to send all of the data over to it, get the answers and errors back and maintain task synchronization. But I didn't write the Lua engine and I don't maintain it.

Doing the stack swapping in the spi_flash component localizes the stack swapping to a single piece of code. It also fixes this issue for everyone, not just me.

[BitsForPeople]

I'd argue that it makes more sense to adapt the Lua engine to the ESP32 than to adapt the IDF to the Lua engine.

[jonsmirl]

Why should I have to be on a specific stack type before calling the spi_flash module? Lua is just one example of this, anything which needs large stacks would hit it. You can hit it from LVGL too if you enable file system support and use a PSRAM stack.

I am continuously running out of internal RAM and my own code only uses 8KB of it. I am always fighting with all of the various components to get THEM to stop using internal RAM. Right now I have a 80KB Lua stack in internal RAM that I can't move to PSRAM.