[BUG]: free(): invalid pointer

[whybeyoung]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

Like issue #1472,
we still have problem in 2.10.0

free(): invalid pointer

c++ code:

#include "pybind11/embed.h"
#include <iostream>
#include <thread>
#include <chrono>
#include <sstream>

namespace py = pybind11;
using namespace std::chrono_literals;

class Wrapper
{
public:
  Wrapper()
  {
    py::gil_scoped_acquire acquire;
    _obj = py::module::import("wrapper").attr("Wrapper")();
    _wrapperInit = _obj.attr("wrapperInit");
    _wrapperFini = _obj.attr("wrapperFini");

  }

  ~Wrapper()
  {
    _wrapperInit.release();
    _wrapperFini.release();
  }

  int wrapperInit()
  {
    py::gil_scoped_acquire acquire;
    return _wrapperInit(nullptr).cast<int>();
  }

  void wrapperFini(int x)
  {
    py::gil_scoped_acquire acquire;
    _wrapperFini(x);
  }

  private:
  py::object _obj;
  py::object _wrapperInit;
  py::object _wrapperFini;
};
void thread_func(int iteration)
{
  Wrapper w;

  for (int i = 0; i < 1; i++)
  {
    w.wrapperInit();
    std::stringstream msg;
    msg << "iteration: " << iteration << " thread: " << std::this_thread::get_id() << std::endl;
    std::cout << msg.str();
        std::this_thread::sleep_for(100ms);
  }
}

int main() {
  py::scoped_interpreter guard{};
  py::gil_scoped_release release; // add this to release the GIL

  std::vector<std::thread> threads;

  for (int i = 0; i < 1; ++i)
    threads.push_back(std::thread(thread_func, 1));

  for (auto& t : threads)
    t.join();

  return 0;
}

wrapper.py code is

class Wrapper():
    serviceId = "mmocr"
    version = "backup.0"


    '''
    服务初始化
    @param config:
        插件初始化需要的一些配置，字典类型
        key: 配置名
        value: 配置的值
    @return
        ret: 错误码。无错误时返回0
    '''

    def wrapperInit(cls, config: {}) -> int:
        import torch
        print(config)

        print("Initializing ..")
        return 0

    def wrapperFini(cls) -> int:
        return 0

we run this code in ubuntu18.04 docker container. and the repo is public.ecr.aws/iflytek-open/opensource/demo/mmocr:v3.1

Reproducible example code

No response

[henryiii]

I'm guessing this is #4105.

[henryiii]

I verified this is not #4105, this code was broken in 2.9 as well.

[rwgk]

I couldn't reproduce the free(): invalid pointer crash using the code here, but there is certainly a GIL issue that you can confirm by using PR #4146. The problem in the reproducer code is that the GIL is not being held when the destructor for Wrapper::_obj is running. You can "fix" it by adding _obj.release(); in the Wrapper destructor. "fix" is in quotation marks because it is simply leaking the Python reference, "masking" would be a more fitting word. To not leak:

--- main_using_embed_h.cpp.orig 2022-10-23 21:29:46.559375849 -0700
+++ main_using_embed_h.cpp      2022-10-23 21:56:25.089334464 -0700
@@ -21,7 +21,12 @@

   ~Wrapper()
   {
+    py::gil_scoped_acquire hold_gil;
+    _obj.dec_ref();
+    _obj.release();
+    _wrapperInit.dec_ref();
     _wrapperInit.release();
+    _wrapperFini.dec_ref();
     _wrapperFini.release();
   }

I'm closing this bug because it's pretty likely that the free(): invalid pointer has nothing to do with a bug in pybind11.

Until we merge PR #4146, I recommend you patch it locally and run all your tests.

[Davidnet]

I am encountering this with the same conditon this is my set-up that can be replicated

# dummy_python_script.py
import torch

def simple_return():
    
    return 1

the simple.cpp

#include <iostream>
#include <future>
#include <pybind11/embed.h>

namespace py = pybind11;

std::future<int> callPythonFunctionAsync(py::object &pyFunction) {
    return std::async(std::launch::async, [&](){
        py::gil_scoped_acquire acquire;
        int result = pyFunction().cast<int>();
        return result;
    });
}

int main() {
    py::scoped_interpreter guard{}; // Start the interpreter and keep it alive

    // Import the Python module
    py::module pyModule = py::module::import("dummy");
    py::object pyFunction = pyModule.attr("simple_return");

    // Call the function asynchronously
    std::cout << "Calling Python function asynchronously..." << std::endl;
    py::gil_scoped_release release;
    auto futureResult = callPythonFunctionAsync(pyFunction);

    // Wait for the result and print it
    try {
        int result = futureResult.get();
        std::cout << "Result from Python: " << result << std::endl;
    } catch (const std::exception& e) {
        std::cerr << "Exception caught: " << e.what() << std::endl;
    }

    return 0;
}

with the following cmake

cmake_minimum_required(VERSION 3.10)  # Updated minimum required version
project(py_cpp_func)

set(CMAKE_CXX_STANDARD 11)  # Setting C++ standard to C++11
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")

# Manually set Python include directories and libraries
set(PYTHON_INCLUDE_DIR /usr/local/include/python3.10)
set(PYTHON_LIBRARY /usr/local/lib/libpython3.10.so)
include_directories(${PYTHON_INCLUDE_DIR})

# Include pybind11
# Include pybind11 from the external directory
add_subdirectory(external/pybind11)
add_executable(py_dummy simple.cpp)
target_link_libraries(py_dummy PRIVATE ${PYTHON_LIBRARIES} pybind11::embed)
configure_file(dummy.py ${CMAKE_BINARY_DIR}/dummy.py COPYONLY)

with the following dockerfile:

FROM ubuntu:18.04

RUN apt-get update && \
    apt-get install -y software-properties-common && \
    add-apt-repository ppa:ubuntu-toolchain-r/test && \
    apt-get update && \
    apt-get install -y \
    gcc \
    g++ \
    cmake \
    libboost-all-dev \
    wget

RUN apt-get remove -y cmake && \
    wget https://cmake.org/files/v3.10/cmake-3.10.0-Linux-x86_64.sh && \
    chmod +x cmake-3.10.0-Linux-x86_64.sh && \
    ./cmake-3.10.0-Linux-x86_64.sh --skip-license --prefix=/usr/local

RUN apt-get install -y git

# Clone pybind11 into the external directory
RUN mkdir -p /external && \
    git clone --branch v2.11.1 https://github.com/pybind/pybind11.git /external/pybind11

# Install Python 3.10.13
ENV PYTHON_VERSION 3.10.13

# Install necessary packages
RUN apt-get update && \
    apt-get install -y software-properties-common wget git \
    build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev \
    libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev liblzma-dev
RUN apt-get install -y libgomp1 libgl1-mesa-glx

# Download Python 3.10 source
RUN cd /tmp && \
    wget https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tar.xz && \
    tar -xf Python-$PYTHON_VERSION.tar.xz

# Compile Python 3.10
RUN cd /tmp/Python-$PYTHON_VERSION && \
    ./configure --enable-optimizations --enable-shared && \
    make -j 8 && \
    make altinstall && \
    ldconfig

# Install pip for Python 3.10
RUN cd /tmp && \
    wget https://bootstrap.pypa.io/get-pip.py && \
    python3.10 get-pip.py && \
    rm get-pip.py

# Install OpenCV for C++
RUN DEBIAN_FRONTEND="noninteractive" apt-get install -y libopencv-dev

WORKDIR /usr/src/three-stage-object-detection
# Install Triton Inference Server
COPY three-stage-object-detection /usr/src/three-stage-object-detection/
RUN python3.10 -m pip install -e .

WORKDIR /usr/src/app
COPY CMakeLists.txt /usr/src/app/
COPY dummy.py /usr/src/app/
COPY simple.cpp /usr/src/app/
RUN mkdir external && \
    ln -s /external/pybind11 external/pybind11
RUN mkdir build && \
    cd build && \
    cmake -DCMAKE_BUILD_TYPE=Debug .. && \
    make

WORKDIR /usr/src/app/build

# Clean up
RUN apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

[rwgk]

I am encountering this with the same conditon this is my set-up that can be replicated

• Does this run successfully if you remove import torch?
• Do you have a stack trace from the crash?
• I don't think that's it, but I'd make this change:

-std::future<int> callPythonFunctionAsync(py::object &pyFunction)
+std::future<int> callPythonFunctionAsync(py::handle pyFunction)

• I don't think any of the maintainers will have the time to reproduce the crash. If this is important to you, I recommend you send a PR that adds a .github/workflows/reproducer.yml job to run in GitHub Actions.

• I really really doubt the root cause is in pybind11.

[Davidnet]

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff6c7f7f1 in __GI_abort () at abort.c:79
#2  0x00007ffff6cc8837 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff6df5a7b "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff6ccf8ba in malloc_printerr (str=str@entry=0x7ffff6df3c76 "free(): invalid pointer") at malloc.c:5342
#4  0x00007ffff6cd6dec in _int_free (have_lock=0, p=0x7fff280e49a8, av=0x7ffff702ac40 <main_arena>) at malloc.c:4167
#5  __GI___libc_free (mem=0x7fff280e49b8) at malloc.c:3134
#6  0x000055555542c508 in __gnu_cxx::new_allocator<std::_Fwd_list_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (this=0x5555556dae78, __p=0x7fff280e6158) at /usr/include/c++/7/ext/new_allocator.h:140
#7  0x000055555542876b in std::allocator_traits<std::allocator<std::_Fwd_list_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::destroy<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., __p=0x7fff280e6158) at /usr/include/c++/7/bits/alloc_traits.h:487
#8  0x000055555542319d in std::_Fwd_list_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_erase_after (this=0x5555556dae78, __pos=0x5555556dae78, __last=0x0) at /usr/include/c++/7/bits/forward_list.tcc:90
#9  0x000055555541e84a in std::_Fwd_list_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~_Fwd_list_base (this=0x5555556dae78, __in_chrg=<optimized out>) at /usr/include/c++/7/bits/forward_list.h:329
#10 0x000055555541a82c in std::forward_list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::~forward_list (this=0x5555556dae78, __in_chrg=<optimized out>) at /usr/include/c++/7/bits/forward_list.h:559
#11 0x000055555540fb3b in pybind11::detail::internals::~internals (this=0x5555556dacd0, __in_chrg=<optimized out>) at /external/pybind11/include/pybind11/detail/internals.h:207
#12 0x0000555555419629 in pybind11::finalize_interpreter () at /external/pybind11/include/pybind11/embed.h:263
#13 0x00005555554196ea in pybind11::scoped_interpreter::~scoped_interpreter (this=0x7fffffffe533, __in_chrg=<optimized out>) at /external/pybind11/include/pybind11/embed.h:308
#14 0x0000555555407d2d in main () at /usr/src/app/simple.cpp:16

I got this backtrace also I was able to run if I update to 20.04 on the docker base image.

someone on gitter helped me to get the trace

[Davidnet]

I am encountering this with the same conditon this is my set-up that can be replicated

• Does this run successfully if you remove import torch?
• Do you have a stack trace from the crash?
• I don't think that's it, but I'd make this change:

-std::future<int> callPythonFunctionAsync(py::object &pyFunction)
+std::future<int> callPythonFunctionAsync(py::handle pyFunction)

• I don't think any of the maintainers will have the time to reproduce the crash. If this is important to you, I recommend you send a PR that adds a .github/workflows/reproducer.yml job to run in GitHub Actions.
• I really really doubt the root cause is in pybind11.

if I do not put torch, the code works, so definitly something with torch

[rwgk]

if I do not put torch, the code works, so definitly something with torch

I'd work on sending them a PR that reproduces the crash.