io_uring_prep_sendmsg (readv/writev) atomicity guarantees

[gmelikov]

tl;dr What's the guarantee about io_uring_prep_sendmsg atomicity?

I try to make async network (TCP stream socket) protocol with "iodepth" (client and server are sequential and have only one thread, but they may generate more than 1 "async" request, you can think about it as a storage protocol, maybe like nvmeotcp), so there may be X commands incoming, and I wanted to answer to them via io_uring_prep_sendmsg, one by one, with some header and payload. In this case order is not critical, but atomicity is - each response header should have corresponding data with it and be whole.

I use

	struct msghdr msg = {
		.msg_iov = ioreq->iovecs, // My "header + data" is here in different iovecs
		.msg_iovlen = 3};

	struct io_uring_sqe *sqe = get_sqe(&ring); // Basic wrapper on io_uring_get_sqe
	io_uring_prep_sendmsg(sqe, fd, &msg, MSG_WAITALL | MSG_NOSIGNAL);
	io_uring_sqe_set_data(sqe, ioreq);
	io_uring_submit(&ring);

, where I send data in one uring SQE and submit it before any manipulations with buffer. BUT when I submitted more than 22 such inflight SQEs with 4+KB of data each, on 1500MTU physical network between client<->server I've got some racy mixed up data from several sendmsgs intersected. I see that CQEs come in different order and that's ok, but send data atomicity is a problem. Maybe it's a question about MSG_WAITALL.

I can't reproduce it if server and client are on the same machine even with 64 "parallel" inflight io_uring_prep_sendmsg SQEs.
Just wanted to check my sanity, thank you.