ggml : fix quantized cpy op

[ggerganov]

ref #12253

This should fix CPY(Q8_0, Q8_0)

[aviallon]

I no longer have garbled output with quantized cache. Only repetitions when reaching context-size, depending on the batch-size and the number of slots.
Tested-by: Antoine Viallon <[email protected]>

[jukofyork]

Is there any chance we could add the copy operations for BF16? Even just BF16 <--> F32 would be enough to test it for the KV-cache types.

[ggerganov]

@jukofyork bc25236 should cover BF16 <-> F32 copies.

[slaren]

This change does not look right to me. If i00 and i10 represent blocks now, then the logic for determining when to move to the next row in if (++i10 == ne0) { i10 = 0; .. does not seem correct, since i10 is a block index, and ne0 is the number of elements. Renaming the variables so that it is clear if they are element of block indices should make the code easier to understand.

llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c

Lines 4166 to 4187 in 938c779

	for (int64_t i01 = ir0; i01 < ir1; i01++) {
	for (int64_t i00 = 0; i00 < nb; i00++) {
	const char * src0_ptr = ((char *) src0->data + i00*nb00 + i01*nb01 + i02*nb02 + i03*nb03);
	char * dst_ptr = ((char *) dst->data + i10*nb0 + i11*nb1 + i12*nb2 + i13*nb3);
	memcpy(dst_ptr, src0_ptr, type_size);
	if (++i10 == ne0) {
	i10 = 0;
	if (++i11 == ne1) {
	i11 = 0;
	if (++i12 == ne2) {
	i12 = 0;
	if (++i13 == ne3) {
	i13 = 0;
	}
	}
	}
	}
	}
	}
	i10 += nb * (ne01 - ir1);

[ggerganov]

Good catch. This code wasn't exercised by the tests it is used when dst is non-contiguous. I added an option to permute the dst tensor for the test_cpy.

I used the nk00 to indicate number of blocks of src0 along dim 0 (i.e. along the row). Respectively, the counter is k00.