Potential issue with extrinsic hash

[jaymansfield]

I'm getting a mismatch of the extrinsic hash when a guarantee is included in the extrinsics.

The first vector to include one in the extrinsics is assurances/1_003.

The vector is expecting an extrinsic hash of 0xacda7941616e5d458f88c0a140396c65dc66d3d44d7bb64a84c6739095af7ddd.
I am arriving at 0x4b545b182c0d0e9b13f0a84e37d42fe5dbe7d5cc4be41bd890a2e2785039dc49.

My algorithm seems to match the GP so I'm not sure if its my issue or an issue with the vectors.

For assurances/1_003, can you please let me know the value of "g" here (prior to being hashed) so we can see which one is correct?

My value:
0x01bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c0d000000030100b269a27a06e07e9616ed1da82408b04ea8bc5cf33f24fd927170205918df5f83221683fa14a84ed95aebb51ff8cd6fedd9b48505399024c16939b5900fb3490a0200a3b4b148ca9b3fb9cc08c8b8fe615b75cb5f375e52338661756eeaad651d50382723dfc8ca268df3e708b68c2190e0783a6088eb10a07dcbd8376cef823afd0703007cd48123d6a5e19d3d8f36233d565e5cc6d70bc30777a3d594ba3678619b3c18c0cc480fb5726cb988fae8ff9372334bee4936e251c6afc02f1239bbfd239802

[sourabhniyogi]

Will publish our raw value and see if we can get to the bottom of this!

[sourabhniyogi]

For this 0.6.2.1 block 1_003.json here is our derivation of the extrinsic hash 0xacda7941616e5d458f88c0a140396c65dc66d3d44d7bb64a84c6739095af7ddd -- can you identify which out of E_{T,P,G,A,D} has the discrepancy so we can zoom in on it?

=== RUN   TestExtrinsicHash
E_T_encoded: 03013f0c489bcce6f070afa5d47151e75594eb9d84ba8b684f011addf723f2cb912f06d7df92e3761916af16ae157210289e86f5e5377387a97108e270af895215c0564f4d2decae42cd536bba30132e300478e4a87a09ebf5e1e5d7d5097a767c251ea6eb2a79cdb39d6ad4e73129263f72a739687904b0aa03ba5f6399e1abd6538c3c46fd81f752582ade78c15e232fcaea203490b958b9bcd02179aa8ddebd1bd27c96a5153412aa93034cadb7bef455fe86485a70e6e7ac65041252ff6e6108ab12a26809d0d8689a3fc0371b78535dbc62fd3cd117d62e82e4673ed65cdea431256a741c029ec575575208b324df0f96c99f91a977f75acdcbc856354f68720ae41ba921b9d4e3f58513e9f5b22bb4e1cb6d27b5b3911c1fdbacf977e92dd7823ea264b6cabbd299d3678fb01a8dc74a9ae4ea6a8fa5a4cd3233859e91e3487a730838714c868c3e4e61a12b1ae008b652d02f62fa20178cd3b2ee73e15c1ca8ea0df5eab0df1e74414880ef3fcd445b797dd9587664d01f79e823722563f0908f5ff996a285d96d6fa2f9d480d062efc9aa93a1e23414e3571d71bb82a76784d01372b28cda9bf39fa24ce0a85fa1999454f37e39443fd1121882ea5a99021bbf5e3787f2a14160892d0170c7388461106d29da1a12e0626a1e18b22118398f92a6526cea3bc3cceb827ed8623fc1884cdfe7f4f77ebc02168d98600c7c52c991a617d48ff018c459f9bbf3344f50e3df7bda1fccc2e2899588c21ea9710546187658e0a1fb2fe4422b89e73603ae4655126e5f8863488087798b234cbc30f28d956c8b9cdd4b2b0aae06ff908b53fdc5084c29d2f02099164a34bcc04352823ad365329ad31e577dbe777d2360917c6c4074a4ea12746056ae3c3f825ec470216559dac6537dc0ab4568652a5f01320044a0c7a4246c55bd3815555506a8164c543e353093fc0e5ad0db02b8f7218a1714039998a9932da9ece054caeb40cae060524426f27e1ecbad0515b6d8c15197a7a8a34fe4e10e1af6f5649c076799e6915ca4e598c96d5667cf97821ea739a44bb79da10f1bc664a3f99719af831eb256f1d974ddb365f9fc641e6ffd66013b24cd72a1fa9e678888d8281305ba9e3d1f680f7a341148e3f80b47bc7735a0be97c36458b6fcdb5a447292bf2d4f169baa6800697240f51c15de9f76ba890f5bf870a46daf7340ba4e976422706d59fe09604e5ce84171b7144ff0778f0b6b03530d08c20c6895ad036ed5c7320a6e98174142b8f56e0c2cc6e8979c058fd573405a0c51b5a882830c6a813c27c3e77332c614590a02e387a83cd57b12ad08ff2d9bdcf55b6ae68fdcc39ca6a9b501278b9bdba0ea6b5b3e6b1d07d7ca14159992f62394cd36f9d35822cdc5d1a28629b92cb148d42e06e07117d686e94b3741816c8a3fe57488385b5521cba9f6248b7520b215677afa1394464057709837dfb22b04b53c69011da7926c341cf6e77a2ed27912c40267c826cca53f876dada4564a3b8d8fa2eb16a6ce8e5e12d8a6f7df76b4b8d09e35c75e14a978398c95e21fdd4a9cdd4fc73971c07204e93947a84882afeb0816993be03152c7074fed1c7db0e1d8202e2fcecfd4d8eb997e1d9a20312969b85c4dfbfb65824d2fdd456fe97053854a56cde98996dcf20a0d672d05ee207f2d35c8f10e64e5fa831f6d09dbf9d22656c3a3873309978c1bde03ac6f565ec31290a7d7d5dcdb67203d0620ed2339aab7ba2d2ea582742c055caa1cd5058f0c2996662ea68904815cbf3b49adb5f673d7cfe270179edcbe127402c353875ae952416ce31f344f252c8b0d53963098ef0616dc3d2860939a4071881d6f753c0c31792bc2309c2259dbc75c4fe108098f12244af677ca1686c430887de724bdb93c91cd156485c94dd20236a6de568b08804a9581bd9956a24e6c3cf67e1f5c262e42825f872c19950f234e8d1035a525471f65af26b1d356514071f5daa665308f133cfeb7b8da49fc6f55586f32160d516a46bae7d5d30042f841d96744bd089991c80da16f7855695b30c38060821b259013098553dad9f78f34b6be90268220d52590a0c49cf2fc9588d22a0fc3ec310e95c073a901953dc7dacdbb762c46da57a71a9633ddcba64aad96cbb8d07b4dd0678af073c4755f91fc5424ae711bd7c4c5ce77a9f302037ff08a17167ef7f8fe0a95734f56b4b14058024593a173876eeff9cbb1a1dbb067857e386965d0a422e1c2a29c3c58c58f65b7862f08739fc18f9e32051d796714606d53f229f75e5d3ae4a8778e2297a5d19e44db42a13aa783f82af91068274235492d7313c2d67c0c73fb6dc5064d4d07206963f008572f0d0b2f30d09067e2bfa01fa235503c0ea66561dccb36005358631a37664928757c2190e4a953ad74896e92dfde91d6866c9401ecef7ea84dc10829335f685881d2afdf69bcea20c11f15458eb2743ae74ae432e4b9edeb02be05a02f2dd113e853d3d5ec9704f087c20988a4870f904caa096be543d1edbeadbcf1664008d19bf3f4519d57c65247ca40b044dcf1ae69ce2e2febec37f6369910a0b20b9dce71b4cd3396e44a90a0a4c404cb170d7ffd2c5467f152bd5daf40b392fc21a4f3bcec22f73d9aaca428b481c04c949d91502820d1090777d63bde77ae2cd4740b19f301a8f2da0aacd5ab81972dfc87b44673d1349de5e46bf2c3731e31ca8bb03512d845d762ac86c57c54392c0d2181ae685a0888bc69f4a1dc916444c1be62883afc051ec56a735fbb4d0f3369b177dfa79886c950bdab702c456779b0afef8753c8a1f3692d2a637df7fc5d9afed727e9e12ee7b870ad89e8103ad62f1fa2949187c490ce842becf36119338287beb7e34e8b91e729b995a8288dfdb600117e8c046d69de18474fe518e5050be4a139677740e5c734822ab70dfa294012c277e6996e4e0260a26493c9f794eb2ad3913b0a67f8429e3d91d11b787156961daa8ab74a9e434387a6b68b6562ed15fe60d963e20940483ef85270a6d2167aebb730966c87cb63bb0fd830eee25681dcaa211d2940c86dabacaa208856947aa41e3ff5a6422b26d64ef846d705508d6389df3848b4060db4f95b4ac787548f9adfa135ff4e05a2f2c73cc782632d396cc5089ae5ee7ff6ab38105582f3d4c0686d68c8f4c1da4cba2b2610b89009492f1a9ebb338bc27241869deb90c6221670c607ddccf6f09e9431b0ac44d532239e742c9c763ef9c4b86d310ab2fcfb0aae1983f87f9723c4e5aad0ae3b936fba534ec75195f0b71f76732c54173f10441845125d65efe95d5cd98c9b
E_P_encoded: 00
Guarantee 0 bytes: bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c0d000000030100b269a27a06e07e9616ed1da82408b04ea8bc5cf33f24fd927170205918df5f83221683fa14a84ed95aebb51ff8cd6fedd9b48505399024c16939b5900fb3490a0200a3b4b148ca9b3fb9cc08c8b8fe615b75cb5f375e52338661756eeaad651d50382723dfc8ca268df3e708b68c2190e0783a6088eb10a07dcbd8376cef823afd0703007cd48123d6a5e19d3d8f36233d565e5cc6d70bc30777a3d594ba3678619b3c18c0cc480fb5726cb988fae8ff9372334bee4936e251c6afc02f1239bbfd239802
E_G_encoded: 0180ebbc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c0d000000030100b269a27a06e07e9616ed1da82408b04ea8bc5cf33f24fd927170205918df5f83221683fa14a84ed95aebb51ff8cd6fedd9b48505399024c16939b5900fb3490a0200a3b4b148ca9b3fb9cc08c8b8fe615b75cb5f375e52338661756eeaad651d50382723dfc8ca268df3e708b68c2190e0783a6088eb10a07dcbd8376cef823afd0703007cd48123d6a5e19d3d8f36233d565e5cc6d70bc30777a3d594ba3678619b3c18c0cc480fb5726cb988fae8ff9372334bee4936e251c6afc02f1239bbfd239802
E_A_encoded: 00
E_D_encoded: 000000
E_T_hash: 23d642487df5c4d65aac7ff40f9b01ac5e9a26cfb2e7478f93673c5db67a5c0c
E_P_hash: 03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314
E_G_hash: b25f2d0247b53e7655ab4ce2017c7d13639478d28afa9716f0d4632ec84bddae
E_A_hash: 03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314
E_D_hash: ab29e6dc16755d0071eba349ebda225d15e4f910cb474549c47e95cb85ecc4d6
Combined data encoded: 23d642487df5c4d65aac7ff40f9b01ac5e9a26cfb2e7478f93673c5db67a5c0c03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314b25f2d0247b53e7655ab4ce2017c7d13639478d28afa9716f0d4632ec84bddae03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314ab29e6dc16755d0071eba349ebda225d15e4f910cb474549c47e95cb85ecc4d6
Final extrinsic hash: acda7941616e5d458f88c0a140396c65dc66d3d44d7bb64a84c6739095af7ddd

If its "g" see "Guarantee 0 bytes" vs "E_G_encoded" (from this within the block) -- happy to zoom in further into that as needed but need a clue as to what to zoom into =)

[jaymansfield]

So the difference is in the start of E_G_encoded ( 0180eb vs 01).

I believe it should start with a compact integer encoding of the number of guarantees and then have the hash of the first guarantee

Yours: 0180eb  bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee908494...

Mine:  01      bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee908494...

Can you see how you derive "80eb" after the length "01" ?

We both have the same report hash afterwards (bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c)

[sourabhniyogi]

Ok we agree with your conclusion =) -- we were encoding [][]byte rather than []GuaranteeIntermediateEncoding and ended up with that strange 80eb -- Thank you for this! Here are the raw values

=== RUN   TestExtrinsicHash
E_T_encoded: 03013f0c489bcce6f070afa5d47151e75594eb9d84ba8b684f011addf723f2cb912f06d7df92e3761916af16ae157210289e86f5e5377387a97108e270af895215c0564f4d2decae42cd536bba30132e300478e4a87a09ebf5e1e5d7d5097a767c251ea6eb2a79cdb39d6ad4e73129263f72a739687904b0aa03ba5f6399e1abd6538c3c46fd81f752582ade78c15e232fcaea203490b958b9bcd02179aa8ddebd1bd27c96a5153412aa93034cadb7bef455fe86485a70e6e7ac65041252ff6e6108ab12a26809d0d8689a3fc0371b78535dbc62fd3cd117d62e82e4673ed65cdea431256a741c029ec575575208b324df0f96c99f91a977f75acdcbc856354f68720ae41ba921b9d4e3f58513e9f5b22bb4e1cb6d27b5b3911c1fdbacf977e92dd7823ea264b6cabbd299d3678fb01a8dc74a9ae4ea6a8fa5a4cd3233859e91e3487a730838714c868c3e4e61a12b1ae008b652d02f62fa20178cd3b2ee73e15c1ca8ea0df5eab0df1e74414880ef3fcd445b797dd9587664d01f79e823722563f0908f5ff996a285d96d6fa2f9d480d062efc9aa93a1e23414e3571d71bb82a76784d01372b28cda9bf39fa24ce0a85fa1999454f37e39443fd1121882ea5a99021bbf5e3787f2a14160892d0170c7388461106d29da1a12e0626a1e18b22118398f92a6526cea3bc3cceb827ed8623fc1884cdfe7f4f77ebc02168d98600c7c52c991a617d48ff018c459f9bbf3344f50e3df7bda1fccc2e2899588c21ea9710546187658e0a1fb2fe4422b89e73603ae4655126e5f8863488087798b234cbc30f28d956c8b9cdd4b2b0aae06ff908b53fdc5084c29d2f02099164a34bcc04352823ad365329ad31e577dbe777d2360917c6c4074a4ea12746056ae3c3f825ec470216559dac6537dc0ab4568652a5f01320044a0c7a4246c55bd3815555506a8164c543e353093fc0e5ad0db02b8f7218a1714039998a9932da9ece054caeb40cae060524426f27e1ecbad0515b6d8c15197a7a8a34fe4e10e1af6f5649c076799e6915ca4e598c96d5667cf97821ea739a44bb79da10f1bc664a3f99719af831eb256f1d974ddb365f9fc641e6ffd66013b24cd72a1fa9e678888d8281305ba9e3d1f680f7a341148e3f80b47bc7735a0be97c36458b6fcdb5a447292bf2d4f169baa6800697240f51c15de9f76ba890f5bf870a46daf7340ba4e976422706d59fe09604e5ce84171b7144ff0778f0b6b03530d08c20c6895ad036ed5c7320a6e98174142b8f56e0c2cc6e8979c058fd573405a0c51b5a882830c6a813c27c3e77332c614590a02e387a83cd57b12ad08ff2d9bdcf55b6ae68fdcc39ca6a9b501278b9bdba0ea6b5b3e6b1d07d7ca14159992f62394cd36f9d35822cdc5d1a28629b92cb148d42e06e07117d686e94b3741816c8a3fe57488385b5521cba9f6248b7520b215677afa1394464057709837dfb22b04b53c69011da7926c341cf6e77a2ed27912c40267c826cca53f876dada4564a3b8d8fa2eb16a6ce8e5e12d8a6f7df76b4b8d09e35c75e14a978398c95e21fdd4a9cdd4fc73971c07204e93947a84882afeb0816993be03152c7074fed1c7db0e1d8202e2fcecfd4d8eb997e1d9a20312969b85c4dfbfb65824d2fdd456fe97053854a56cde98996dcf20a0d672d05ee207f2d35c8f10e64e5fa831f6d09dbf9d22656c3a3873309978c1bde03ac6f565ec31290a7d7d5dcdb67203d0620ed2339aab7ba2d2ea582742c055caa1cd5058f0c2996662ea68904815cbf3b49adb5f673d7cfe270179edcbe127402c353875ae952416ce31f344f252c8b0d53963098ef0616dc3d2860939a4071881d6f753c0c31792bc2309c2259dbc75c4fe108098f12244af677ca1686c430887de724bdb93c91cd156485c94dd20236a6de568b08804a9581bd9956a24e6c3cf67e1f5c262e42825f872c19950f234e8d1035a525471f65af26b1d356514071f5daa665308f133cfeb7b8da49fc6f55586f32160d516a46bae7d5d30042f841d96744bd089991c80da16f7855695b30c38060821b259013098553dad9f78f34b6be90268220d52590a0c49cf2fc9588d22a0fc3ec310e95c073a901953dc7dacdbb762c46da57a71a9633ddcba64aad96cbb8d07b4dd0678af073c4755f91fc5424ae711bd7c4c5ce77a9f302037ff08a17167ef7f8fe0a95734f56b4b14058024593a173876eeff9cbb1a1dbb067857e386965d0a422e1c2a29c3c58c58f65b7862f08739fc18f9e32051d796714606d53f229f75e5d3ae4a8778e2297a5d19e44db42a13aa783f82af91068274235492d7313c2d67c0c73fb6dc5064d4d07206963f008572f0d0b2f30d09067e2bfa01fa235503c0ea66561dccb36005358631a37664928757c2190e4a953ad74896e92dfde91d6866c9401ecef7ea84dc10829335f685881d2afdf69bcea20c11f15458eb2743ae74ae432e4b9edeb02be05a02f2dd113e853d3d5ec9704f087c20988a4870f904caa096be543d1edbeadbcf1664008d19bf3f4519d57c65247ca40b044dcf1ae69ce2e2febec37f6369910a0b20b9dce71b4cd3396e44a90a0a4c404cb170d7ffd2c5467f152bd5daf40b392fc21a4f3bcec22f73d9aaca428b481c04c949d91502820d1090777d63bde77ae2cd4740b19f301a8f2da0aacd5ab81972dfc87b44673d1349de5e46bf2c3731e31ca8bb03512d845d762ac86c57c54392c0d2181ae685a0888bc69f4a1dc916444c1be62883afc051ec56a735fbb4d0f3369b177dfa79886c950bdab702c456779b0afef8753c8a1f3692d2a637df7fc5d9afed727e9e12ee7b870ad89e8103ad62f1fa2949187c490ce842becf36119338287beb7e34e8b91e729b995a8288dfdb600117e8c046d69de18474fe518e5050be4a139677740e5c734822ab70dfa294012c277e6996e4e0260a26493c9f794eb2ad3913b0a67f8429e3d91d11b787156961daa8ab74a9e434387a6b68b6562ed15fe60d963e20940483ef85270a6d2167aebb730966c87cb63bb0fd830eee25681dcaa211d2940c86dabacaa208856947aa41e3ff5a6422b26d64ef846d705508d6389df3848b4060db4f95b4ac787548f9adfa135ff4e05a2f2c73cc782632d396cc5089ae5ee7ff6ab38105582f3d4c0686d68c8f4c1da4cba2b2610b89009492f1a9ebb338bc27241869deb90c6221670c607ddccf6f09e9431b0ac44d532239e742c9c763ef9c4b86d310ab2fcfb0aae1983f87f9723c4e5aad0ae3b936fba534ec75195f0b71f76732c54173f10441845125d65efe95d5cd98c9b
E_P_encoded: 00
Guarantee 0 Bytes: bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c0d000000030100b269a27a06e07e9616ed1da82408b04ea8bc5cf33f24fd927170205918df5f83221683fa14a84ed95aebb51ff8cd6fedd9b48505399024c16939b5900fb3490a0200a3b4b148ca9b3fb9cc08c8b8fe615b75cb5f375e52338661756eeaad651d50382723dfc8ca268df3e708b68c2190e0783a6088eb10a07dcbd8376cef823afd0703007cd48123d6a5e19d3d8f36233d565e5cc6d70bc30777a3d594ba3678619b3c18c0cc480fb5726cb988fae8ff9372334bee4936e251c6afc02f1239bbfd239802
E_G_encoded: 01bc73bbb98b756d6687090f155eed04bdd513ed5b20d3f319ee9084943429280c0d000000030100b269a27a06e07e9616ed1da82408b04ea8bc5cf33f24fd927170205918df5f83221683fa14a84ed95aebb51ff8cd6fedd9b48505399024c16939b5900fb3490a0200a3b4b148ca9b3fb9cc08c8b8fe615b75cb5f375e52338661756eeaad651d50382723dfc8ca268df3e708b68c2190e0783a6088eb10a07dcbd8376cef823afd0703007cd48123d6a5e19d3d8f36233d565e5cc6d70bc30777a3d594ba3678619b3c18c0cc480fb5726cb988fae8ff9372334bee4936e251c6afc02f1239bbfd239802
E_A_encoded: 00
E_D_encoded: 000000
E_T_hash: 23d642487df5c4d65aac7ff40f9b01ac5e9a26cfb2e7478f93673c5db67a5c0c
E_P_hash: 03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314
E_G_hash: b1d242dabff557ac739f5e5fd85a3caf9cee06bb114d79b7d5dcb633e9585a82
E_A_hash: 03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314
E_D_hash: ab29e6dc16755d0071eba349ebda225d15e4f910cb474549c47e95cb85ecc4d6
Combined data encoded: 23d642487df5c4d65aac7ff40f9b01ac5e9a26cfb2e7478f93673c5db67a5c0c03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314b1d242dabff557ac739f5e5fd85a3caf9cee06bb114d79b7d5dcb633e9585a8203170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314ab29e6dc16755d0071eba349ebda225d15e4f910cb474549c47e95cb85ecc4d6
Final extrinsic hash: 4b545b182c0d0e9b13f0a84e37d42fe5dbe7d5cc4be41bd890a2e2785039dc49
0x4b545b182c0d0e9b13f0a84e37d42fe5dbe7d5cc4be41bd890a2e2785039dc49
--- PASS: TestExtrinsicHash (0.00s)

[jaymansfield]

Final extrinsic hash: 4b545b182c0d0e9b13f0a84e37d42fe5dbe7d5cc4be41bd890a2e2785039dc49 -- we are in alignment now!

[sourabhniyogi]

Addressed in 0.6.2.2

[danicuki]

I am trying to figure out the difference in my g encoding, but it seems that the last assurances/1_003.json in the file is different from the one I am looking at. Would you please share the link of the block file / json you used in this issue?

[danicuki]

found it: https://github.com/jam-duna/jamtestnet/blob/0.6.2.1/data/assurances/blocks/1_003.json

[danicuki]

I suggest using the same syntax on reported field as the w3f vectors use in the state component:

"reported": [
    {
        "hash": "0x016cb55eb7b84e0d495d40832c7238965baeb468932c415dc2ceffe0afb039e5",
        "exports_root": "0x935f6dfef36fa06e10a9ba820f933611c05c06a207b07141fe8d87465870c11c"
    },
    {
        "hash": "0x76bcb24901299c331f0ca7342f4874f19b213ee72df613d50699e7e25edb82a6",
        "exports_root": "0xc825d16b7325ca90287123bd149d47843c999ce686ed51eaf8592dd2759272e3"
    }
]

https://github.com/davxy/jam-test-vectors/blob/01e036c66727b8d6d058d5ad15c2eeec37d1d1f4/history/data/progress_blocks_history-4.json#L43