Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClickHouse ndc_client error on FixedString fields #10676

Open
bitjson opened this issue Feb 7, 2025 · 4 comments
Open

ClickHouse ndc_client error on FixedString fields #10676

bitjson opened this issue Feb 7, 2025 · 4 comments
Labels
c/v3-ndc-clickhouse k/v3-bug Bug affecting Hasura v3 (DDN)

Comments

@bitjson
Copy link

bitjson commented Feb 7, 2025

I've followed the Get Started with Hasura DDN and ClickHouse guide using the following table and data:

CREATE TABLE IF NOT EXISTS chaingraph.block
(
  internal_id          UInt64 COMMENT 'A unique, UInt64 identifier for this block assigned by Chaingraph. This value is not guaranteed to be consistent between Chaingraph instances.' CODEC(DoubleDelta, ZSTD),
  height               UInt32 COMMENT 'The height of this block: the number of blocks mined between this block and its genesis block (block 0).' CODEC(Delta, ZSTD),
  version              Int32 COMMENT 'The "version" field of this block; a 4-byte field typically represented as an Int32. While originally designed to indicate a block''s version, this field has been used for several other purposes. BIP34 ("Height in Coinbase") enforced a minimum version of 2, BIP66 ("Strict DER Signatures") enforced a minimum version of 3, then BIP9 repurposed most bits of the version field for network signaling. In recent years, the version field is also used for the AsicBoost mining optimization.' CODEC(T64, ZSTD),
  timestamp            UInt32 COMMENT 'The Uint32 current Unix timestamp claimed by the miner at the time this block was mined. By consensus, block timestamps must be within ~2 hours of the actual time, but timestamps are not guaranteed to be accurate. Timestamps of later blocks can also be earlier than their parent blocks.' CODEC(T64, ZSTD),
  hash                 FixedString(32) COMMENT 'The 32-byte, double-sha256 hash of the block header (encoded using the standard P2P network format) in big-endian byte order. This is used as a universal, unique identifier for the block. Big-endian byte order is typically seen in block explorers and user interfaces (as opposed to little-endian byte order, which is used in standard P2P network messages).', -- While a hash, mining produces less-random values, so default compression is enabled rather than CODEC(NONE)
  previous_block_hash  FixedString(32) COMMENT 'The 32-byte, double-sha256 hash of the previous block''s header in big-endian byte order. This is the byte order typically seen in block explorers and user interfaces (as opposed to little-endian byte order, which is used in standard P2P network messages).', -- References block hashes, so also uses default compression
  merkle_root          FixedString(32) COMMENT 'The 32-byte root hash of the double-sha256 merkle tree of transactions confirmed by this block. Note, the unusual merkle tree construction used by most chains is vulnerable to CVE-2012-2459. The final node in oddly-numbered levels is duplicated, and special care is required to ensure trees contain minimal duplication.' CODEC(NONE),
  bits                 UInt32 COMMENT 'The Uint32 packed representation of the difficulty target being used for this block. To be valid, the block hash value must be less than this difficulty target.' CODEC(T64, ZSTD),
  nonce                UInt32 COMMENT 'The uint32 nonce used for this block. This field allows miners to introduce entropy into the block header, changing the resulting hash during mining.',
  size_bytes           UInt32 COMMENT 'The network-encoded size of this block in bytes including transactions.' CODEC(T64, ZSTD) -- Requires migration to UInt64 beyond ~4.2GB
)
ENGINE = MergeTree
ORDER BY (internal_id)
COMMENT 'A blockchain block.';

And some sample data:

INSERT INTO chaingraph.block (`internal_id`, `height`, `version`, `timestamp`, `hash`, `previous_block_hash`, `merkle_root`, `bits`, `nonce`, `size_bytes`) VALUES (883295, 881225, 696573952, 1736993947, '\0\0\0\0\0\0\0\0\0e??w=ܼ??A?3C???7?\b?', '\0\0\0\0\0\0\0\0\0?????-p???`??`?*"??B7', 'b?&bBV?"P?:?b|?Sn?Ձ@m?x\f\00)', 402794910, 2174162995, 105300), (883294, 881224, 541065216, 1736992984, '\0\0\0\0\0\0\0\0\0?????-p???`??`?*"??B7', '\0\0\0\0\0\0\0\0\0>?\\?ժ?w?~\r????b3??', '\0?KrW??n?ʓ/?\t}??\f????^G?\\?R', 402795160, 2767849536, 16946), (883293, 881223, 536952832, 1736992824, '\0\0\0\0\0\0\0\0\0>?\\?ժ?w?~\r????b3??', '\0\0\0\0\0\0\0\0\0?v?\'??trU??)5\fߏO??,, '????{??5???\rI?rݼ6?AY?WV-??]', 402795183, 3150015996, 57903), (883292, 881222, 636264448, 1736992264, '\0\0\0\0\0\0\0\0\0?v?\'??trU??)5\fߏO??,, '\0\0\0\0\0\0\0\0??j[lH?lV?L0? 0???Ӄ', '??Zf?\n?1?TӤ???\n$?2}??\n1?ޟ?\n?', 402795393, 3363543219, 12386), (883291, 881221, 1040187392, 1736992033, '\0\0\0\0\0\0\0\0??j[lH?lV?L0? 0???Ӄ', '\0\0\0\0\0\0\0\0\b??,?r?|????\0T?|I?\r?@?U', 'زr?*ǫ????ؒ??V??>?;?c\tZ?????', 402795454, 2916308377, 41081), (883290, 881220, 794255360, 1736991539, '\0\0\0\0\0\0\0\0\b??,?r?|????\0T?|I?\r?@?U', '\0\0\0\0\0\0\0\0\0?q???!?}B1?<_?(????r?', '?W???g!???f^D?b\'c?=l??P<?0?G?\'', 402795651, 1693175809, 17300), (883289, 881219, 593846272, 1736991284, '\0\0\0\0\0\0\0\0\0?q???!?}B1?<_?(????r?', '\0\0\0\0\0\0\0\0%ף???w?I??M????o', '<????8?E\r??C??ӼX??3a?X"?\t?g?R?', 402795170, 1472665657, 133537), (883288, 881218, 1006632960, 1736989840, '\0\0\0\0\0\0\0\0%ף???w?I??M????o', '\0\0\0\0\0\0\0\0L???2DX??3ZV??}???\0B', '?yN?q???`*V?ц?)H?p?Sj???W', 402795488, 1505436101, 16579), (883287, 881217, 577945600, 1736989799, '\0\0\0\0\0\0\0\0L???2DX??3ZV??}???\0B', '\0\0\0\0\0\0\0\0\0l"???\b٣???+H~X?S??', '?`j?DA]??2??Yj`???ՠ?ITH}?h??(??', 402795659, 597684090, 33317), (883286, 881216, 603979776, 1736989498, '\0\0\0\0\0\0\0\0\0l"???\b٣???+H~X?S??', '\0\0\0\0\0\0\0\0?u}??\tzb?.%??&???is??,?', '?Ug?1bV}??.?*}?a>4XB?}??"q$?', 402795930, 236196732, 12783);

Queries requesting e.g. numbers are successful:

{
  chaingraphBlock(limit: 10, order_by: {height: Desc}) {
    height
    sizeBytes
  }
}
{
  "data": {
    "chaingraphBlock": [
      {
        "height": 881225,
        "sizeBytes": 105300
      },
      {
        "height": 881224,
        "sizeBytes": 16946
      },
      {
        "height": 881223,
        "sizeBytes": 57903
      },
      {
        "height": 881222,
        "sizeBytes": 12386
      },
      {
        "height": 881221,
        "sizeBytes": 41081
      },
      {
        "height": 881220,
        "sizeBytes": 17300
      },
      {
        "height": 881219,
        "sizeBytes": 133537
      },
      {
        "height": 881218,
        "sizeBytes": 16579
      },
      {
        "height": 881217,
        "sizeBytes": 33317
      },
      {
        "height": 881216,
        "sizeBytes": 12783
      }
    ]
  }
}

While queries which request a FixedString field fail with the error: internal error: ndc_unexpected: ndc_client error: request to connector failed with status code N/A: error decoding response body

E.g.:

{
  chaingraphBlock(limit: 10, order_by: {height: Desc}) {
    height
    hash
  }
}
{
  "data": {
    "chaingraphBlock": null
  },
  "errors": [
    {
      "message": "internal error",
      "path": [
        "chaingraphBlock"
      ]
    }
  ]
}
@bitjson bitjson added the k/v3-bug Bug affecting Hasura v3 (DDN) label Feb 7, 2025
@bitjson
Copy link
Author

bitjson commented Feb 7, 2025

Am I misconfiguring something? Preferably, I'd like the results to be returned using ClickHouse's lower and hex functions, e.g. lower(hex(hash)).

@BenoitRanque
Copy link
Contributor

I'm not sure what's happening here, but I get this error when trying to run the insert in your reproduction steps:

Code: 62. DB::Exception: Cannot parse expression of type FixedString(32) here: '\0\0\0\0\0\0\0\0\0?v?\'??trU??)5\fߏO??,, '????{??5???\rI?rݼ6?AY?WV-??]', 402795183, 3150015996, 57903), (883292, 881222, 636264448, 1736992264, '\0\0\0\0\0\0: While executing ValuesBlockInputFormat. (SYNTAX_ERROR) (version 25.1.3.23 (official build))

I'm using the http interface

@bitjson
Copy link
Author

bitjson commented Feb 10, 2025

@BenoitRanque Ah sorry, I assumed ClickHouse's FORMAT SQLInsert would just work. Had to use:

SELECT concat(
    'INSERT INTO block (',
    'internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes',
    ') VALUES (',
      toString(internal_id), ', ',
      toString(height), ', ',
      toString(version), ', ',
      toString(timestamp), ', ',
      'unhex(\'', hex(hash), '\'), ',
      'unhex(\'', hex(previous_block_hash), '\'), ',
      'unhex(\'', hex(merkle_root), '\'), ',
      toString(bits), ', ',
      toString(nonce), ', ',
      toString(size_bytes),
    ');'
) AS insert_statement
FROM block
ORDER BY height DESC
LIMIT 10

A cleaned up insert:

INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883295, 881225, 696573952, 1736993947, unhex('30303030303030303030303030303030303036353066613330376133373733646463626362316161343163323333343364623366636233376631623830386233'), unhex('30303030303030303030303030303030303039646261663738636136326430343730663864666339363062326333313836306333326132323938396234323337'), unhex('36326566393432363632343230323536663032323530613333613839363237636136353336656538643538313430376636646165373830633030313933303239'), 402794910, 2174162995, 105300);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883294, 881224, 541065216, 1736992984, unhex('30303030303030303030303030303030303039646261663738636136326430343730663864666339363062326333313836306333326132323938396234323337'), unhex('30303030303030303030303030303030303033656236356365376435616162633737383137653064613631356663393664303166363233333132303663656532'), unhex('30303366346237323537383461313134366539356361393331363266653630393764383539613063663664666638656261383565343764643563316338653532'), 402795160, 2767849536, 16946);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883293, 881223, 536952832, 1736992824, unhex('30303030303030303030303030303030303033656236356365376435616162633737383137653064613631356663393664303166363233333132303663656532'), unhex('30303030303030303030303030303030303061623736663331323237643530346661373437323535643266363239333530636466386634663863663532633162'), unhex('66376432656363383762613066313335653439323035653139386465306434396264373264646263333663313431353964303537313135363264656164353564'), 402795183, 3150015996, 57903);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883292, 881222, 636264448, 1736992264, unhex('30303030303030303030303030303030303061623736663331323237643530346661373437323535643266363239333530636466386634663863663532633162'), unhex('30303030303030303030303030303030303163646462366135623663303634383866366335363865346333303836323030343330623065346361313764333833'), unhex('66626363356136363830306161383331656135346433613461636636336630613234666330333332376465346330306133316236313364653966393730616337'), 402795393, 3363543219, 12386);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883291, 881221, 1040187392, 1736992033, unhex('30303030303030303030303030303030303163646462366135623663303634383866366335363865346333303836323030343330623065346361313764333833'), unhex('30303030303030303030303030303030303130383964393032636433373261393763653064356366636330303534636337633439643430643865343065393535'), unhex('64386232373264353035326163376162396638613933633864383932383365333536613262623365386133626437363330393561386463386534633030366163'), 402795454, 2916308377, 41081);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883290, 881220, 794255360, 1736991539, unhex('30303030303030303030303030303030303130383964393032636433373261393763653064356366636330303534636337633439643430643865343065393535'), unhex('30303030303030303030303030303030303064393731616364316564323138663764343233313131633331643363356661633238653663336639643137326465'), unhex('63323537336663666635363731363231666663316466363630353565343462353632323736336539336436633933396635303363386633306232343766383237'), 402795651, 1693175809, 17300);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883289, 881219, 593846272, 1736991284, unhex('30303030303030303030303030303030303064393731616364316564323138663764343233313131633331643363356661633238653663336639643137326465'), unhex('30303030303030303030303030303030303232356437613366653033666264333737383734393961636234643931316331356433303230666634656231313666'), unhex('33633832613738666338333861623435306462393838343363626535643362633538613562313333363139623538316132326137303962663637663835326635'), 402795170, 1472665657, 133537);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883288, 881218, 1006632960, 1736989840, unhex('30303030303030303030303030303030303232356437613366653033666264333737383734393961636234643931316331356433303230666634656231313666'), unhex('30303030303030303030303030303030303230663463646164636636333234343139353862633839313933333561353664656338376461366535653630303432'), unhex('39373739346565663731393763616566393036303261353663656431383662393131303732393438393037306330353336616632386162663037653264383537'), 402795488, 1505436101, 16579);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883287, 881217, 577945600, 1736989799, unhex('30303030303030303030303030303030303230663463646164636636333234343139353862633839313933333561353664656338376461366535653630303432'), unhex('30303030303030303030303030303030303036633232623138653031613230386439613366656437666530663262303534383765353865623533643331666262'), unhex('64633630366161613434343135643961383833323932663135393661363064316634643864356130386634393534343837646635363866386365323839343934'), 402795659, 597684090, 33317);
INSERT INTO block (internal_id, height, version, timestamp, hash, previous_block_hash, merkle_root, bits, nonce, size_bytes) VALUES (883286, 881216, 603979776, 1736989498, unhex('30303030303030303030303030303030303036633232623138653031613230386439613366656437666530663262303534383765353865623533643331666262'), unhex('30303030303030303030303030303030303138383735376464396633303937613632666632653235383862623236383938386334363937333965633632633862'), unhex('66323535363765613331363235363764666165323265633332613764303338343066363133653334353834326461303137646438646632323133373132343961'), 402795930, 236196732, 12783);

@bitjson
Copy link
Author

bitjson commented Feb 10, 2025

And just a note: I'm now using views over all tables rather than directly pointing Hasura Models to the underlying tables.

It would still be nice to have FixedString "just work" for Models based directly on tables, but in my case, using a view seems to be a more natural solution:

CREATE TABLE default.block_table
(
  internal_id UInt64 CODEC(DoubleDelta, ZSTD),
  height UInt32 CODEC(Delta, ZSTD),
  version Int32 CODEC(T64, ZSTD),
  timestamp UInt32 CODEC(T64, ZSTD),
  hash FixedString(32),
  previous_block_hash FixedString(32),
  merkle_root FixedString(32) CODEC(NONE),
  bits UInt32 CODEC(T64, ZSTD),
  nonce UInt32 CODEC(T64, ZSTD),
  size_bytes UInt32 CODEC(T64, ZSTD)
)
ENGINE = MergeTree
ORDER BY (internal_id)
COMMENT 'The source table for the block view.';

-- GraphQL API views

CREATE VIEW default.block
(
    `internal_id` UInt64 COMMENT 'A unique, UInt64 identifier for this block assigned by Chaingraph. This value is not guaranteed to be consistent between Chaingraph instances.',
    `height` UInt32 COMMENT 'The height of this block: the number of blocks mined between this block and its genesis block (block 0).',
    `version` Int32 COMMENT 'The "version" field of this block; a 4-byte field typically represented as an Int32. While originally designed to indicate a block''s version, this field has been used for several other purposes. BIP34 ("Height in Coinbase") enforced a minimum version of 2, BIP66 ("Strict DER Signatures") enforced a minimum version of 3, then BIP9 repurposed most bits of the version field for network signaling. In recent years, the version field is also used for the AsicBoost mining optimization.',
    `timestamp` UInt32 COMMENT 'The Uint32 current Unix timestamp claimed by the miner at the time this block was mined. By consensus, block timestamps must be within ~2 hours of the actual time, but timestamps are not guaranteed to be accurate. Timestamps of later blocks can also be earlier than their parent blocks.',
    `hash` String COMMENT 'The 32-byte, double-sha256 (hex-encoded) hash of the block header (encoded using the standard P2P network format) in big-endian byte order. This is used as a universal, unique identifier for the block. Big-endian byte order is typically seen in block explorers and user interfaces (as opposed to little-endian byte order, which is used in standard P2P network messages).',
    `previous_block_hash` String COMMENT 'The 32-byte, double-sha256 (hex-encoded) hash of the previous block''s header in big-endian byte order. This is the byte order typically seen in block explorers and user interfaces (as opposed to little-endian byte order, which is used in standard P2P network messages).',
    `merkle_root` String COMMENT 'The 32-byte (hex-encoded) root hash of the double-sha256 merkle tree of transactions confirmed by this block. Note, the unusual merkle tree construction used by most chains is vulnerable to CVE-2012-2459. The final node in oddly-numbered levels is duplicated, and special care is required to ensure trees contain minimal duplication.',
    `bits` UInt32 COMMENT 'The Uint32 packed representation of the difficulty target being used for this block. To be valid, the block hash value must be less than this difficulty target.',
    `nonce` UInt32 COMMENT 'The uint32 nonce used for this block. This field allows miners to introduce entropy into the block header, changing the resulting hash during mining.',
    `size_bytes` UInt32 COMMENT 'The network-encoded size of this block in bytes including transactions.'
)
AS
(SELECT
    internal_id,
    height,
    version,
    timestamp,
    lower(hex(hash)) AS hash,
    lower(hex(previous_block_hash)) AS previous_block_hash,
    lower(hex(merkle_root)) AS merkle_root,
    bits,
    nonce,
    size_bytes
FROM default.block_table) COMMENT 'A blockchain block.';

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/v3-ndc-clickhouse k/v3-bug Bug affecting Hasura v3 (DDN)
Projects
None yet
Development

No branches or pull requests

3 participants