Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AssertionError in toposort_items (rmscene/crdt_sequence.py) #32

Closed
Seb-sti1 opened this issue Oct 18, 2024 · 2 comments
Closed

AssertionError in toposort_items (rmscene/crdt_sequence.py) #32

Seb-sti1 opened this issue Oct 18, 2024 · 2 comments

Comments

@Seb-sti1
Copy link
Contributor

Seb-sti1 commented Oct 18, 2024 •
edited
Loading

(Thank you for creating this and the rmc project!)

Issue

While I was working on Seb-sti1/rmtree!1 with Seb-sti1/rmc (a fork of rmc that uses rmscene v0.5.0), I stumble upon 2 rm files that appear to violate the assertion on next_items in toposort_items (rmscene/crdt_sequence.py).

I am open to sending the relevant rm files (but by email, don't really want to publish them publicly on GitHub) and try to help :)

Tests

Using the current version of rmscene (master) and Seb-sti1/rmc@dev, I tried the following for both rm file:

  • Editing the layer
  • Creating a new layer and removing the old one

In all 6 cases (original, layer edited, and new layer version), the blocks and blocks-data exporters work but not the markdown and svg:

document original version (reMarkable vUnknown) layer edited (reMarkable v3.14.1.9) new layer (reMarkable v3.14.1.9)
a AssertionError AssertionError AssertionError
b AssertionError AssertionError ValueError

Here is the output when running python -m rmc.cli -t markdown -vvv a-original.rm. It is similar for all AssertionError.

the output 
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 9
DEBUG:rmscene.scene_stream:Reading AuthorIdsBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 0
DEBUG:rmscene.scene_stream:Reading MigrationInfoBlock
DEBUG:rmscene.tagged_block_reader:Block header: 0 1 10
DEBUG:rmscene.scene_stream:Reading PageInfoBlock
DEBUG:rmscene.tagged_block_reader:Block header: 0 1 13
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 4
DEBUG:rmscene.scene_stream:Reading SceneGroupItemBlock
DEBUG:rmscene.tagged_block_reader:Block header: 2 2 5
DEBUG:rmscene.scene_stream:Reading SceneLineItemBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 8
DEBUG:rmscene.scene_stream:Reading SceneTombstoneItemBlock
DEBUG:rmscene.tagged_block_reader:Block header: 2 2 5
DEBUG:rmscene.scene_stream:Reading SceneLineItemBlock
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 184, in <module>
    cli()
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 58, in cli
    convert_rm(Path(fn), to, fout)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 132, in convert_rm
    print_text(f, fout)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/exporters/markdown.py", line 13, in print_text
    anchor_ids = set(collect_anchor_ids(tree.root))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/sebspace/rmc/src/rmc/exporters/markdown.py", line 65, in collect_anchor_ids
    yield from collect_anchor_ids(child)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/exporters/markdown.py", line 64, in collect_anchor_ids
    for child in item.children.values():
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/crdt_sequence.py", line 69, in values
    return [self[item_id] for item_id in self]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/crdt_sequence.py", line 61, in __iter__
    yield from toposort_items(self._items.values())
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/crdt_sequence.py", line 134, in toposort_items
    assert next_items
           ^^^^^^^^^^
AssertionError

Regarding the ValueError... I've just discovered it...

Maybe it is easier if I look a bit into it and then open a new issue if necessary?
Also, it feels linked more to the reMarkable v3.14.1.9 than the AssertionError.

EDIT: It appears to be similar to #16, maybe the fact that i have the rm file before deleting the layer can help this other issue...

Here is the output when running python -m rmc.cli -t svg -vvv b-layer-removed.rm

the output 
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 9
DEBUG:rmscene.scene_stream:Reading AuthorIdsBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 0
DEBUG:rmscene.scene_stream:Reading MigrationInfoBlock
DEBUG:rmscene.tagged_block_reader:Block header: 0 1 10
DEBUG:rmscene.scene_stream:Reading PageInfoBlock
DEBUG:rmscene.tagged_block_reader:Block header: 0 1 13
WARNING:rmscene.tagged_block_reader:Some data has not been read. The data may have been written using a newer format than this reader supports.
INFO:rmscene.tagged_block_reader:In MainBlockInfo(offset=132, size=46, extra_data=b'', block_type=13, min_version=0, current_version=1) only read 33 bytes
DEBUG:rmscene.tagged_block_reader:Excess bytes:
 5c080000007c05000050070000
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 184, in <module>
    cli()
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 58, in cli
    convert_rm(Path(fn), to, fout)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 134, in convert_rm
    tree = read_tree(f)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 896, in read_tree
    build_tree(tree, read_blocks(data))
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 846, in build_tree
    for b in blocks:
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 827, in read_blocks
    yield from _read_blocks(stream)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 795, in _read_blocks
    with stream.read_block() as block_info:
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 1
DEBUG:rmscene.scene_stream:Reading SceneTreeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 7
DEBUG:rmscene.scene_stream:Reading RootTextBlock
INFO:rmscene.tagged_block_reader:In MainBlockInfo(offset=941, size=86, extra_data=b'', block_type=7, min_version=1, current_version=1) only read 68 bytes
DEBUG:rmscene.tagged_block_reader:Excess bytes:
 5c0d0000001c080000001f00002400000000
Stack (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 184, in <module>
    cli()
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 58, in cli
    convert_rm(Path(fn), to, fout)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 134, in convert_rm
    tree = read_tree(f)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 896, in read_tree
    build_tree(tree, read_blocks(data))
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 846, in build_tree
    for b in blocks:
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 827, in read_blocks
    yield from _read_blocks(stream)
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 795, in _read_blocks
    with stream.read_block() as block_info:
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 2 2
DEBUG:rmscene.scene_stream:Reading TreeNodeBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 4
DEBUG:rmscene.scene_stream:Reading SceneGroupItemBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 4
DEBUG:rmscene.scene_stream:Reading SceneGroupItemBlock
DEBUG:rmscene.tagged_block_reader:Block header: 1 1 4
DEBUG:rmscene.scene_stream:Reading SceneGroupItemBlock
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 184, in <module>
    cli()
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 58, in cli
    convert_rm(Path(fn), to, fout)
  File "/home/seb-sti1/sebspace/rmc/src/rmc/cli.py", line 134, in convert_rm
    tree = read_tree(f)
           ^^^^^^^^^^^^
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 896, in read_tree
    build_tree(tree, read_blocks(data))
  File "/home/seb-sti1/.cache/pypoetry/virtualenvs/rmc-T-zEGfyf-py3.12/lib/python3.12/site-packages/rmscene/scene_stream.py", line 869, in build_tree
    raise ValueError(
ValueError: Node does not exist for SceneGroupItemBlock: None

Can't it come from Seb-sti1/rmc?

From my understanding, I don't think it comes from my modification of rmc (or rmc in general). Especially regarding the markdown exporter, I didn't change a lot of things compare to your mr (vs Seb-sti1/rmc exporters/markdown.py).

As a side note, that is not a part of the "rm stack" that I am very familiar with (especially of the crdt stuff) and, while I will try to help, I will need a bit of time 😅...

Let me know if you need others tests/info/the rm files.
@Seb-sti1 Seb-sti1 mentioned this issue Nov 18, 2024
Merged
@ricklupton
Copy link
Owner

Hi @Seb-sti1 , thanks for this! If you’re happy to share a file that shows the assertion error, I can try to have a look (email in git commits)

@ricklupton ricklupton mentioned this issue Nov 20, 2024
Merged
@Seb-sti1
Copy link
Contributor Author

Seb-sti1 commented Nov 21, 2024 •
edited
Loading

#36 fixes this issue for all my affected files

ty

@Seb-sti1 Seb-sti1 mentioned this issue Nov 21, 2024
Merged
@Seb-sti1 Seb-sti1 mentioned this issue Dec 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ricklupton @Seb-sti1