Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Falcon7b T3K/TG demo 47% prefill perf regression #18148

Open
skhorasganiTT opened this issue Feb 21, 2025 · 1 comment
Open

Falcon7b T3K/TG demo 47% prefill perf regression #18148

skhorasganiTT opened this issue Feb 21, 2025 · 1 comment
Assignees

Comments

@skhorasganiTT
Copy link
Contributor

Falcon7b T3K/TG demo perf has regressed (for t3k seq 1024, prefill by ~47% and decode by ~5%). Verified locally that the failure started with a0ea595.

First T3K failing: https://github.com/tenstorrent/tt-metal/actions/runs/13444040365/job/37565282915
No t3k last passing, t3k demo pipelines have been hung for last couple weeks

Last TG passing: https://github.com/tenstorrent/tt-metal/actions/runs/13360145482/job/37308605343
First TG failing: https://github.com/tenstorrent/tt-metal/actions/runs/13402593833/job/37436533588

T3K seq 1024 Command: WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest --disable-warnings -q -s --input-method=json --input-path='models/demos/t3000/falcon7b/input_data_t3000.json' models/demos/t3000/falcon7b/demo_t3000.py -k "perf_mode_1024_stochastic_verify and 8"

T3K seq 1024 perf before a0ea595:

Image

T3K seq 1024 perf with a0ea595:

Image

cc @uaydonat @blozano-tt

@blozano-tt
Copy link
Contributor

@TT-BrianLiu should we revert this commit to get closer to green?
cc: @ttmchiou

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants