Add Backtrace Screen #1270

MrCirdo · 2023-07-19T16:18:57Z

Hello everyone!

This is my first big pull request 😃

The goal of this PR is to add a backtrace screen for process or thread.
Here's what it looks like for a thread :

And for a process:

Behind, I use the tool called eu-stack provided by elf-utils.
The standard output is parsed and printed to the screen.
Currently, I have implemented only the Refresh button. And my world is inspired by TraceScreen and OpenFilesScreen.

I still have some work to do before my work is ready (Formatting, bug fixes, ...).
Currently, this is more of a demonstration than a cool feature 😄 .

What do you think about my work? Is it a feature that can be added?

BenBE

Thank you very much for your PR. The feature itself looks interesting and fits quite nicely with existing functionality.

But, the use of external tools is a bit problematic and is best to be avoided. For the case of retrieving stack traces, there are several libraries available, that might be worth a look (e.g. libunwind). Also I'm not sure if the code as-is would properly run on e.g. FreeBSD or Darwin. Thus I strongly prefer a solution that allows to split out these platform-dependent parts where necessary (in the case of lsof, things are portable enough across all our target platforms so it's not an issue there; not sure about eu-stack though).

While reading through your PR I noticed that this seems to handle debug information. As such, it would be nice to have the module, source file and line be available separately (where available). Also the setting of hiding path names should be respected for module filenames in order to be consistent with the rest of the UI. Taking the highlight basename setting into account would be nice too.

Another code refactoring task is related to a recent addition of the generalized columns code that @natoscott recently worked on. Please have a look there if you like.

Also there's a few further notes regarding the code which you can find below. Please feel free to rebase the fixes to those issues directly into the existing commits as you see fit.

If you need further assistance feel free to ask.

Action.c

BacktraceScreen.c

BacktraceScreen.h

MrCirdo · 2023-07-20T16:04:05Z

Thank you for your reactivity and your review.

I agree with you when you say eu-stack is not a good idea. I only used it to make a quick demo of my idea.
I'm also unsure if eu-stack is supported on the BSD platforms.

Moreover, I think also the library unwind is more appropriate. I just have to deal with DWARF information.
I will probably close all suggestions regarding the execution/parsing of eu-stack.

I saw very quickly the work of @natoscott and I will wait until his work is finished.

BacktraceScreen.c

BenBE · 2023-07-20T21:34:47Z

Thank you for your reactivity and your review.

You're welcome.

I agree with you when you say eu-stack is not a good idea. I only used it to make a quick demo of my idea. I'm also unsure if eu-stack is supported on the BSD platforms.

There seems to be some patches re FreeBSD, but I'd not actually call this support … ;-)

That's with Darwin aside entirely … ;-)

Moreover, I think also the library unwind is more appropriate. I just have to deal with DWARF information. I will probably close all suggestions regarding the execution/parsing of eu-stack.

Sure, go ahead. Maybe check for similar places elsewhere in case something similar was missed in other places of this PR.

I saw very quickly the work of @natoscott and I will wait until his work is finished.

If you prepare your PR to anticipate that change, e.g. by preparing the data structures in a way that makes this move easy, you could even prepare things now. Given the remaining aspects in that other PR will take some time to sort out, don't feel obliged to wait for those changes to land. If the structures introduced in this PR are clean enough there's not too much work later to move things over to this new interface. Most of the work actually has already been done.

Explorer09 · 2023-09-05T16:07:07Z

Please don't merge the htop-dev:main branch. Rebase it instead. You know what git rebase is, don't you?

MrCirdo · 2023-09-05T20:02:32Z

The flake commit is just for me. I will remove it at the end.
I don't start my work. However, I keep updating my branch. So it's not ready to review.

MrCirdo · 2023-11-16T18:33:09Z

Hey,

I didn't have a lot of time recently for this PR. But I have some questions.

If you prepare your PR to anticipate that change, e.g. by preparing the data structures in a way that makes this move easy, you could even prepare things now. Given the remaining aspects in that other PR will take some time to sort out, don't feel obliged to wait for those changes to land. If the structures introduced in this PR are clean enough there's not too much work later to move things over to this new interface. Most of the work actually has already been done.

I'm looking for Row workflow. I don't know if it's a good idea to use it. Row_display uses the object Settings to get fields. Does it mean I have to make my own Settings object? With all process settings (like highlightThreads)? And do I have to make my ScreenSettings? Is it okay?

BenBE · 2023-11-16T18:45:06Z

I didn't have a lot of time recently for this PR. But I have some questions.

If you prepare your PR to anticipate that change, e.g. by preparing the data structures in a way that makes this move easy, you could even prepare things now. Given the remaining aspects in that other PR will take some time to sort out, don't feel obliged to wait for those changes to land. If the structures introduced in this PR are clean enough there's not too much work later to move things over to this new interface. Most of the work actually has already been done.

I'm looking for Row workflow. I don't know if it's a good idea to use it. Row_display uses the object Settings to get fields. Does it mean I have to make my own Settings object? With all process settings (like highlightThreads)? And do I have to make my ScreenSettings? Is it okay?

The Settings and ScreenSettings objects are shared between all tabs. The Settings provides with the global settings, applicable to all screens, while ScreenSettings is applicable to only to the current tab only. As you are basically inheriting the settings from the tab you are opening the backtrace details on, you can just copy the pointer to those settings. These are held in the Screen mostly for convenience and to reduce dependency scope. I.e. if you don't need any of the values from the settings (and none of the functions you're relying on) you could even skip these objects entirely.

MrCirdo · 2023-11-17T15:44:32Z

I didn't have a lot of time recently for this PR. But I have some questions.

If you prepare your PR to anticipate that change, e.g. by preparing the data structures in a way that makes this move easy, you could even prepare things now. Given the remaining aspects in that other PR will take some time to sort out, don't feel obliged to wait for those changes to land. If the structures introduced in this PR are clean enough there's not too much work later to move things over to this new interface. Most of the work actually has already been done.

I'm looking for Row workflow. I don't know if it's a good idea to use it. Row_display uses the object Settings to get fields. Does it mean I have to make my own Settings object? With all process settings (like highlightThreads)? And do I have to make my ScreenSettings? Is it okay?

The Settings and ScreenSettings objects are shared between all tabs. The Settings provides with the global settings, applicable to all screens, while ScreenSettings is applicable to only to the current tab only. As you are basically inheriting the settings from the tab you are opening the backtrace details on, you can just copy the pointer to those settings. These are held in the Screen mostly for convenience and to reduce dependency scope. I.e. if you don't need any of the values from the settings (and none of the functions you're relying on) you could even skip these objects entirely.

Thank you very much for the clarification. I was not sure I would be able to modify it (I mean If my changes would be accepted).

BenBE

Please no custom build system files from e.g. Flake …

BacktraceScreen.c

BacktraceScreen.h

MrCirdo · 2024-01-09T09:29:14Z

Hi @BenBE ,
I completely forgot to say to not review my code. I just updated my branch.
I deeply apologize.

BenBE · 2024-01-09T12:13:42Z

Hi @BenBE , I completely forgot to say to not review my code. I just updated my branch. I deeply apologize.

Don't worry. No need to apologize. The PR is marked as draft anyway, thus the only things I remarked upon was what I noticed immediately. I also ticked off some of the previous comments that no longer apply to the current state of this PR. Basically some maintenance.

NB: This PR would be scheduled for 3.4.x earliest anyway given that 3.3.0 is about to ship anytime soon.

MrCirdo · 2024-03-16T17:35:44Z

Hi,

This PR is ready for review.
I closed all previous conversations.

Currently, I add only the support of Linux.

configure.ac

BacktraceScreen.c

BacktraceScreen.h

configure.ac

BenBE · 2024-03-17T11:00:15Z

Given that this feature will be available on multiple platforms and requires changes in different places I'd suggest to have a top-level feature flag for backtrace support and additional flags for the various libraries (iberty, eu-stack, …) The actual implementation for the stack traces should go in the platform code for each system, leaving the actual screen implementation mostly platform independent.

MrCirdo · 2024-03-19T21:04:26Z

Given that this feature will be available on multiple platforms and requires changes in different places I'd suggest to have a top-level feature flag for backtrace support and additional flags for the various libraries (iberty, eu-stack, …) The actual implementation for the stack traces should go in the platform code for each system, leaving the actual screen implementation mostly platform independent.

Ok I see what you mean, is it a good idea to be inspired by for example Machine_new?

BenBE · 2024-03-19T21:26:29Z

Given that this feature will be available on multiple platforms and requires changes in different places I'd suggest to have a top-level feature flag for backtrace support and additional flags for the various libraries (iberty, eu-stack, …) The actual implementation for the stack traces should go in the platform code for each system, leaving the actual screen implementation mostly platform independent.

Ok I see what you mean, is it a good idea to be inspired by for example Machine_new?

Maybe a better candidate may be the way in which Platform_getNetworkIO and Platform_getProcessLocks are implemented in <platform>/*Platform.[ch] …

MrCirdo · 2024-03-20T20:36:58Z

Given that this feature will be available on multiple platforms and requires changes in different places I'd suggest to have a top-level feature flag for backtrace support and additional flags for the various libraries (iberty, eu-stack, …) The actual implementation for the stack traces should go in the platform code for each system, leaving the actual screen implementation mostly platform independent.

Ok I see what you mean, is it a good idea to be inspired by for example Machine_new?

Maybe a better candidate may be the way in which Platform_getNetworkIO and Platform_getProcessLocks are implemented in <platform>/*Platform.[ch] …

Okay thank you, I'll take a look

MrCirdo · 2025-01-07T18:38:08Z

Hello,

I've improved the option used by cplus_demangling and added some safety guards to avoid exploding the memory.
Unfortunately, I wasn't able to reproduce the crash you encountered @BenBE. I tested in Ubuntu with teamviewerd but nothing. The safeguard should work.
Also, I add some messages during the configuration.

configure.ac

dragonflybsd/Platform.c

freebsd/Platform.c

solaris/Platform.c

unsupported/Platform.c

linux/Platform.c

freebsd/Platform.c

dragonflybsd/Platform.c

configure.ac

Makefile.am

Explorer09 · 2025-02-12T08:43:20Z

linux/Platform.c

@@ -681,6 +695,115 @@ bool Platform_getNetworkIO(NetworkIOData* data) {
   return true;
 }

+void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) {


It wouldn't hurt to initialize *error at the start of the function.

Suggested change

void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) {

void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) {

*error = NULL;

What do you think about adding an assert that checks if *error is not null? Is it a good idea?

assert(*error != NULL)

The goal is to prevent an error on another error.

The issue with output parameters is always how you handle pre-initialized values. As there is just a single caller, an assert(error && !*error); should be fine.

The point is, you are using the error parameter for output only, not for input. It should be a good practice for the callee to initialize the output value to something stable. Expecting the caller to do it is more prone to errors. And notice that you only call this function in one place, that extra line of initialization shouldn't add any extra code from the compiler when it inlines it.

@Explorer09 I agree with adding the initializer inside the function. My point with the assert covering both error and *error is about catching a mal-behaved caller that passes some non-NULL value for the error return value.

An alternative would be change the API for void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error); to char* Platform_getBacktrace(Vector* frames, const BacktracePanel* panel); (ignoring my comment below regarding the BacktracePanel* for a moment).

Another note. I believe the whole function should be wrapped in a #ifdef HAVE_BACKTRACE ... #endif conditional.

Also we can replace the "The backtrace screen is not implemented" dummy screen and dummy message with a compule time error (#error "Backtrace is enabled without engine selected.").

My point with the assert covering both error and *error is about catching a mal-behaved caller that passes some non-NULL value for the error return value.

I was looking at asprintf(3) that would return the output string in a similar way. If asprintf doesn't need an assertion for this, we won't need it either.

assert(error) can be covered by ATTR_NONNULL to the whole function. assert(!*error) won't do really much here, because whether *error is a malloc'd buffer should be tracked by the compiler through other means.

An alternative would be change the API for void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error); to char* Platform_getBacktrace(Vector* frames, const BacktracePanel* panel); (ignoring my comment below regarding the BacktracePanel* for a moment).

I'm not a fan of this alternative. I chose char **error because that indicates to the developer this variable is for error. I found this pattern when I learned Objective-C and NSError. Returning char * that returns an error, I think, it's not very clear on the data returned.

Also we can replace the "The backtrace screen is not implemented" dummy screen and dummy message with a compule time error (#error "Backtrace is enabled without engine selected.").

I don't understand the compute time error. If during the configuration, the backtrace feature is disabled, Does it produce a compute time error? However, it's a good suggestion to change the message...

I just realized that I prefer that error should be an optional parameter. If the developer that uses Plateform_getBacktrace doesn't want to check the error, it's his choice. But it involves checking if error is null.
Currently, it's easier to add an assert, it will make the code more readable than adding an if to every error set.

linux/Platform.c

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

BenBE · 2025-02-14T12:11:13Z

Looking at the source a bit closer, I noticed something that is kinda bad from an architecture PoV:

For void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) you are passing the BacktracePanel* panel to the platform code, which should not have any idea about any of the GUI stuff. I see this was added as BacktraceFrame* frame = BacktraceFrame_new(panel); wants the panel, when all it should require is the panel->process instead. Please update the BacktraceFrame structure to only reference the Process * it belongs to. Functions that need access to the actual panel or displayOptions can easily be provided with these parameters directly (AFAICS this should only involve code, that is called by the BacktracePanel itself anyway).

Explorer09 · 2025-02-14T12:32:07Z

linux/Platform.c

+         frame->offset = offset;
+         frame->isSignalFrame = unw_is_signal_frame(&cursor);
+
+         frame->functionName = xStrndup(procName, 2048);


One thing to be verified: Does unw_get_proc_name always NUL-terminate the string even when the length of the procName string to print is exactly 2048 bytes?

This is to prevent a potential off-by-one error. And I don't like length coded as a magic number here (should have been using sizeof()).

One thing to be verified: Does unw_get_proc_name always NUL-terminate the string even when the length of the procName string to print is exactly 2048 bytes?

It's a good question. And I just checked. Yes, the unw_get_proc_name null terminates the string.

Moreover, writing a sizeof here is a good suggestion. Thank you!

@MrCirdo According to my tracing just now:

src/mi/Gget_proc_name.c: line 100 unw_get_proc_name calls

src/mi/Gget_proc_name.c: line 49 unw_get_proc_name_by_ip

unw_get_proc_name_by_ip has two paths. If it goes to intern_string(src/mi/Gget_proc_name.c line 30), then we can ensure the string is null terminated. But there is another path that calls *(unw_accessors_t *)a->get_proc_name (a function pointer), and the path would go to...

src/coredump/_UCD_accessors.c: line 35 _UCD_accessors.get_proc_name = _UCD_get_proc_name

src/coredump/_UCD_get_proc_name.c: line 131 _UCD_get_proc_name calls

src/coredump/_UCD_get_proc_name.c: line 83 elf_w (CD_get_proc_name) calls

src/elfxx.c: line 593 elf_w (get_proc_name_in_image) calls

src/elfxx.c: line 337 elf_w (lookup_symbol) calls

src/elfxx.c: line 313 elf_w (lookup_symbol_callback) calls

strncpy from libc (strncpy does not guarantee the string is always null terminated.)

EDIT: My mistake. I missed the line d->buf[d->buf_len - 1] = '\0'; just below elf_w (lookup_symbol_callback). So the string can be safely assumed to be null terminated.

Explorer09 · 2025-02-14T12:34:51Z

linux/Platform.c

+         if (unw_get_elf_filename(&cursor, elfFileName, sizeof(elfFileName), &offsetElfFileName) == 0) {
+            frame->objectPath = xStrndup(elfFileName, 2048);
+
+            char *lastSplash = strrchr(frame->objectPath, '/');


I thought you mean "slash".

Suggested change

char *lastSplash = strrchr(frame->objectPath, '/');

char* lastSlash = strrchr(frame->objectPath, '/');

Yes 😄 . Thank you!

MrCirdo · 2025-02-21T14:49:32Z

For void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) you are passing the BacktracePanel* panel to the platform code, which should not have any idea about any of the GUI stuff. I see this was added as BacktraceFrame* frame = BacktraceFrame_new(panel); wants the panel, when all it should require is the panel->process instead. Please update the BacktraceFrame structure to only reference the Process * it belongs to. Functions that need access to the actual panel or displayOptions can easily be provided with these parameters directly (AFAICS this should only involve code, that is called by the BacktracePanel itself anyway).

The goal of passing a BacktracePanel into a BactraceFrame was to avoid using a static variable for display options (static variables can easily become a nightmare). My concern is how I can use some variables from BacktracePanel like printingHelper into BacktraceFrame_display. Do I have to use a static variable?

BenBE · 2025-02-21T15:15:45Z

For void Platform_getBacktrace(Vector* frames, const BacktracePanel* panel, char** error) you are passing the BacktracePanel* panel to the platform code, which should not have any idea about any of the GUI stuff. I see this was added as BacktraceFrame* frame = BacktraceFrame_new(panel); wants the panel, when all it should require is the panel->process instead. Please update the BacktraceFrame structure to only reference the Process * it belongs to. Functions that need access to the actual panel or displayOptions can easily be provided with these parameters directly (AFAICS this should only involve code, that is called by the BacktracePanel itself anyway).

The goal of passing a BacktracePanel into a BactraceFrame was to avoid using a static variable for display options (static variables can easily become a nightmare). But here it should avoid a bad design.

Oversight on my side. The prototype for Object_display somewhat forces the frame to contain a reference to all additional data. So placing the reference into the frame itself is not the problem.

My point regarding architecture was publishing the BacktracePanel type to the platform code, thereby introducing some dependency of the platform code on the BacktracePanel implementation. All that the platform code should ever have to handle is platform objects like processes. As we are already casting quite a few things, passing the BacktracePanel* as Object* would reduce exposing this implementation detail inside the platform code.

BenBE added the new feature Completely new feature label Jul 19, 2023

BenBE requested changes Jul 19, 2023

View reviewed changes

BenBE mentioned this pull request Jul 20, 2023

Introduce Row and Table classes for tabular screens beyond top-processes #1243

Closed

Explorer09 reviewed Jul 20, 2023

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

Explorer09 reviewed Jul 20, 2023

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

Explorer09 reviewed Jul 20, 2023

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

MrCirdo force-pushed the main branch from 0b0d5f9 to f5e3801 Compare September 5, 2023 20:01

MrCirdo force-pushed the main branch from f5e3801 to 82f41e3 Compare January 8, 2024 20:35

BenBE requested changes Jan 9, 2024

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

BacktraceScreen.c Outdated Show resolved Hide resolved

BacktraceScreen.c Show resolved Hide resolved

BacktraceScreen.c Show resolved Hide resolved

BacktraceScreen.h Show resolved Hide resolved

MrCirdo force-pushed the main branch from 82f41e3 to cfe5300 Compare March 16, 2024 17:33

MrCirdo marked this pull request as ready for review March 16, 2024 17:35

Explorer09 reviewed Mar 16, 2024

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

Explorer09 reviewed Mar 16, 2024

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

MrCirdo force-pushed the main branch 2 times, most recently from c53af06 to 792bdba Compare March 16, 2024 20:46

Explorer09 reviewed Mar 17, 2024

View reviewed changes

BacktraceScreen.c Outdated Show resolved Hide resolved

BacktraceScreen.c Outdated Show resolved Hide resolved

BacktraceScreen.h Outdated Show resolved Hide resolved

configure.ac Outdated Show resolved Hide resolved

MrCirdo force-pushed the main branch from fe07e11 to c55a3a6 Compare January 7, 2025 18:36

Explorer09 reviewed Jan 7, 2025

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

BenBE requested changes Jan 7, 2025

View reviewed changes

MrCirdo force-pushed the main branch 3 times, most recently from c9b2068 to 5e79cdb Compare January 14, 2025 16:37

Explorer09 reviewed Jan 14, 2025

View reviewed changes

configure.ac Show resolved Hide resolved

Explorer09 reviewed Jan 14, 2025

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

MrCirdo force-pushed the main branch 2 times, most recently from 2fc69d5 to 05dd74b Compare January 28, 2025 10:18

Explorer09 mentioned this pull request Feb 8, 2025

Backtrace feature similar to strace #686

Open

BenBE linked an issue Feb 8, 2025 that may be closed by this pull request

Backtrace feature similar to strace #686

Open

MrCirdo requested review from BenBE and Explorer09 February 11, 2025 18:13

Explorer09 reviewed Feb 12, 2025

View reviewed changes

Makefile.am Show resolved Hide resolved

Explorer09 reviewed Feb 12, 2025

View reviewed changes

linux/Platform.c Outdated Show resolved Hide resolved

MrCirdo force-pushed the main branch from 05dd74b to 81c5632 Compare February 14, 2025 10:46

MrCirdo and others added 5 commits February 14, 2025 11:47

Add unwind-ptrace library

ecaa5a8

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

Add libiberty library

aec56dd

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

Add backtrace screen

443efd2

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

Add backtrace Linux dependencies for the CI.

8184e26

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

Update the man page for backtrace screen

88d616e

Co-authored-by: Benny Baumann <[email protected]> Co-authored-by: Kang-Che Sung <[email protected]>

MrCirdo force-pushed the main branch from 81c5632 to 88d616e Compare February 14, 2025 10:48

Explorer09 reviewed Feb 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Backtrace Screen #1270

Add Backtrace Screen #1270

MrCirdo commented Jul 19, 2023

BenBE left a comment •

edited

Loading

MrCirdo commented Jul 20, 2023

BenBE commented Jul 20, 2023

Explorer09 commented Sep 5, 2023

MrCirdo commented Sep 5, 2023 •

edited

Loading

MrCirdo commented Nov 16, 2023 •

edited

Loading

BenBE commented Nov 16, 2023

MrCirdo commented Nov 17, 2023

BenBE left a comment

MrCirdo commented Jan 9, 2024

BenBE commented Jan 9, 2024

MrCirdo commented Mar 16, 2024 •

edited

Loading

BenBE commented Mar 17, 2024

MrCirdo commented Mar 19, 2024

BenBE commented Mar 19, 2024

MrCirdo commented Mar 20, 2024

MrCirdo commented Jan 7, 2025 •

edited

Loading

Explorer09 Feb 12, 2025

MrCirdo Feb 14, 2025 •

edited

Loading

BenBE Feb 14, 2025

Explorer09 Feb 14, 2025

BenBE Feb 14, 2025

Explorer09 Feb 14, 2025

Explorer09 Feb 14, 2025

MrCirdo Feb 21, 2025 •

edited

Loading

BenBE commented Feb 14, 2025 •

edited

Loading

Explorer09 Feb 14, 2025

MrCirdo Feb 21, 2025

Explorer09 Feb 21, 2025 •

edited

Loading

Explorer09 Feb 14, 2025

MrCirdo Feb 21, 2025

MrCirdo commented Feb 21, 2025 •

edited

Loading

BenBE commented Feb 21, 2025

	char *lastSplash = strrchr(frame->objectPath, '/');
	char* lastSlash = strrchr(frame->objectPath, '/');

Add Backtrace Screen #1270

Are you sure you want to change the base?

Add Backtrace Screen #1270

Conversation

MrCirdo commented Jul 19, 2023

BenBE left a comment • edited Loading

Choose a reason for hiding this comment

MrCirdo commented Jul 20, 2023

BenBE commented Jul 20, 2023

Explorer09 commented Sep 5, 2023

MrCirdo commented Sep 5, 2023 • edited Loading

MrCirdo commented Nov 16, 2023 • edited Loading

BenBE commented Nov 16, 2023

MrCirdo commented Nov 17, 2023

BenBE left a comment

Choose a reason for hiding this comment

MrCirdo commented Jan 9, 2024

BenBE commented Jan 9, 2024

MrCirdo commented Mar 16, 2024 • edited Loading

BenBE commented Mar 17, 2024

MrCirdo commented Mar 19, 2024

BenBE commented Mar 19, 2024

MrCirdo commented Mar 20, 2024

MrCirdo commented Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

MrCirdo Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCirdo Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

BenBE commented Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Explorer09 Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCirdo commented Feb 21, 2025 • edited Loading

BenBE commented Feb 21, 2025

BenBE left a comment •

edited

Loading

MrCirdo commented Sep 5, 2023 •

edited

Loading

MrCirdo commented Nov 16, 2023 •

edited

Loading

MrCirdo commented Mar 16, 2024 •

edited

Loading

MrCirdo commented Jan 7, 2025 •

edited

Loading

MrCirdo Feb 14, 2025 •

edited

Loading

MrCirdo Feb 21, 2025 •

edited

Loading

BenBE commented Feb 14, 2025 •

edited

Loading

Explorer09 Feb 21, 2025 •

edited

Loading

MrCirdo commented Feb 21, 2025 •

edited

Loading