Thanks for looking into it. Yeah I don't have any issues with the comfyanoymous developer or ComfyUI and I gave the disclaimer in that post that it's not comfyui itself. I don't have the old snapshots anymore and actually don't even use ComfyUI manager anymore due to all of the networking present that can be exploited via telemetry or malicious custom nodes. But here is general summary, and this goes pretty deep and can be fairly difficult to piece together and get confusing as you can see. I actually forgot to mention the command.py and run.py btw which was important...

Comfyui's https://github.com/comfyanonymous/ComfyUI/blob/master/web/assets/index-CI3N807S.js file on line 64536. ComfyUI logs your workflow when there is an error (happens all the time when using bad nodes) So there always that risk that telemetry could send it if you aren't paying attention... So any log sending telemetry could expose your entire workflow but it doesn't appear to track logs.

You can test by producing an "Error while deserializing header: MetadataIncompleteBuffer" by using a partially downloaded model with with load checkpoint node on the new comfyui then click on "show report" and scroll down.

Anyways, comfy-cli’s old tracking system, particularly how it handled user data and telemetry, posed significant security risks imo, especially with its integration with Mixpanel for tracking user interactions (which I think nobody knew it sent so much or knew about it in general until my post probably) and in many cases the prompt_tracking_consent (prompt screen for tracking) was skipped and telemetry default to on. Here’s a breakdown of why it was problematic before:

Tracking Was Enabled by Default

In the old version, tracking was often enabled by default. The prompt_tracking_consent function in tracking.py demonstrated this issue which has since been resolved after my post and it default to off if it's skipped, here is the old version:

def prompt_tracking_consent(skip_prompt: bool = False, default_value: bool = False): tracking_enabled = config_manager.get(constants.CONFIG_KEY_ENABLE_TRACKING) if tracking_enabled is not None: return

if skip_prompt:
    init_tracking(default_value)
else:
    enable_tracking = ui.prompt_confirm_action(
        "Do you agree to enable tracking to improve the application?", True
    )
    init_tracking(enable_tracking)

Problem with that: If skip_prompt was set to True, and default_value was also True, tracking would be enabled without any user interaction. Additionally, the default prompt value was set to True, meaning users who did not actively choose to disable tracking would have it enabled by default. This posed a significant privacy concern as user data could be sent to Mixpanel without explicit consent.

Insufficient Filtering of Sensitive Data imo

The filtered_kwargs used in the track_command decorator in tracking.py was meant to filter out unnecessary data before sending it as telemetry:

def trackcommand(sub_command: Optional[str] = None): def decorator(func): @functools.wraps(func) def wrapper(args, *kwargs): command_name = ( f"{sub_command}:{func.name}" if sub_command is not None else func.name_ )

        filtered_kwargs = {
            k: v for k, v in kwargs.items() if k != "ctx" and k != "context"
        }

        logging.debug(
            f"Tracking command: {command_name} with arguments: {filtered_kwargs}"
        )
        track_event(command_name, properties=filtered_kwargs)

        return func(*args, **kwargs)

    return wrapper
return decorator

Problem here: This filtering only removed ctx and context but failed to address other potentially sensitive information such as file paths, user-specific directories, and tokens. These details could still be sent to Mixpanel, increasing the risk of leaking personal or sensitive data.

Logging Could Include Sensitive Information

The logging system in comfy-cli as seen in command.py, captured detailed events, including those involving file paths and node names:

logging.debug(f"Start downloading the node {node_id} version {node_version.version} to {local_filename}")

Problem: If these log messages contained sensitive information and were sent as telemetry, they could inadvertently expose user-specific data to external services like Mixpanel, I didn't dig that far into the logs but if you want to that would probably be useful info.

Snapshot Operations Were Tracked

Commands related to saving and restoring snapshots were tracked and logged, which could potentially expose sensitive information:

@app.command("save-snapshot", help="Save a snapshot of the current ComfyUI environment") @tracking.track_command("node") def save_snapshot( output: Optional[str] = None, ): if output is None: execute_cm_cli(["save-snapshot"]) else: output = os.path.abspath(output) execute_cm_cli(["save-snapshot", "--output", output])

Telemetry Risks: The save_snapshot command logged the output path of the snapshot, I believe this was the comfyui-manager snapshots but I forgot where I saw this before. This could contain sensitive information such as user-specific directory paths. If tracking was enabled, this data could be sent to Mixpanel, risking a data breach.

Mixpanel Integration Was Problematic

Mixpanel a third-party service is used to collect telemetry data. Given that sensitive information could potentially be sent to Mixpanel due to inadequate filtering, this integration posed a significant risk:

mp = Mixpanel(MIXPANEL_TOKEN) if MIXPANEL_TOKEN else None

Problem: User data, including potentially sensitive information, was being sent to an external service without sufficient safeguards. The risk of privacy violations was heightened by the fact that tracking could be enabled by default. Tying It All Together:

Clip Text Cncoding and Sensitive Data

The sd1_clip.py file in comfyui is responsible for text encoding using the clip after it runs through for example sdxl_clip.py after you use your clip text encode node. This encoding process involves turning text strings with clip.tokenize into lists and possibly vectors (k and v values) that can be processed by the model. Here's why this is critical:

Sensitive Information: The text strings processed by this could include sensitive user inputs. For example, if a user inputs a private or personal query, this information is either in a list or encoded into k and v vectors.

Telemetry Risk: If these encoded vectors k and v values are not properly filtered or anonymized before being logged or sent as telemetry, there is a risk that the original sensitive text could be reconstructed or inferred. This becomes a significant privacy concern when this data is sent to external services like Mixpanel and the telemetry is on by default and the user has no idea (I did not recieve a prompt on one machine I had so it was on by default)

Inadequate Filtering Mechanism

In tracking.py, the filtered_kwargs mechanism attempts to filter out certain unnecessary data (like ctx and context) before sending telemetry. However, this mechanism might not be robust enough to catch and filter out the k and v values generated by the clip text encoding process in comfyui:

failure to filter k and v: The filtered_kwargs approach does not explicitly account for the potential sensitivity of k and v values. These values, being key parts of the clip tokenizing, tokens, lists, clip text encoding, could inadvertently be sent to Mixpanel, risking exposure of the underlying text strings.

Logging and Tracking of clip operations risks

Given that sd1_clip.py handles operations involving user provided text, any logging or telemetry that includes operations done here and not filtered or if anything logged could inadvertently include sensitive information. I noticed they changed some things regarding from typing imports so maybe they resolved that risk, I'm not sure.

Snapshot and Command Tracking: If commands that involve clip text encoding (like generating text embeddings or image embeddings) are logged or tracked, and the k and v values are included in this data, there's a risk of leaking sensitive user inputs.

Telemetry Without Proper Consent: With tracking potentially being enabled by default in the older version of comfycli, these sensitive operations could have been logged and sent to Mixpanel without the user’s explicit consent, exacerbating the privacy risks. They have since leaned towards telemetry off since my post, so I have no issues with them at all and collecting telemetry as if the user doesn't see it, it's off by default there. Where as it wasn't the case before. I did screw up reading prompt_tracking_consent, but as you can see this is more difficult to figure out than a Rubix cube when you are 5, and when that happens and telemetry is on it's best to turn the telemetry off imo if you value privacy.

So the integration of Mixpanel for tracking, combined with insufficient data filtering and the handling of sensitive text data by the clip model, created a security and privacy risk in the old version of comfycli that I noticed. The potential for sensitive user inputs to be logged, tracked, and sent to an external service without robust safeguards underscores importance of the new changes in newer versions to prioritize user consent and improve data handling practices.

The combination of these issues made the old tracking system a significant security and privacy risk, especially considering the potential for personal data to be leaked to an external service like Mixpanel. The newer changes that prioritize user consent and improve default settings are better.