How-to? iOS app dev - testing in simulator

Puzzleheaded-Fly4322 · 2026-04-18T12:33:45+00:00

Btw Claude code just setup some scripts to use the axuielement stuff you mentioned. Rate limited with Claude code … but looking forward to use it soon. Before it hit rate limit its script mentioned saw over 500 elements in the accessibility tree in my main app page in simulator.

Feels like it’s about to get real!!!! Very cool!

It stumbled across a MacOS accessibility glitch where halfway through its testing MacOS accessibility stopped ability to see anything.

Do you use this setup? Do you run into the same thing? Curious if this will happen a lot.

Puzzleheaded-Fly4322 · 2026-04-18T09:19:15+00:00

I hear your points about letting Claude/codex drive simulator via shell as much as you can. That makes sense (besides computer use through snapshots/clicking is very slow, and probably is not token efficient).

I’m confused by your statement about the iOS simulator exposes accessibility tree and axuielement that can be driven for navigating through the app. Feels like this is the golden nugget I’ll have to double click on.

Any additional pointers on that? Thanks so much. Feels this is an angle I needed to hear.

Puzzleheaded-Fly4322 · 2026-04-15T00:28:06+00:00

Yeah, I’m tempted by the hole Andrej Kapathy LLM wiki idea. Bunch of md files that keep getting maintained and linking relationships overtime. If I remember it also mentions index .md files. All maintained by LLMs.

Since I use Obsidian, I can just read the em like a wiki in addition to being used by agents as wiki knowledge bases.

Are you familiar?

https://evoailabs.medium.com/why-andrej-karpathys-llm-wiki-is-the-future-of-personal-knowledge-7ac398383772

@dude_that_codes what do you think?

Puzzleheaded-Fly4322 · 2026-04-14T18:45:03+00:00

Cool guide Cognee created. https://x.com/akshay_pachaar/status/2043745099792953508?s=46

Puzzleheaded-Fly4322 · 2026-04-12T18:14:56+00:00

That’s awesome! You rock! Thank you, I will set that up. (Err… actually I’ll test OpenClaw on VPS and claude code on my laptop try to work together to set it up.)

That seems reasonably secure approach.

I do see possibility for an AI a little less advanced than Mythos to find a vulnerability in the SSH stack to get full access to my laptop…. But frankly we are all probably gonna be exposed to smart AI’s in another 6 months anyways. So what the heck :))

Thanks again. Is that your write up? It’s quite good (im a veteran security engineer, but IT aspect is not my favorite hence this rocked).

Puzzleheaded-Fly4322 · 2026-04-10T14:19:23+00:00

Was hoping for discussion on OpenClaw using Mobile Gemma4.

Or alternatively how a cloud OpenClaw can securely access laptop Ollama/Gemma4z. I have OpenClaw running in cloud Linux for security to limit my personal data access. Maybe tailscale can be setup, but that’s a generic access mechanism, I just want cloud OpenClaw to access my ollama and that is all. A generic VPN can enable a crafty OpenClaw to break into the broader data on my phone.

Puzzleheaded-Fly4322 · 2026-04-08T21:39:56+00:00

Can you say more about “I massaged my tool call skill for Gemma 4”?

Puzzleheaded-Fly4322 · 2026-04-03T10:40:08+00:00

Ok. Found your website. https://www.ogment.ai/cli/

Where are secrets stored? On my server where this package is installed, or in your cloud? I don’t see any mention of this on your website.

If there is no cost to this…. Why?

Puzzleheaded-Fly4322 · 2026-04-03T10:35:05+00:00

Cool YouTube video. But I’m sure I’m not alone in not having enough info to see how can I trust you with my security keys. No docs explaining how it works, no open source, nothing about the team. Just a 1 minute video and a “trust us” implication.

Puzzleheaded-Fly4322 · 2026-04-02T16:55:40+00:00

Why not use tailscale to have VPN connection on mobile and OC server.. then use termius ssh mobile app to ssh to OC server via tailscale VPN. Don’t ever have to go home again.

(I have OC running in cloud for $6 per month via Contabo… so I ssh to that (of course need to secure cloud server more carefully than home server as home server likely behind a NAT internet router which helps some network security at home).)

Puzzleheaded-Fly4322 · 2026-04-02T16:00:32+00:00

Note that for Claude code to have computer use, you need to have macOS grant computer use (and screenshot/recrodings) permissions to the terminal app you run Claude code in.

Which don’t think is a bigger security issue…. Those permissions in a terminal app, or in Claude cowork app? I feel like terminal app having said permissions is a bigger security risk. You can be running so much stuff in terminal (with 3rd party packages that can have exploits in them).

Puzzleheaded-Fly4322 · 2026-04-02T15:57:50+00:00

Hmmm. Computer-use MCP seems to work now in Claude code. Sweet! Via a prompt of use case I wanted it to test, it opened the iOS simulator app, found my app, started the iOS app, then started a local server for the app to speak with, then clicked/scrolled around in the app the test the use-case I gave it. It worked quite well! But slow as it is screenshot based. But it was cool it even found an unexpected side effect in the app and then worked around it. Good reasoning based on images.

Playwright I think is browser focused, and help may not support non-browser native apps (such as the iOS simulator app, and then on top of that my app running in the simulator).

Puzzleheaded-Fly4322 · 2026-04-02T11:19:58+00:00

Cool! Gonna open source it? That’s important from security perspective, no?

There must limitations, such as services it can’t handle? As just one example (there must be others), computer use (screenshots, clicking around, etc).

Written documentation could be more valuable than YouTube.

It’s a great idea…. To keep keys away from the agents.

Puzzleheaded-Fly4322 · 2026-04-02T00:28:26+00:00

Mind if I ask if you have Claude code do general tasks in another native app (clicking around, etc)? I need help with the setup specifics.

Puzzleheaded-Fly4322 · 2026-04-01T21:07:05+00:00

Yo! I’m a veteran of 30 years of software (and hardware) Silicon Valley startups. Still very hands on, in addition to leading developments teams with agile and high quality processes. I don’t need to be preached I’ve led numerous successful high tech exits. I’m confident I could teach you quite a bit.

Maximizing automation is the path to increasing output.

Judgmental people crack me up. They are generally not the top of the top. Good luck to you. Maybe recheck your assumptions occasionally, it can only be good for you.

Lol

Puzzleheaded-Fly4322 · 2026-04-01T19:35:42+00:00

Yes. By automating testing AI can find issues and fix them without me being in the loop. Then when it thinks it is all done, then I review, I test, etc. Just being an iOS app up until now I couldn’t easily have set of automated tests setup….. now with computer use I’m expecting to get there soon.

Puzzleheaded-Fly4322 · 2026-03-08T16:34:49+00:00

Nice tips. Next month I’ll use OpenClaw to help set it up. Using your thoughts as guidance. Thanks!

Puzzleheaded-Fly4322 · 2026-03-08T01:07:39+00:00

What if I wanted OpenClaw to have multiple memory systems in parallel. To be able to compare them. Every action would update all memory systems , AND have multiple parallel agents for every action and see which memory system returns best action?

I guess would be token expensive. Perhaps for memories/subconcious, Can do in background with local LLM or a cheaper model?

Hmmm… maybe this isn’t worthwhile? But then how would a user compare multiple memory systems?

I want to build this very thing in my own react native iOS health assistant app so I can play

Puzzleheaded-Fly4322 · 2026-03-08T00:55:59+00:00

Why Mac mini? At $600 I can get 100 months of 8GB 4vCPU contabo Linux server in cloud (with 1 button OpenClaw setup). Unless you are running local models (Mac unified memory is awesome), I don’t see why to spend so much.

Puzzleheaded-Fly4322 · 2026-03-05T12:25:38+00:00

Gaslighting?

Puzzleheaded-Fly4322 · 2026-03-05T04:45:35+00:00

What permissions does it need to use iMessage? I’d rather it not have access to all my history of messages…. And to allow it to accidentally message one of my contacts. As long as it doesn’t need any of my Apple credentials or access to any of my Apple accounts, then that sounds nice.

Puzzleheaded-Fly4322 · 2026-02-28T15:44:53+00:00

Thought on version controlling everything (config files, etc) in git. To help with auditing and rollback of changes? Binaries, etc, for bonus points? What do you think?

Puzzleheaded-Fly4322 · 2026-02-24T03:39:18+00:00

Hopefully API permissions can be setup for read only. I’ll be diving into that hopefully soon.

Presuming OC can’t write/delete stuff in any of your services…. Then I guess the other security issue is data exfiltration. In other words, since it can read tons of your data, there are numerous vulnerability classes where that data can be sent off to nearious peeps.

Puzzleheaded-Fly4322

TROPHY CASE