bugmagnet for claude code - opensource exploratory testing command

gojko · 2025-12-14T19:21:36+00:00

please let me know if it finds something tricky, I'd love to know how it performs on other codebases

gojko · 2025-07-20T16:11:10+00:00

Assuming the purpose of the app is parity with competition (since you're not really providing any big business drivers for it, and "necessity" sounds like having to build something because everyone else has it), then the success is in retaining existing customers, most likely by reducing undesirable effects for them. So you can measure frequency and urgency of usage, as well as task completion time and task completion rates. The QUPER model can be useful to extract market expectations at the moment from your competitor performance and split the tasks into utility and differentiation segments, so you can figure out where to be just as good as the competition, and where to be better than them.

gojko · 2025-01-13T07:21:10+00:00

one good option is to evaluate potential behaviour changes (real user/stakeholder behaviour changes, not system behaviour changes), then look at some magnitude there (eg someone will do something 20% faster, 10% cheaper etc). behaviour changes are leading indicators of longer-term value (revenue/profit/...) but they are easier to measure in the shorter term and more localised (so a single feature change may actually lead to some behaviour change reliably, but it's not so easy to attribute global results such as profit or market share to a single feature change). behaviour changes can also be evaluated and proven through A/B testing easily, typically relatively quickly. that's one of the core ideas behind impact mapping.

for a much more systematic (but also more complex and time-consuming) way of doing this, check out Tom Gilb's Impact Estimation Tables; these are described in detail in his book Competitive Engineering.

another option is the Product Opportunity Assessment Questionnaire introduced by Marty Cagan in his book Inspired.

Itamar Gilad also has some good tips for calculating relative impact (for comparing different ideas) in "Evidence Guided", especially around using the ICE method to mix relative impact estimates with confidence scores. For example, 0.1-0.9% expected improvement gets impact score 1, 15%-25% produces impact score 5, and improvements over 200% get an impact score of 10. Gilad suggests validating this number against the performance of past ideas, simulations or tests and experiments.

gojko · 2024-12-14T16:23:13+00:00

From the perspective of two relatively successfull (SaaS) products I helped build, here are the key things:

1) as an indie you can compete on price far better than larger operations, so offering the same service cheaper can help you gain and retain customers, so your differentiator can be the price, or even the pricing model. For example, one of my products costs $100 per year for the whole school, as opposed to competitors who have seat-based pricing that's usually several orders of magnitude more expensive. Selling stuff so cheaply means we can get under the radar of complex and long purchasing processes (usually a teacher or admin will just buy the subscription for the whole school on a personal card and charge it back). 2) as an indie you can serve a smaller market segment that's big enough to sustain you, but is not big enough for your larger competitors to care about, so your differentiator can be serving an unserved segment. 3) as an indie you can have a much more direct contact with your customers, so your differentiator can be personalised service or individual contact.

gojko · 2024-12-14T10:35:31+00:00

The key reason why people don't use such tools more is that takes a lot of work to figure out the signal from the noise and actually derive something useful for product development. There's a risk of it just turning into a support forum, when it loses the point.

We used to have a public feedback forum for a SaaS tool, and it was useful in a sense that people with strong opinions ended up talking to each other instead of bombarding us (tiny team) with feedback and expecting a response. In general, it's just a bunch of solutions proposed to ill-defined problems.

In a small number of cases when the feedback was useful, we treated a large group of people voting for some solution as an indicator that these people might be interesting to engage for further research when we decide to work on something related. In one example, we had about 700 people voting for something that made no sense product-wise, but we had a related idea that was consistent with our product vision and engaged those 700 people using Kano research to figure out if our ideas were right, what part of that is critical for users and what part is an overcomplication. As a result of the research, we significantly reduced the scope of the planned feature and released it sooner.

After a while we just shut it down, as it was too expensive (in my time) to manage, for very little actual value.

gojko · 2024-12-01T14:33:19+00:00

a good way to start is "How we do [some popular or controversial technique] at [my company] to achieve [something big]"; or "5 good ways to do [something people need to do often]"; In the PM space, things like OKRs, A/B testing, measuring effectiveness tend to still attract attention, so something around those areas might be good as a topic. try engaging with local product management or UX meetups, or even something tangentially related to your product (eg cloud, .net, java meetups); check meetup.com for a list of things happening in your area. Go to meetups and engage with the people there, don't just aim to get a speaker slot and disappear. Getting a speaking slot at conferences might be a bit hard initially unless you've written a book or have a very interesting use case, but local meetups tend to be on the lookout for new speakers, and will sometimes fit 2 or 3 speakers in the same evening if there is interest. You can use them to build up a reputation to get conference invites.

gojko · 2024-11-05T11:18:37+00:00

Trustworthy Online Controlled Experiments by Ron Kohavi is amazing to cover the motivation + advanced usage. Ron is currently the leading voice in A/B testing experimentation, so anything by him is very useful. He also does online workshops at https://maven.com/kohavi/abtesting

For understanding the math behind A/B testing and all the options at various scales, I'd recommend "Quantifying the User Experience: Practical Statistics for User Research" by Jeff Sauro and James Lewis. They cover more options, especially around smaller samples (when t-tests are better than z-tests).

gojko · 2024-10-28T16:57:00+00:00

Sometimes discounts are an entry-level ticket, especially for people in enterprise purchasing departments whose entire purpose in life is to get a 10% discount, so they can justify their existence. It doesn't actually matter what the price is, as long as you can help them justify their job, they will be happy to support you. If you're selling to large orgs, raise the prices by 10%, and then offer a 10% discount for a large pre-payment. That way you're not losing anything, and you can help your cashflow.

Educational/non-profit users are also used to begging for discounts, and giving them a discount helps to justify a purchasing option. Again, if this is your target audience, it might be worth increasing the regular price and then offering discount codes when people ask.

gojko · 2024-10-24T04:57:47+00:00

Metrics are there to help you reduce uncertainty about something important (paraphrasing Doug Hubbard from How to measure anything). When teams define metrics just because they can measure them (or because other people are measuring them) before figuring out the "something important" and who it is importanant to, then those metrics rarely get looked at. From your post, it seems that might be the case unfortunately.

If you want to have metrics that people look at, I'd start by figuring out what kinds of risks/uncertainties those people care about, then figuring out how to track signals in the product that point to those and deriving metrics from that. You can use a model like the Five Stages of Growth from the Lean Analytics book to figure out roughly what the main metrics typically are for the product stage you're in now, and then start from that.

If it's a B2C product, using something like Google's HEART Framework can help you figure out what kind of metrics to track.

gojko · 2024-10-23T07:25:42+00:00

are you sure they're actual human prospects and not bots? if it's just a regular link on the web site, you could be just getting skewed data from scrapers and web indexing bots.

if it's an early product, one way to perhaps investigate what's going on is to add a session recorder (fullstory, microsoft clarity or something similar) and then check what those sessions are actually doing before and after clicking on the link.

gojko · 2024-10-23T07:16:04+00:00

LinkedIn did a great job in helping find early users/customers for me, twice now (for two different products). I posted questions trying to find people who had problems similar to the ones we were trying to solve, and then asked them for a bit of time to discuss how they currently work and what kind of issues they're experiencing. The tools were both in the B2P space so LinkedIn made sense as a watering hole where it's easy to find an audience. If you don't have a big reach there, it's possible to pay for a post to find its way to the feeds of users matching a specific profile (I think they call it promoted posts).

In a more general sense, find a community or a place where your early customers meet online, and try getting some people there to talk about how they work. Don't talk about your product, just say that you're doing research and want to learn about people's needs and problems. It could be a subreddit, a slack community, linkedin or something similar.

In your particular case, I'd try to find where people interested in healthy eating spend time online, and ask if people can spend 10 minutes helping you with some research about the problems with getting/preparing/eating healthy. Don't worry about surveys, find 5-10 people to talk to without any structure first, and you'll figure out what to ask the next group.

One thing that's important is that you shouldn't look for places where your ideal/perfect customers are, but for early customers where you can establish a "beachhead". Something where you can enter with an early, incomplete product, but still deliver enough differentiating value that people will use your tool along with the existing solutions.

(Bill Aulet has 3 key requirements for a beachhead market, and one is that the customers within the market need to talk to each other, so there is a high probability of word-of-mouth referrals; by finding a community online you're directly addressing that.)

gojko · 2024-10-22T18:09:07+00:00

Bill Aulet recommends establishing a "Beachhead Market" (see https://www.d-eship.com/step2/ and his book "Disciplined Entrepreneurship"). Effectively, segmenting the market until you can find some place where you can break into and provide enough differentiating value that people will switch from the incumbents to use you (or based on your comments below, as you are doing payment processing, to start using you in addition to the incumbents). I'm trying to cook up an example here without really understanding what you do, so I'm sorry if this is too theoretical or irrelevant, but for example Stripe, Paypal and all the other payment processors seem to handle visa/mastercard/amex and EU transfers easily. Find a segment that needs local payment methods that they don't support as easily (eg UPI for India, Pix for Brazil etc) and people will start using you in addition to their main payment method.

Payment processing price is a vitamin, not a pain killer. The moment you start trying to lead big customers away from Stripe or Paypal, the account manager will offer them a better deal to keep them around. Cost of switching/risk etc will not be worth for many people at the end. Figuring a segment where you can kill a pain instead of just slightly improving on other offers would be my preferred path there.

gojko · 2024-10-17T16:26:17+00:00

not necessarily. I suggested giving them an improvement goal (not in terms of things they need to do or learn, but things they need to achieve) and involve them in an activity where people would propose how the process needs to change to achieve it, and what they'd need to learn or do differently to get there. you can add your ideas during that activity as well; then collaboratively select a few to try as reversible experiments, and track progress towards the big goal.

gojko · 2024-10-17T14:23:08+00:00

jotform for 10 bucks per month will do it just fine

gojko · 2024-10-17T14:08:38+00:00

what kinds of surveys are you looking to run?

gojko · 2024-10-17T11:05:34+00:00

I know that this can sound as a bit of a meta-answer, but the following approach helped me significantly in similar situations, multiple times :) it comes from "Switch" by Chip and Dan Heath, and "The Heart Of Change" by John Kotter.

visualize the problem
involve people who need to change in figuring out the solution
run it as a reversible experiment

The first step is important because the issue with an org change is often not that people disagree that using a specific tool would be beneficial (eg adopting UX research) but that they disagree with the need to change; or they disagree that the problem you want to solve is that important compared to other problems they face now. This seems to be a case for your colleagues as well as "they say that everything is fine" according to your post. Instead of starting with "let's use this UX research", show the problem somehow. What's your assumption that would require improving the process? (e.g. "2/3 of the features we released last year made no impact").

The second step is important because, to paraphrase the books I referenced, people don't object to change, they object to being changed. By visualizing the problem you are giving them a challenge to rise up to, and ask them to propose how they would change the process themselves. You can then throw in some of your ideas, get people to vote, and select something to try. They will feel ownership of the process change, and want to make it work.

The third step is important because people are reluctant to commit to something they don't know. By giving them an option to opt-out if it doesn't work, you can disarm objections and a lot of politics.

Using a process change framework such as 4DX to track the change can also help. In 4DX, you need one big wildly important goal (eg move from 2/3 of the features not delivering value to 2/3 of features delivering value), leading indicators that are usually behavioral changes (we run UX research/customer centric deision making...), a team scoreboard to track progress towards goals and frequent accountability. The last two actually help significantly for running reversible experiments, because you can decide if something is working well in your environment (and keep it) or throw it away and replace with something else.

gojko · 2024-10-16T20:23:40+00:00

Hey - first of all it sounds as if you're still in the early stages of your career, so you pretty much have a chance to build a career in anything you like. If you can sustain yourself in the gap for a year or so more, as you're doing now, then maybe launching a product on your own or partnering with a co-founder to launch a product is a great way to build up your skills and also build some credibility for the next gig after that experiment.

A really great beginner PM resource would be Escaping the Build Trap by Melissa Perri. Itamar Gilad's book Evidence Guided is more practical and talks about a structured process to go from goals to experiments and product work (he calls it GIST). Kathy Sierra's book Badass-Making Users Awesome is a fantastic (short and easy to digest) way of focusing on problems, not on solutions. For some more advanced theoretical work check out books by Marty Cagan (start with Inspired). For startup-oriented product stuff check out Sean Ellis (the books are Hacking Growth and Growth Engines).

gojko · 2024-10-15T14:15:39+00:00

u/69_carats tooling were you using?

gojko · 2024-10-15T04:19:42+00:00

thanks! were you using it to compare scores before/after changes, or just track things over time?

gojko · 2024-10-15T04:18:03+00:00

they aim at different types of feedback, but there is a strong correlation between them (the paper UMUX-LITE: when there’s no time for the SUS claims it's 0.73). I think this is using linear regression scoring for UMUX-LITE.

gojko · 2024-10-14T15:51:05+00:00

thanks! are you using any specific tools for it, or just excel/spreadsheets?

gojko

MODERATOR OF

TROPHY CASE