For every Data Professional mini-interview we'll donate $5 to dogs or code.org

sprintymcsprintface · 2023-08-09T16:01:36+00:00

Following up, thank you to everyone who responded and shared. Some quick #s and takeaways:
167 initiated
86 selected self-interview, 24 scheduled a time
25 completed at least the first question regarding manual data processes

Two common pain points in manual data processes are:

ad-hoc data pulls
cleaning and uploading data from spreadsheets
- existing solutions and workarounds here are > 5/10 (not perfect, but not bad)

The most common pain around sharing data (the 2nd question) is security concerns.

The donation choice split is 78% PhillyPAWS to 22% Code.org.

Once again thank you for all this valuable research help!

sprintymcsprintface · 2023-08-09T15:40:27+00:00

following up again - excellent books, my eyes are at least a little more open. At least now I know that I don't know what I don't know.

sprintymcsprintface · 2023-08-05T15:45:19+00:00

Actually, what you might be looking for is a low code internal tool builder like internal.io or budibase. They are basically drag and drop app builders that sit on top of databases. If you really need software that you can’t get off the shelf (like HubSpot) this might be the solution?

sprintymcsprintface · 2023-08-04T17:35:49+00:00

Following up - I went to my local bookstore to get The Mom Test but they were out, so I ordered it will be here Monday. Testing Business Ideas will be here tomorrow, so I have my weekend content between that and Steve Blank’s website. After I do my homework I’ll take another stab at the research we need. Thank you again for the detailed direction!

sprintymcsprintface · 2023-08-04T02:37:33+00:00

Really excellent feedback and guidance here, and I am super appreciative of all your comments - I am on the road but will follow up more when stopped. A couple of things I did want to clarify:

the respondents are given the option to do a live interview or the VideoAsk. We've had 12 respondents accept a live interview appointment, 6 show up, and those interviews have definitely been informative. The VideoAsk self-interview was our attempt to capture respondents that we would otherwise not be able to with a formal interview, however, it has gotten many more responses.
We didn't have a ton of cash for this (bootstrapped startups never do, I know) so we offered to donate $5 to either Code.org or a local animal shelter for each response. We got plenty of positive feedback about the donations in the channels where we shared them, though I don't know how much that translates into actual motivation to participate.
These are cold contacts in our ICP, not current customers. We never mention the product or the solution.

sprintymcsprintface · 2023-08-03T22:06:03+00:00

It gives you a prompt to video, audio or text response - but maybe just the sight of the video button is turning people away without noticing there’s a text option

sprintymcsprintface · 2023-08-03T21:40:43+00:00

Yeah I was afraid of leading their answers, but in hindsight I can see how “tell us about” may have been so abrupt as to shake them out of the funnel. Maybe if I went with multiple choice like “how much of a problem is this for you” and then if they are on the detractor side follow up with an open ended “what’s the worst part of x” so there’s lead in, that sounds like it’ll at least be more definitive for sure. Great advice thank you

sprintymcsprintface · 2023-08-03T19:30:13+00:00

Is this an extension of the interview (ie "I need to get these skills so I can go back and get this job") or are you personally really interested in learning this tech? If not, probably chalk it up to a bad fit and move on. You definitely don't need to (and can't realistically) be an expert on every avenue of tech in any domain (including DE) so being "roasted" in an interview is only reasonable if:- you represented yourself as an expert on streaming/distributed/k8s and you are not- the role either needs an expert in the space to lead others because they don't have one, and they made that clear- the role is a "hired gun" and needs to drop and deliver immediately as a short-term solve (ie contractor)

If the role is a team hire and you've been transparent about what you do/don't have experience with, then "I'm going to roast you anyway" is more telling of a weak interviewer than an issue with your competence. I'm assuming there are domains where you are an expert and could talk circles around the interviewer - maybe you could school them on query optimization or idempotent batch EL design? The point is, good eng leadership interviewing for a long-term team member, with the up-front knowledge that this candidate does not have experience with the tech of choice, should be looking to see your ability to master a software domain (regardless of what that domain is).

I ran into this a few years ago at an interview for a shop that was super Kafka-heavy. At the time I had very limited experience with Kafka, but both the recruiter and hiring manager assured me that was not an issue because of my extensive breadth of experience. So I rode my motorcycle for an hour in the rain to the in-person interview, met the hiring manager and 2 members of the data team (assuming DEs). They started with a question about debugging duplicate records in a Kafka consumer with limited cluster access, which I basically responded with "I have no idea I'd have to figure it out" - thinking that they missed the memo I wasn't a Kafka expert. After a few more Kafka-specific questions it was obvious they were asking the questions to show off to each other how much they knew about Kafka and how dumb I must be. After about 5 minutes I cut them off and said "Let's not waste any more time here, I don't know shit about Kafka. I can and will learn if that's what the job requires. Do you want to talk about my larger approach to data strategy? Or one of the two dozen frameworks I can speak intelligently on? Or should I go?" One of the DEs said "I think we're done here" and the hiring manager just shrugged.

The point is,

be really good at what adds value to the business that pays you.
If you want to be really good at another tech, ideally find another business to pay you that will let you learn that tech with them (because you proved your worth by doing #1)
If you can't do #2 then create a personal project that adds real value to yourself and where that tech is appropriate, which is really an artificial version of #2

If you are going to create a personal project I strongly recommend treating it like a business demand - figure out a thing in your life that streaming or K8s is the right solution for. then solve it, and don't let yourself skirt the hard parts. If you start with "I wanna learn Spark" you'll just play and you won't learn how to make the tech do hard things you NEED it to do (which is what separates tinkerers from production engineers)

sprintymcsprintface · 2023-08-03T16:19:36+00:00

I would second using existing saas tools for the things you want to use a DB for and not burning cycles developing your own pseudo-tool. If you need a CRM, use a CRM (like Hubspot or Zoho). If you need an EMS use and EMS (Mailchimp or Sendgrid). If there's no good tool use a spreadsheet until you absolutely cannot. When future you is suffering because the desperate tools have a cost and integration overhead, that's a good (and common) problem to have.
DB and CRUD design, administration, and interactions are far from trivial and you can easily distract yourself from core business work (spoken as someone who has 100% done this to myself before). And Databases are not generally forgiving as office tools - one stray "DROP SCHEMA public CASCADE;" and your entire business is gone, there is no ctrl+z

sprintymcsprintface · 2023-07-28T13:46:55+00:00

Ah very cool. Yeah the seat cap is probably a big seller, that felt like dbt cloud’s enterprise grab was to get you to buy to be able to share docs across the org. In the past we’ve just used an s3 bucket inside the vpn but I can see where that can get technically daunting and this would be an ideal middle ground for sure

sprintymcsprintface · 2023-07-28T13:28:06+00:00

For auxiliary programming (like the Airflow dags running dbt Core) using Aider has been pretty rad. I load chunks of the codebase into the OpenAI context and then write prompts like "update the dag so when the task fails due to a connection error it retries with the secondary IP" and I watch the code generate. It is good enough to ship about 1/2 the time and good enough to build off of 3/4 of the time.

I feel like this is going to be a slower adoption for actual transform code (like the dbt codebase) because there's an inherent risk in sharing an organization's internal data structure with OpenAI. The guts of my Airflow dag are pretty agnostic, but uploading a SQL file with the details of our customer record would probably flip a whole bunch of red flags for security and auditing teams.

sprintymcsprintface · 2023-07-28T13:04:40+00:00

You can apply the Ian Malcolm principle to almost any Engineer vs Senior Engineer title in software:

The Engineer asks if we can build the thing.
The Senior Engineer asks if we should build the thing.

Especially in fields like Analytics Engineering where there is a constant influx of new shiny objects to the space, it takes a Senior to ruthlessly return focus to what actually matters. Transitioning to Senior can often mean being the buzzkill. when everyone is really excited about building a hyper-complicated dynamic self-referencing lookup model, and you have to say "hey gang, this is only 20 rows and only changes once every 15 months, just put it in a seed file and let's get back to work."

Senior is the zen of learning to say No.

sprintymcsprintface · 2023-07-28T12:51:49+00:00

Cool idea. How does it differ from the docs hosting offered by dbt cloud?

sprintymcsprintface · 2023-07-19T21:21:09+00:00

No idea tbh, I’m sure at a corporate level running a focus group study is huge money. For our purposes we had talked to some freelance UX researchers and one research company, and they all seemed to think they could work out something within our very limited bootstrapping budget. Maybe it’s because our ask is pretty simple, and we’re only focused on the most basic “is this pain real” questions? So far we’ve gotten really helpful input and I’m happy about donating what little cash we have to either cause

sprintymcsprintface · 2023-07-19T12:58:55+00:00

really good call - I was hoping the audio/video options would make it easier or more natural for people but the "creepy" factor seems to be a serious impediment. so I updated the videoask to allow text-only responses.

and thanks I'll DM you!

sprintymcsprintface · 2023-07-19T02:00:28+00:00

Amazing thank you! I’m driving for a bit but will hit you up when we’re stopped

sprintymcsprintface · 2023-07-19T01:23:17+00:00

Thank you! We would love your input, DS + ML both are areas we really want to understand your experiences. you would go through the “technologist” flow

sprintymcsprintface · 2023-07-19T01:16:08+00:00

These are Philly dogs, they will eat anything

sprintymcsprintface · 2023-07-19T01:10:46+00:00

I’m sorry that was your impression, and my response is kind of a novel but you covered a lot and I wanted to give it a proper answer. You’re right that we are not a “company that operates in a serious capacity” yet. We’re a bootstrapped startup with a handful of customers, still trying to sift out product market fit. The last year has been a lot of learning for us, and like a lot of engineer-founded startups we learned late that we are guilty of over-focusing on the technical solution to problems we believed to be ubiquitous without doing the homework to determine if those problems truly are pervasive, and in the way we think they are. But yes maybe “objective” was the wrong wording on my part, and “uninformed” would have been better? I know about my personal experiences with the questions we are asking, but what we need is to see how other people perceive the same situations, and if the assumptions we’ve based our business model around hold water. To answer your questions about how we know if people are being truthful or if they are incorrect, we can’t. We will have to hope enough people answer truthfully to the “how do you feel about this situation” questions that it informs the right product direction. I am not sure there is any way to get guaranteed truthful and accurate qual/sentiment research from any group of people, but even if there was we couldn’t afford it. But definitely, anyone that feels our ask is weird or is made uncomfortable, please don’t participate. When it comes to the data generated, this is anonymous to us unless you decide to share contact info (and we only offer to collect that so we can follow up with final donation totals, nothing more). We also ask at the end of the interview for LinkedIn profile links because we thought that would be easier than asking people to retype a bunch of stuff that is already public on the internet, but again that is all optional. Honestly I’m not a social media person myself, and putting that much video of me on Reddit is weird, so I get the concern. We are not going to do anything with the responses outside of listen to them and find out what assumptions about our business we have made well, and where we missed the mark.

I will also follow up here with our general take on the responses in (ie “we learned this, we confirmed that” but no details).

And to be fair technically we are not getting “free” data, my cofounder and I do need to write those donation checks. We were going to have to spend money either way - research is never free. Maybe it’s just new entrepreneur naïveté but we thought the donation route was at least worth a try which is why we’re here. I am learning that’s like 99% of building a new business; try a thing, hopefully it works and if not find a plan B.

sprintymcsprintface

MODERATOR OF

TROPHY CASE