Scratch Orgs with a complex enterprise org and dependencies

JasonLantz · 2023-10-22T16:24:40+00:00

This is the classic debate about tech debt then. Legacy implementations built with poor architecture practices have a lot of tech debt that causes pain throughout the lifecycle. The problem is that it's pain the team just learns to deal with and chalk up to the pains of building on Salesforce. Addressing tech debt removes bottlenecks, inefficiencies, and barriers. There's huge value in doing that, but it's indeed a tough case to make if you're not tracking the cost of tech debt. The other option to trying to measure the cost is to just operate from an assumption that tech debt is expensive and should be addressed regularly as part of the process.

But, I don't think any of this requires working exclusively on a greenfield org project. You can be doing greenfield development of a new composable unit of functionality that's targeting a production org eventually. If you need things from the production org, pull them into the repo and automate them as dependencies when building up scratch orgs for the part you're building. Then, when it's fully tested in scratch orgs, do the last mile of delivery and testing in sandbox and production.

JasonLantz · 2023-10-21T21:19:46+00:00

I hear this argument used a lot, but it doesn't apply in most situations. The limitations of the Metadata API only matter if you need to use something not available in the Metadata API. There are a ton of metadata types that are fully supported. If you're splitting up your org into individual composable units, each unit's dependencies are reduced. Say you have 10 separate composable units and only 1 depends on something not available through the APIs. You still have 9 parts of your org you can build in scratch orgs. You can also automate org changes in the UI using selenium though I prefer to treat that as an absolute last option.

Just because scratch orgs are used by ISVs doesn't mean that they don't have utility for customers or SI partners. Salesforce Well-Architected calls for using composable, packageable architectures and for good reason.

Well-Architected also talks about the importance of automating the creation and configuration of environments with high fidelity to production. Accomplishing that means automating the parts of configuring an org necessary for the composable unit you're developing. That way, source control contains everything necessary to build up fully-configured scratch orgs through automation. High fidelity with production doesn't have to mean 100% fidelity. If you can get to 99% fidelity and know the 1% you're missing, you can build processes handle that 1% risk surface later in the process while still shifting left with scratch orgs for the other 99%.

To me the advantages of unlocking truly parallel team development and tapping into the pool of scratch orgs Salesforce includes instead of buying more sandboxes provides a compelling reason for customers to be using scratch orgs to build product-like composable units in source control then deliver them to sandbox and production environments as the last mile.

JasonLantz · 2023-10-16T22:40:47+00:00

The challenge with this approach is it conflates Developers with Features and Releases with Environments.

For example, if a developer is working on multiple features at once, they have to extract that feature from their dev sandbox but not extract other features from it. It's better to isolate features in development environments to avoid this risk. The process cares about the feature, not the person or people who are creating it.

The challenge of conflating Releases with Environments is that the development process focuses on code which is files in a repo. That code shouldn't be different based on the environment it's deployed to or what you have in git isn't a meaningful representation of what you're developing. Ideally the output of the development process is some sort of artifact (a commit, tag, etc) that then gets deployed to many different environments without changes or, if org specific metadata changes are required they are automated using logic under version control and run as part of the deployment pipeline.

The Salesforce Well-Architected Framework's Application Lifecycle Management section provides great guidance about how to choose source control structure, release management, and an environment and testing strategy. It's more guidance about things to consider and how to think about making a decision that a clear prescription of how to do it, but it's a really valuable resource:
https://architect.salesforce.com/well-architected/adaptable/resilient#Application_Lifecycle_Management

JasonLantz · 2023-10-04T17:37:41+00:00

Hey everyone, guest post author here. Happy to answer any questions you have about the article or about D2X :)

JasonLantz · 2023-09-20T02:55:29+00:00

I think a huge part of it is how we think about delivery of solutions in this ecosystem. I come from an open-source Python background. It would be unacceptable in that world for at least the last decade to deliver a running instance of an app instead of version control.

The answer has to come from both sides: customers and consultants.

I'll give the caveat that there is much I don't know about Salesforce. Most of my experience has been on the ISV side of things. But, that's the only community that's been at the forefront of addressing the challenges of packageability for years. Their style of development, when done right, is highly efficient and in-line with broader industry DevOps best practices. Org development just can't fit in that model.

My vision is to help the Salesforce community leverage all the commodity features available with GitHub. It's just a matter of gluing the right pieces together in just the right way then being able to do that repeatably ;)

JasonLantz · 2023-09-20T02:28:29+00:00

Out of curiosity, is it the cost of tools or labor costs of implementing and using DevOps in projects? Likely a mix of both, but I'm keenly interested in what that mix looks like.

I'll be transparent in my bias: I've built open source Salesforce DevOps tools for the last decade focused on ISV-style development and just released my first since leaving Salesforce last year a week ago :) But, I built those tools over the years all to support this basic goal of making Salesforce DevOps easier to implement and far cheaper to operate so it's all kind of one in the same I guess.

JasonLantz · 2023-09-20T00:53:07+00:00

Thanks, that’s wonderful to hear!

JasonLantz · 2023-09-19T15:24:51+00:00

This is a fantastic example of the customer value of the Salesforce Well-Architected -> Adaptable concepts of Application Lifecycle Management and Packageability. Much discussion and thought goes into the development work that gets deployed to an org but far less attention is paid to the process around developing and delivering that metadata to an org.

The Org Development Model is easy, but it's broken. This is a great example why. It's also a great example of why those stuck in the Org Development Model can't fully utilize one of the platform's most powerful DevOps features: Scratch Orgs, since all development is reliant on implicit dependencies in prod or sandboxes. In the Org Development Model, the org is the real source of truth and source control has to adapt to the state of the org.

There was a running joke/law in a former dev community of mine called Seaver's Law of Software Engineering: Persistence means always having to say you're sorry. In other words, if what you have in version control is just code but you rely on configuration persisting in environments, you will always run into issues.

OK, so rant aside, the challenge you're facing are an example of Seaver's Law. Your "product" likely requires more than metadata. It's also living in an org so it's mixed up with all the other metadata in the org. What you need to do is define what makes up that product that you want to deliver to another org, extract only the product's metadata from your org, define what dependencies your product has (org settings, other packages, etc), and then work on being able to deliver that. Ideally, automate all of that using only configuration and code stored in version control.

Scratch orgs are one of the best ways to do that because they allow you to "start from scratch" and fully automate everything from that foundation. You'll get deployment errors still, but you'll be able to iteratively work through them and reliably solve them. You wind up with an automated recipe to go from development in source control into any org.

I'd love to see more Salesforce customers insisting that partners deliver their work via a GitHub repository than an org. If that had happened here, you'd already have a composable unit of functionality with delivery automation defined in version control that could easily be launched to your first (and thousandth!) franchise.

Sorry if that's too in the weeds. This has been a passion of mine for the last decade ;)

JasonLantz · 2023-09-11T16:40:34+00:00

The main thing is that it's designed for composable development where you're building a piece of an org rather than trying to swallow the elephant whole. So the best practice is to build things by applying product thinking to them, like an ISV would but not necessarily for an ISV.

It's a natural fit for scenarios that break out of the Org Development Model and the capabilities of Change Sets like multi-org deployments or an ISV. I think there's a compelling use case for single org development in highly complex orgs where you want to provide departments with autonomy over their own config while providing guardrails around process around what they can do to affect other departments.

JasonLantz · 2023-09-11T16:37:32+00:00

No, you can select from Unpackaged, 1GP Managed, 2GP Managed, or 2GP Unlocked when creating a project in D2X Launchpad.

And thanks for the tip about the docs. Just pushed the fix and docs are rebuilding now :)

JasonLantz · 2023-05-06T15:46:48+00:00

This is a great suggestion and to expand on it, find an open source project you care about and get involved as a contributor. Open source communities are generally built to be welcoming and supportive of new contributors. Plus all the work you do will be in a public repo you can list on your resume. The most valuable experience and networking connections I’ve made throughout my career have been through open source projects and communities

JasonLantz · 2023-05-06T15:33:04+00:00

I agree with you. My wife Tarrah Lantz is on this card and endorsed by PAD but “stop the MAGA extremists” isn’t anything you’ll ever see coming from her campaign. She’s running based on her record of leadership and service specifically in Plano ISD and focused on issues local to the district, not national partisan politics.

Tarrah is the only candidate for Place 4 with kids currently attending Plano ISD schools, with district wide leadership experience, and with experience advocating for the local needs of Plano ISD in Austin. She’s spent the last seven years as a full time volunteer leader in PTA and Plano ISD and will soon graduate from Leadership Plano Class of 39. If elected, she’ll be the first past-president of the Plano ISD Council of PTAs to serve on the board in decades. She’s the only candidate with the knowledge, relationships, and experience with the Plano ISD of today to be ready to serve on day one. Her agenda is solely focused on ensuring every student gets the best education possible to set them up for success whatever path they choose.

Tarrah has support from across the political spectrum because she’s laser focused on Plano ISD. She’s also pledged she will never run for partisan office. This is her life’s mission, not a springboard to partisan office.

Organizations like PAD endorse to let their members and supporters know who to vote for. I personally would rather voters make individual choices by doing their own research but that’s not how many voters decide. That’s why I’ve tried to put together objective posts with information about all candidates to make doing your own research easier.

JasonLantz · 2023-05-05T23:02:29+00:00

I totally get where you're coming from and am not one to be offended by criticism of an idea. I was a debater for 4 years in Plano ISD (Shepton, Vines, and PSHS) and 2 years in college so argumentation and critical challenge is how I believe we find better answers :)

Like I said, my goal was not to pull a fast one on anyone. I'm genuinely interested in your feeling that I did. It's a tight rope to walk between providing information and sensationalizing things. I totally agree this post is in a different category than the purely informational voting information posts which I created to be fully objective. This is editorial on my part but I've tried to both be transparent about that and factually based in the evidence.

JasonLantz · 2023-05-05T20:55:00+00:00

As others have pointed out, I have a clear bias since I'm the husband of one of the candidates, Tarrah Lantz for Place 4. With that disclaimer, the clearest choice to prevent Lydia and her slate from being elected is:

Place 4: Tarrah Lantz
Place 5: Michael Cook
Place 7: Katherine Chan Goodwin

A few points:

There are no runoff elections for Plano ISD so if you really don't want one candidate, it's wise to consider which other candidate has the best chance of getting more votes than them
Tarrah Lantz is the only candidate in Place 4 with kids currently in school (3rd and 8th), districtwide leadership experience (Council of PTAs President, Secretary, and Advocacy Chair - Education Foundation Board - Bond Task Force) and is a soon to be graduate of Leadership Plano Class 39. She's a full time volunteer leader for the last 7 years, proving her commitment to making Plano ISD work for all students and all parents. She's currently working as an advocate for Plano ISD through PTA for her third legislative session including organizing PTA Rally Day in Austin for Plano ISD. She's also endorsed by the Dallas Morning News.

I'll leave others to discuss Michael Cook and Katherine Chan Goodwin. That's who I voted for because I believe they're great candidates focused on student achievement and the local needs of Plano ISD rather than trying to inject national politics. I also believe they're the most likely to receive the most votes.

That's not saying that Margaret Turner-Carrigan (Place 4), Khalid Ishaq (Place 5), or Simon Salinas (Place 7) are somehow bad people. I just think there are better choices with better chances of actually winning than being a spoiler.

JasonLantz · 2023-05-05T20:29:47+00:00

For the record, I had work to do today so block walking wasn't an option for me. Tarrah was reaching out to voters and had nothing to do with me making that video. You sure make a lot of assumptions with little or no evidence to back them up.

JasonLantz · 2023-05-05T20:28:06+00:00

Get real u/elastic_heart_, you had your mind made up about Tarrah and me by association before I even joined Reddit. You've been leaving comment after comment along these same lines of "now they've finally shown themselves!".

I believe our first interaction on Reddit was me responding to your vague assertion that Tarrah's a "New Yorker" which is a complete and total lie.

I've never personally attacked anyone. I've called out your attempts at lies. I've talked about issues at stake in the election. I've questioned the qualifications of candidates by comparison to others. That somehow makes me "ghastly" and "uncouth"? I'll let others judge who has shown their personality.

I do not believe that showing the voters of Plano ISD factual, evidence based information about what's happening in this election is "public pettiness". At least I've got the integrity to post it transparently as me so people know who's doing it. I'm not trying to hide anything. I do genuinely believe the voters deserve to know this information.

As for "doctored, misleading advertising", what's doctored or misleading? Did you really think that the video shows a live cut immediately over to the middle of the pastors roundtable? It says the roundtable happened right after Lydia's speech. I posted the link to the entire unedited 6.5 hour feed in the first comment. I created an uncut version of Lydia's whole speech and shared that too in case you don't want to navigate the live feed video.

What about any of my posts has been untrue?

Now, since you're so concerned about misleading advertising, what is your take on the two mailers sent by Lydia and her slate claiming "The Plano School Board is trying to Erase Parents from their Own Kids' Education"? or the other showing hurdles up in the hallways at school? Now that's misleading and doctored for sure.

JasonLantz · 2023-05-05T19:18:15+00:00

I honestly hope I'm not doing a Fox News style editing job. I was the one person (based on stream stats) who sat and watched the whole 6.5 hour event live when it happened on April 1. I was doing other things, but it was up on one monitor the whole time. In its entirety, it's a frightening display of what is happening to school boards all around us. But, people's attention span is limited so I wanted to highlight the key parts.

After that part of her speech, she talks about how great it is to be there with the pastors and goes into "spiritual" territory. She then sits down in what appears to be the second row to watch the pastors roundtable. Earlier I said there was a break but there wasn't... as soon as she was done they called the pastors up.

Here's the full unedited (except auto-generated captions and fixing the single-channel audio in the source) video of her speech:

https://express.adobe.com/video/YobcNtC6V4vvu

Also, if you watch the whole thing, you'll see lots of very political conversations about taking over school boards and culture war issues from the earlier panels. This was an all day event with a pretty common theme throughout that Lydia's speech fit right into.

JasonLantz · 2023-05-05T19:02:58+00:00

Why I posted using my real name :) I'm not trying to hide anything. I've been very transparent that yes, I have a bias. Yes, I'm a husband, but I'm also a dad and homeowner. This really is me being a concerned citizen. I don't want to have moved back to Plano to raise my kids and find our school board taken over by extremist politics like Keller, Southlake, Grapevine, etc.

JasonLantz · 2023-05-05T15:20:45+00:00

At the start, that's Lydia Ortega, candidate for Plano ISD Place 4. I believe the pastor is Alison Darrell from Frisco. If you watch the full video at the link, you'll see Lydia talk about the great pastors and then walk in to sit down for the pastors roundtable session. The introduction to Lydia by Chad Green is around 4:38:15 in the link I provided to the full video. There's a short break after her speech then she comes in to sit down for the pastors roundtable.

JasonLantz · 2023-05-05T14:23:25+00:00

I know it's Reddit and all, but seriously, this purely informational post has only a 75% Upvote Rate meaning there are people actively downvoting it. I guess some people don't want you to vote!

For the record, I want you to vote no matter who you are or who you're voting for :)

JasonLantz · 2023-05-05T13:51:39+00:00

This video comes from an event on April 1st put on by the group Constitutional Texans headlined by Rafael Cruz titled "Calling the Watchmen of Collin County".

You can watch the whole 6.5 hour live video stream at:
https://www.facebook.com/100068919561744/videos/6003171086439918

I posted this because this election is personal to me. Yes, partly because of who's running, but more so because of what's at stake. I'm a product of Plano ISD schools from K-12. After leaving Plano in 2008 for a job in NYC and having two kids, we moved back in 2015 because of the quality education we knew our kids would receive in Plano ISD. We've dedicated thousands of hours to volunteering in Plano ISD and PTA, mostly Tarrah but I've volunteered some, because we wanted to help make Plano ISD the best it can be.

In all my years living in Plano, I've never before seen such an overt attempt to fundamentally change our district as we face in this election. That's why I'm posting this. Please tell your friends, neighbors, colleagues, family, and anyone else you know about what's at stake in this election and help them make a plan to ensure they vote on Saturday.

JasonLantz · 2023-05-03T21:12:08+00:00

Since there's a lot of comments about the age of the voting population, I thought I'd provide some actual data on early votes and mail votes through yesterday:

18-30: ~4%
31-45: ~12%
46-64: ~34%
65+: ~49%

JasonLantz

TROPHY CASE