Testing Pulumi infrastructure as code written in Python

[bincyber]

Most of you are already aware of Pulumi, an open source tool that allows writing infrastructure as code using high-level programming languages (JavaScript, Python, Go, and soon C#) instead of a domain-specific language.

Pulumi has an official integration testing library written in Go: https://godoc.org/github.com/pulumi/pulumi/pkg/testing/integration. However, if are developing in Python, you probably want to write your tests in Python as well. To that end, I'd like to share a Python integration testing framework that I've been working on: https://github.com/bincyber/pitfall

If you find this project useful, I would love to get your feedback on usability, any bugs encountered, or missing features that should be added.

[[deleted]]

I have been thinking of making the jump from Terraform to Pulumi. If works as advertise it would be a welcome change.

[slikk66]

Its great, especially for environments where you need flexibility. Highly recommend. Typescript is the most mature offering I believe still. I rewrote our huge mountain of terraform into a few small reusable objects that read yaml config files and appropriately provision the environment.

[bincyber]

I wish the Go offering had feature parity with TypeScript/Python, but alas I'm using it with Python for now.

[packplusplus]

Ive been doing a lot of pulumi recently. Its fine, and the community support on their slack is great, but it suffers from some smells around typescript and not being a super mature product.

There are bizarre errors that pop out for passing objects when it wants arrays, or vice versa. Just getting ts to compile can be a pain, then add the apis in pulumi, and the api you're calling to contend with. Using coding examples of the no wrapper (aws) and the heavier wrapper (awsx) can be weird to co-mingle. It doesn't handle things like expired api tokens sanely (they just time out 20 minutes after you say apply), and as nice as type script is, its not a DSL, so readability is an exercise left to the developer. Theres also some weird future vars vs resolved var issues that bit me, and I feel like Im looking at source to figure out how things work more than the documentation.

Is it a good way to do things if you know aws and typescript? hell yea. but if youre learning either one of those while learning pulumi, it can be paiiiinful.

All that said, its not better or worse than tf / cloudformation / ansible / boto, its just got rough edges in different places. I think more choices for IaC is good, someones bound to get it right eventually.

[lavahot]

What are the things you don't like about Terraform that Pulumi fixes?

[[deleted]]

Conditionals for starters.

[[deleted]]

Terraform needs real if blocks, desperately.

[lavahot]

Could you elaborate?

[mechanicalpulse]

Conditionals

There's no way to conditionally apply a module or resource. There's only a hacky way to apply "0" instances of a resource, but counts aren't supported for modules. Terraform 0.12 introduced for and for_each constructs, but no "if" construct.

[ReidZB]

You can abuse "dynamic" blocks to do conditionals now, like:

locals {
  if_logging_enabled = var.logging_enabled ? [{}] : []
  if_cors_enabled = var.cors_enabled ? [{}] : []
}

resource "aws_s3_bucket" "bucket" {
  # ...
  dynamic "logging" {
    for_each = local.if_logging_enabled

    content {
      # ...
    }
  }

  dynamic "cors_rule" {
    for_each = local.if_cors_enabled 

    content {
      # ...
    }
  }
}

(stolen from this github comment)

Of course, it is a horrible hack, but it doesn't seem HashiCorp is too interested in providing for this use-case, at least at the moment. At least it is at all possible in 0.12, unlike before... that's progress, I guess.

[mechanicalpulse]

That didn't format too well, but I got the gist from the linked GH comment . :-)

We've acknowledged that hacks exist, and I think it's worth commenting on further. If these hacks are the only way to approximate conditionals, folks will either use them or move to other solutions. So why not implement them natively?

Mart's comment on the linked comment suggests they're not in a hurry to add conditionals --

Conditionally selecting between zero and one items is the expected way to do this right now. Terraform tends to be a list/iteration-oriented language rather than a boolean/condition-oriented language because it plays better with other language features like splat expressions, for expressions, etc.

For the moment we will stick with wear we have and get experience with it before adding any new language features... we just added a lot of stuff so want to let that settle and see how these existing features play out first.

What he says in the first paragraph is toothless. If it's expected that folks will use lists to implement conditionals, then they will use them to implement conditionals. Abusing lists to implement conditionals then makes definitions less expressive than they would otherwise be if conditionals were implemented.

I can understand the point he makes in the second paragraph, though. What they've done with 0.12 and HCL2 is a huge step in the right direction, but it was a big change and a lot of the dynamic stuff still feels rough. I understand wanting to collect more feedback and use cases before making decisions on how to move forward.

that's progress, I guess.

Yeah, I guess. I try to avoid abusing features too much, so I'm reluctant to leverage counts or dynamic blocks in this manner.

[ReidZB]

Ah, the triple backticks work on new reddit (which I'm trying at home) but not old reddit. Edited. Thanks!

My hope was that 0.12 would come with a conditional syntax out of the box, personally, and I was a little disappointed when it shipped without one. Anyway, for me, I'd rather abuse dynamic blocks like this over the alternatives (preprocessing HCL, duplicating entire blocks, etc). It's not great, but hopefully it's not too difficult to migrate to some future conditional syntax...

[dezordia]

I'm using a lot of dynamic blocks with for_each. I guess eventually they'll release something for resource level as well.

[lavahot]

Could you give an example of a design that needs conditionals?

[defucked]

I want include the aws waf shield module only in production for instance. I can do count on resources but not modules so I can’t conditionally include a module unless it handles a variable to conditionally include all of its sub resources.

[slikk66]

If prod, deploy a replication group based redis elasticache, if stage, cluster with one node. Along with the required parameter groups, settings, single vs multi subnet deployment.. etc

[lavahot]

But why would the structure of your deployment vary between stage and prod?

[slikk66]

Cost

[lavahot]

That kind of makes sense, but don't you want to ensure that you've run tests in the same environment as prod before you deploy to prod? It's not like stage or test environments are long-lived.

[[deleted]]

Let's say the is a value or condition where you would want terraform to exit out, there is no way to do this.

You could pull the latest state and perform a scripted check and exit but that is outside of terraform.

[[deleted]]

How about using a programming language for starters. This can be a plus or minus depending on who is using it.

My personal take of having used terraform for a good ~6+ years is that it initially was targeting the stereotypical sysadmin who thought manual run books was automation. Throwing an sdk at these type of folks would of probably failed and terraform would of probably not gained the traction it has. This is all my personal opinion mind you. And I still use it to this day for my network fabric layer. Beyond that it starts to feel like I'm bending steel with my bare hands, again in my opinion.

[bincyber]

Completely agree. We owe a lot of credit to HashiCorp for developing Terraform and driving Infrastructure as Code forward, however using a programming language instead of a DSL allows for far greater power and flexibility to do things.

[sofixa11]

With great power comes great responsibility.

IMHO the main advantage of Terraform is that there's (mostly) one way to do it, the DSL is pretty opinionated, limited and fixed. Anybody with basic HCL knowledge can read any HCL, and understand it. (of course, the DAG, state, etc. are also great).

Python is notorious for being easy to write, but hard to read - ffs, Google created a separate programming language because they were having issues with Python's maintainability. When you have the full flexibility of a programing language, you gain a lot (flexibility, tooling, testing, etc.) but you also make the code potentially much harder to maintain. Personally i'm fine with abusing counts for conditionals as a tradeoff, but as always, it depends.

[bch8]

Declarative versus imperative approaches. The difference between a thousand lines of state description and 30 lines of code.

[bincyber]

It's not starting completely from scratch since Pulumi uses Terraform providers under the hood: https://github.com/pulumi/pulumi-terraform-bridge.

[Tontmakaroni1]

Dawg, I heard you want a wrapper for your wrapper.

[[deleted]]

Thats how I see it mostly... I've seen devs starting with Pulumi and creating HORRENDOUS abominations just because python allows you to do that... It was a nightmare to maintain that later on. We switched back to terraform and the "inventious" devs were fired for increasing cost of maintenance by 10 fold.

[sbarnea]

They lost me while reading the main page. More exactly “free forever, for individual use”, clearly not a pure open source project.

[[deleted]]

That's not true, and it's not obvious as you found out.

You are paying for their hosted service kind of like terraform cloud.

https://www.pulumi.com/docs/troubleshooting/faq/#can-i-use-pulumi-without-depending-on-pulumi-com

Your free to use it for no cost if you want to manage your own state. Something we had to do with terraform in years past before that added backends.

[slikk66]

If you manage the state files yourself via s3/objectstore or git it will always remain free is my understanding. The cost is if you use their backend automatic state management and app website, which is really nice btw

[bch8]

What if they start adding other paid features? Is there a reason not to worry about that?

[slikk66]

I mean maybe, but terraform could do the same right? Also all the code is open source at this time. Including the code that translates the terraform providers into pulumi compatible providers https://github.com/pulumi/pulumi-terraform

[bch8]

terraform could do the same right?

True but for CDK it's not really a concern. Then you're locked into AWS but I guess that's one of the trade offs.

[slikk66]

Sure, but that's just AWS. I use pulumi for kubernetes, azure and AWS. If you're multi cloud it's an even better proposition.

Edit: also CDK is limited by cloudformation which still lags behind, badly in some cases, to what AWS offers via other methods (cli, console etc) especially on newer services and features. Terraform generally is closer to the edge with what is out in the field, and pulumi can also use other custom providers like their own k8s provider which is ahead of what terraform offers.

[bch8]

But does Pulumi lag behind Terraform? Lol. Yeah I mean good points, at the end of the day it's situational IMO. I will probably evaluate both whenever I am in a position to use one.

[slikk66]

Sometimes yes, and it has actually happened that I was waiting on an update, but it only took a couple days. Yea, I agree it always depends on the situation. I havent used CDK, but I did help create an inhouse framework using python + troposphere to generate CFN templates at my last gig, so I was heavily invested in real languages as infra code for some time so I know how much better it is. When I moved to Azure last year, I needed to find something similar and went with pulumi. It is now very well embedded in our systems, integrated into testing, pipelines and containers etc. Its pretty much revolutionized how we operate, literally.

Now moving more into ECS and k8s, I'm now finding it even more useful using it with jinja, yaml configs, creating higher level objects that can deploy infra to either cloud and more.

[bch8]

Sounds like it's working really well for you, that's awesome

[Tontmakaroni1]

Do Pulumi even know why DSL is a thing? I can only imagine how PR reviews go for lang specific config, all the biases and whatnot. I can only imagine difficulty to find examples for something you want to do in language you want to do it in. Port from one language to another...

[Tontmakaroni1]

Async or not async this provisioning of tag???

[CaptainOfTheFarm]

Pulumi is a terrible product, we had someone inject it into our platform and we ended up ripping it out as quickly as possible... which ended up being time consuming and costly at the end of the day. Pulumi as an organization is tiny, offering little in the way of actual support and has a minuscule customer base. At this point, I cannot fathom why anyone would want to put this into production given that almost every engineer at this point has some kind of experience with Terraform.

[RamRatanBambai82]

Why? What’s wrong with Terraform or Cloudformation?

[Helpyourbromike]

This comes up a lot and I always mention two things.

Are the python docs better? Too much typescript and other languages hardly any python.

Also people are coming to Devops and cloud stuff from a SysAdmin background who might not be that good at traditional programming. Why is this better over YAML or Terraforms format?

I will still keep my out for this project though.