Support more than one AWS region with a single Provider

[SamuelDudley]

Hi,
You didn't have a template for feature requests so I have used a blank issue.

Background

I have been using pulumi for about 4 years now, managing quite large AWS deployments (50+ AWS accounts). I really appreciate the tool and the effort that has been spent sustaining and improving it.

I use pulumi to manage whole AWS organizations, the accounts in these organizations and the resources in these accounts. This means providers with multiple AWS accounts and regions all in the same pulumi stack managing thousands of resources.

As the number of AWS accounts and regions I am targeting has grown, the number of AWS and AWS native providers I instantiate has increased too. This is because the concept of a pulumi provider is designed to work in a single AWS account and in a single AWS region. These are defined when the provider is instantiated and can't be changed at runtime (as far as I am aware).

The issue

Recently I began working on deploying AWS Config recorders, aggregators and delivery channels to all AWS regions (~16 regions are active by default and can't be opted out of) in my AWS accounts.

When a provider is instantiated by the pulumi runtime it creates a new process and this process consumes ~170MB of memory. This causes a situation where you can quickly run out of memory as the number of AWS accounts you are targeting with the stack increase:

Work around

Currently I am creating a dynamic resource which uses boto3 to manage the resources in other regions. While this works, it adds a lot of code and is throwing away all the benefits from using the underlying terraform provider to check and manage the resource state.

The boto3 approach works because it is far more memory efficient, allowing the creation of a Session() and then many clients in many regions from this single Session. In pulumi, it appears the concept of a boto3 Session and Client (both with the same region) are represented by a pulumi Provider.

Proposed feature

Create an interface similar to the boto3 api, where a single authorised Session() allows the definition of a Client() that takes a region name as an argument. The client inherits the auth model from the session without the significant memory overhead of creating a new provider resource.

Alternatively, continue to keep the 1 to 1 session to client pattern and greatly reduce the memory overhead of a Provider.

---

Would this be something the pulumi team would consider?
I know managing such large environments is niche, but IMHO that is where IaC tools such as pulumi really shine and add value. I also know that multiple stacks can be used to overcome some of what is discussed above, but it leaves you with much more complex multi-stage deployments which I have worked hard to avoid up to now.

---

Edit

I'm not a golang programmer, but is my understand that the pulumi-aws provider uses the terraform-provider-aws under the hood correct? If so, I see the underlying AWS go SDK has the concept of a client but the configuration to the provider is passed directly to that single client. https://github.com/hashicorp/terraform-provider-aws/blob/b20501eba261918f535897e47dacec8dbc8d6c7a/internal/provider/provider.go#L482

If the above is correct I imagine that modification to that code is outside the scope of pulumi-aws and is very unlikely.

[t0yv0]

Thanks so much for a detailed issue! This totally makes sense, I will make sure we take a hard look here. Pulumi provider model has historically been tied to "one provider configuration per one process" assumption, and undoing this while possible is no small undertaking as any code relying on this assumption will need to be revisited. That said, in the AWS case specifically the RAM tax you mention becomes prohibitive for some usage scenarios.

[t0yv0]

but is my understand that the pulumi-aws provider uses the terraform-provider-aws under the hood correct? If so, I see the underlying AWS go SDK has the concept of a client but the configuration to the provider is passed directly to that single client.

Yes that is not going to be a problem technically, again the difficulties would be coming from backwards-compat and being careful various assumptions current code might have on having the entire process to itself for a given configuration.