Introduction

In recent years, Terraform has become an essential skill not just for cloud and infrastructure engineers, but also for other professionals such as application engineers and data engineers. This is because Terraform is a powerful Infrastructure as Code (IaC) tool that enables users to easily create and manage cloud environments. Its ease of use and learning curve have made it increasingly popular among various engineering disciplines.

However, managing multiple private Terraform modules could become more complex compared to handling a single cloud environment. The process of using private Terraform modules differs from that of public ones, and there are several methods to consider for managing them effectively.

In this post, we will explore these methods and check the guidance on how to use them, helping you decide which approach best suits your needs.

Why is important to manage the version in Terraform module

Before diving into the different methods of managing private Terraform modules, it’s important to understand why version control is crucial in Terraform module management. If you are already familiar with the importance of version control, feel free to skip this section and proceed to the next one.

Managing versions in Terraform modules is vital from a technical perspective as it ensures maintainability, ease of management, and an improved development experience. Proper version control allows you to easily track changes, revert to previous versions if necessary, and ensure that different team members are working with compatible module versions. This ultimately leads to reduced maintenance costs and a smoother development process.

Terraform’s version constraint syntax is one such feature that simplifies the management of module versions. For example, using a constraint like version = ">= 1.2.0, < 2.0.0" allows you to specify a range of acceptable versions for a module. This ensures that your infrastructure remains compatible with the specified module versions, while still benefiting from updates and bug fixes within the allowed version range.

A practical example of when this version constraint could be useful is when a particular module has received several updates to fix minor bugs or improve performance. Suppose you’re using version 1.2.0 of the module and want to benefit from these improvements without upgrading to a potentially breaking 2.0.0 release. In this case, specifying the version constraint as version = ">= 1.2.0, < 2.0.0" allows you to automatically receive updates within the 1.x.x range, ensuring a stable and up-to-date infrastructure without the risk of breaking changes introduced in version 2.0.0. By leveraging Terraform’s versioning features, you could efficiently manage module versions and streamline your development process.

4 Ways to Manage Terraform Modules

In the realm of technical solutions, there is rarely a one-size-fits-all answer. Even the best practices may not be the most suitable option if the associated costs outweigh the benefits. That being said, we will explore four distinct ways to manage Terraform modules, including two methods that are generally recommended. It’s essential to carefully consider which approach is best suited to address your specific needs and challenges.

One repository per module with Git tags for versioning (recommended)

One of the most common and officially recommended approaches to managing Terraform modules is to create one repository per module, with Git tags for versioning. This method allows you to utilize the full range of features Terraform has to offer, including its version syntax. It is particularly suitable if you have a small number of modules to manage.

However, if you have a larger number of modules, such as 30 or 50, managing one repository per module can become quite burdensome. Besides the challenge of managing numerous repositories, this approach has other drawbacks, including potential inconsistencies across modules and increased complexity when updating or coordinating changes among them.

Pros:

• Officially recommended by Terraform
• Straightforward and easy to understand
• Supports full features of Terraform modules, including version constraints syntax
• Clear separation of concerns with one repository per module

Cons:

• Can be burdensome to manage when dealing with a large number of modules
• Requires more effort to maintain multiple repositories and track changes across them
• Potential duplication of code or resources if not managed carefully

In summary, using one repository per module with Git tags for versioning is a well-established and recommended approach for managing a small number of Terraform modules. However, as the number of modules increases, this method may introduce additional challenges in terms of management and coordination.

Mono-repo with Terralist (personally recommended)

The mono-repo with Terralist approach is based on an open-source application called Terralist, which serves as a private registry for Terraform modules. The key features of this are Login Functionality, Modules Registry, Provider Registry. Out of them, The Modules Registry centralizes all your Terraform modules in a single repository, making it easier to manage and maintain. With Terralist, you can organize and version your modules in a more streamlined manner.

For example, instead of managing multiple repositories for each module, you can have a single repository containing all your modules with Terralist managing their versions and dependencies. With this method, you can call the module in the same way as using the public Terraform Registry in your private Terraform registry like this

module "vpc" {
  source  = "your-terralist-registry-url/your-repo/vpc"
  version = ">= 1.2.0, <= 1.6.0"

  # module inputs...
}

This simplifies the overall structure and makes it easier to use modules and track changes and updates.

While Terralist offers numerous advantages, it also has some downsides. First, there are costs associated with hosting and maintaining the application, as well as the time and effort required to learn and manage it. Second, relying on open-source software can introduce dependency issues.

While open-source projects can be cost-effective and widely adopted, it’s crucial to carefully consider their potential risks, especially for critical systems or projects. Unexpected technical issues or discontinuation of development can lead to dependencies that are difficult to resolve, potentially resulting in significant costs to find and implement alternative solutions.

Terralist is an excellent project that has proven valuable to many users, including myself. However, its development history is still relatively short, it is important to carefully consider potential dependency issues when selecting development tools for your projects, as you would with any other tool.

Pros:

• Consolidates all Terraform modules into a single repository, simplifying management
• Provides a private registry for Terraform modules through Terralist
• Encourages reusability and modularity within your Terraform codebase
• Can easily track and manage dependencies between modules

Cons:

• Involves additional costs, such as hosting, learning, and managing the Terralist application
• Introduces dependency on an open-source project with a relatively short development history
• May require additional effort to ensure proper versioning and dependency management
• Potential risk of project abandonment or technical issues with the Terralist application

Mono-repo with Git tags and subdirectories

This approach entails utilizing a single repository to house multiple Terraform modules, each within its own separate subdirectories. This method can be advantageous in certain scenarios, as it simplifies the management of multiple modules by consolidating them into a single repository. For example, a company with a smaller number of modules could find it easier to track and maintain them all in one place.

Specifically, Git tags serve as a way to mark specific points in the repository’s history, representing different versions of the modules. This allows you to manage and reference different versions of each module within the same repository.

Let’s look at an example to illustrate this:

Suppose you have a mono-repo with the following directory structure:

terraform-modules/
├── module1/
│   ├── main.tf
│   └── variables.tf
└── module2/
    ├── main.tf
    └── variables.tf

Initially, you create a Git tag for the first version of each module:

module1/v1.0.0
module2/v1.0.0

Now, let’s say you make updates to module1. After making changes and committing them, you create a new Git tag for the updated version of module1:

module1/v1.1.0

At this point, your Git tags represent the following versions:

module1/v1.0.0 - initial version of module1
module1/v1.1.0 - updated version of module1
module2/v1.0.0 - initial version of module2

When you want to use a specific version of a module in your Terraform code, you reference the corresponding Git tag in the source attribute of the module block:

module "module1" {
  source = "git::https://github.com/my-org/terraform-modules.git//module1?ref=module1/v1.1.0"

  # module inputs...
}

In this example, the source attribute points to the module1 subdirectory in the terraform-modules repository and specifies the module1/v1.1.0 Git tag as the desired version.

By using Git tags in this way, you can manage multiple versions of each module within the same repository. However, it requires extra attention to ensure that the correct Git tags are used and maintained when updating modules.

However, there are a few drawbacks to this method. One significant limitation is that it does not support the native version syntax of Terraform modules, which can lead to potential issues with module versioning and dependency management. Additionally, a mono-repo setup can become difficult to manage as the number of modules grows, leading to increased complexity and potential confusion for developers working with the repository.

Mono-repo with Git branches

With this approach, you maintain separate branches for each version of a module within a single repository. Each branch contains the specific version of each module in the repository.

Here’s a breakdown of how to use this approach:

1. Create a mono-repo containing all your Terraform modules in separate directories:

terraform-modules/
├── module1/
└── module2/

2. Create a new branch for each version of the module you want to maintain. Use a naming convention that indicates the module name and version, for example:

# For module1
git checkout -b module1/v1.0.0
git checkout -b module1/v1.1.0

# For module2
git checkout -b module2/v2.0.0
git checkout -b module2/v2.1.0

3. When you make changes to a specific module version, ensure you’re on the corresponding branch, and commit the changes to that branch:

# Switch to the desired branch
git checkout module1/v1.1.0

# Make changes to module1 files, then commit
git add module1/*
git commit -m "Update module1 to v1.1.0"

4. In your Terraform code, reference the specific module and branch by updating the source attribute:

module "module1" {
  source = "git::https://github.com/my-org/terraform-modules.git//module1?ref=module1/v1.1.0"

  # module inputs...
}

This way, Terraform will use the contents of the module1 subdirectory from the specified branch (which represents the desired version).

Note that this approach can become harder to manage as the number of modules and versions grows since you’ll have multiple branches to maintain. Additionally, merging changes between branches can be challenging if there are significant differences between the module versions.

Summary

We’ve explored various approaches to managing Terraform modules in this article. It’s important to note that there isn’t a one-size-fits-all solution for module management, as the most suitable method can vary depending on the specific circumstances and requirements. To help readers make an informed decision, I’ve created a table summarizing the pros and cons of each approach. Please keep in mind that this table is for reference purposes only and may not cover every aspect or detail.

	standard way	Terralist	the way with git tags and subdirectories	the way with git branches
the proper number of modules for	a small number of modules	many modules	a small number of modules	a small number of modules
convenience about module management	high	high	middle	low
module number extensibility	middle	high	low	very low
financial cost	depending on the number of repository	Application Hosting Cost	none	none
learning cost	almost none	high	none	none
maintenance cost	no	yes	no	no
management cost	depending on the number of repository	middle	low	very low
potential end-of-support risk	no	yes	no	no

Conclusion

In conclusion, effectively managing your Terraform modules is crucial for maintaining a smooth and efficient infrastructure-as-code workflow. By examining the four different methods presented in this article, you can determine the most suitable approach for your specific needs and preferences.

Always remember that the ideal method may vary depending on factors such as team size, infrastructure complexity, and organizational policies. Keep exploring and experimenting to find the best solution for your unique situation, and continue to refine your Terraform module management strategy for optimal results.