Skip to content

Latest commit

 

History

History
177 lines (147 loc) · 16.1 KB

README.md

File metadata and controls

177 lines (147 loc) · 16.1 KB

Materialize on AWS Cloud Platform

Terraform module for deploying Materialize on AWS Cloud Platform with all required infrastructure components.

Warning This is provided on a best-effort basis and Materialize cannot offer support for this module.

The module has been tested with:

  • PostgreSQL 15
  • Materialize Helm Operator Terraform Module v0.1.1

Providers Configuration

The module requires the following providers to be configured:

provider "aws" {
  region = "us-east-1"
  # Other AWS provider configuration as needed
}

# Required for EKS authentication
provider "kubernetes" {
  host                   = module.materialize_infrastructure.eks_cluster_endpoint
  cluster_ca_certificate = base64decode(module.materialize_infrastructure.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    args        = ["eks", "get-token", "--cluster-name", module.materialize_infrastructure.eks_cluster_name]
    command     = "aws"
  }
}

# Required for Materialize Operator installation
provider "helm" {
  kubernetes {
    host                   = module.materialize_infrastructure.eks_cluster_endpoint
    cluster_ca_certificate = base64decode(module.materialize_infrastructure.cluster_certificate_authority_data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = ["eks", "get-token", "--cluster-name", module.materialize_infrastructure.eks_cluster_name]
      command     = "aws"
    }
  }
}

module "materialize_infrastructure" {
  source = "git::https://github.com/MaterializeInc/terraform-aws-materialize.git"
  # Other required variables
}

Note: The Kubernetes and Helm providers are configured to use the AWS CLI for authentication with the EKS cluster. This requires that you have the AWS CLI installed and configured with access to the AWS account where the EKS cluster is deployed.

You can also set the AWS_PROFILE environment variable to the name of the profile you want to use for authentication with the EKS cluster:

export AWS_PROFILE=your-profile-name

Requirements

Name Version
terraform >= 1.0
aws ~> 5.0
helm ~> 2.0
kubernetes ~> 2.0
random ~> 3.0

Providers

Name Version
aws 5.88.0

Modules

Name Source Version
database ./modules/database n/a
eks ./modules/eks n/a
networking ./modules/networking n/a
operator github.com/MaterializeInc/terraform-helm-materialize v0.1.7
storage ./modules/storage n/a

Resources

Name Type
aws_cloudwatch_log_group.materialize resource
aws_iam_access_key.materialize_user resource
aws_iam_role.materialize_s3 resource
aws_iam_role_policy.materialize_s3 resource
aws_iam_user.materialize resource
aws_iam_user_policy.materialize_s3 resource
aws_caller_identity.current data source
aws_region.current data source

Inputs

Name Description Type Default Required
availability_zones List of availability zones list(string) 
[
  "us-east-1a",
  "us-east-1b",
  "us-east-1c"
]
 no
bucket_force_destroy Enable force destroy for the S3 bucket bool true no
bucket_lifecycle_rules List of lifecycle rules for the S3 bucket 
list(object({
    id                                 = string
    enabled                            = bool
    prefix                             = string
    transition_days                    = number
    transition_storage_class           = string
    noncurrent_version_expiration_days = number
  }))
 
[
  {
    "enabled": true,
    "id": "cleanup",
    "noncurrent_version_expiration_days": 90,
    "prefix": "",
    "transition_days": 90,
    "transition_storage_class": "STANDARD_IA"
  }
]
 no
cluster_enabled_log_types List of desired control plane logging to enable list(string) 
[
  "api",
  "audit",
  "authenticator",
  "controllerManager",
  "scheduler"
]
 no
cluster_version Kubernetes version for the EKS cluster string "1.32" no
create_vpc Controls if VPC should be created (it affects almost all resources) bool true no
database_name Name of the database to create string "materialize" no
database_password Password for the database (should be provided via tfvars or environment variable) string n/a yes
database_username Username for the database string "materialize" no
db_allocated_storage Allocated storage for the RDS instance (in GB) number 20 no
db_instance_class Instance class for the RDS instance string "db.t3.large" no
db_max_allocated_storage Maximum storage for autoscaling (in GB) number 100 no
db_multi_az Enable multi-AZ deployment for RDS bool false no
enable_bucket_encryption Enable server-side encryption for the S3 bucket bool true no
enable_bucket_versioning Enable versioning for the S3 bucket bool true no
enable_cluster_creator_admin_permissions To add the current caller identity as an administrator bool true no
enable_monitoring Enable CloudWatch monitoring bool true no
environment Environment name (e.g., prod, staging, dev) string n/a yes
helm_chart Chart name from repository or local path to chart. For local charts, set the path to the chart directory. string "materialize-operator" no
helm_values Additional Helm values to merge with defaults any {} no
install_materialize_operator Whether to install the Materialize operator bool true no
install_metrics_server Whether to install the metrics-server for the Materialize Console bool true no
kubernetes_namespace The Kubernetes namespace for the Materialize resources string "materialize-environment" no
log_group_name_prefix Prefix for the CloudWatch log group name (will be combined with environment name) string "materialize" no
materialize_instances Configuration for Materialize instances 
list(object({
    name                 = string
    namespace            = optional(string)
    database_name        = string
    environmentd_version = optional(string, "v0.130.4")
    cpu_request          = optional(string, "1")
    memory_request       = optional(string, "1Gi")
    memory_limit         = optional(string, "1Gi")
    create_database      = optional(bool, true)
    in_place_rollout     = optional(bool, false)
    request_rollout      = optional(string)
    force_rollout        = optional(string)
  }))
 [] no
metrics_retention_days Number of days to retain CloudWatch metrics number 7 no
namespace Namespace for all resources, usually the organization or project name string n/a yes
network_id The ID of the VPC in which resources will be deployed. Only used if create_vpc is false. string "" no
network_private_subnet_ids A list of private subnet IDs in the VPC. Only used if create_vpc is false. list(string) [] no
node_group_ami_type AMI type for the node group string "AL2023_ARM_64_STANDARD" no
node_group_capacity_type Capacity type for worker nodes (ON_DEMAND or SPOT) string "ON_DEMAND" no
node_group_desired_size Desired number of worker nodes number 2 no
node_group_instance_types Instance types for worker nodes.

Recommended Configuration for Running Materialize with disk:
- Tested instance types: m6g, m7g families (ARM-based Graviton instances)
- AMI: AWS Bottlerocket (optimized for container workloads)
- Note: Ensure instance store volumes are available and attached to the nodes for optimal performance with disk-based workloads.		 list(string) 
[
  "r8g.2xlarge"
]
 no
node_group_max_size Maximum number of worker nodes number 4 no
node_group_min_size Minimum number of worker nodes number 1 no
operator_namespace Namespace for the Materialize operator string "materialize" no
operator_version Version of the Materialize operator to install string null no
orchestratord_version Version of the Materialize orchestrator to install string "v0.130.4" no
postgres_version Version of PostgreSQL to use string "15" no
private_subnet_cidrs CIDR blocks for private subnets list(string) 
[
  "10.0.1.0/24",
  "10.0.2.0/24",
  "10.0.3.0/24"
]
 no
public_subnet_cidrs CIDR blocks for public subnets list(string) 
[
  "10.0.101.0/24",
  "10.0.102.0/24",
  "10.0.103.0/24"
]
 no
service_account_name Name of the service account string "12345678-1234-1234-1234-123456789012" no
single_nat_gateway Use a single NAT Gateway for all private subnets bool false no
tags Default tags to apply to all resources map(string) 
{
  "Environment": "dev",
  "Project": "materialize",
  "Terraform": "true"
}
 no
use_local_chart Whether to use a local chart instead of one from a repository bool false no
vpc_cidr CIDR block for VPC string "10.0.0.0/16" no

Outputs

Name Description
cluster_certificate_authority_data Base64 encoded certificate data required to communicate with the cluster
database_endpoint RDS instance endpoint
eks_cluster_endpoint EKS cluster endpoint
eks_cluster_name EKS cluster name
materialize_s3_role_arn The ARN of the IAM role for Materialize
metadata_backend_url PostgreSQL connection URL in the format required by Materialize
oidc_provider_arn The ARN of the OIDC Provider
operator_details Details of the installed Materialize operator
persist_backend_url S3 connection URL in the format required by Materialize using IRSA
s3_bucket_name Name of the S3 bucket
vpc_id VPC ID

Post-Deployment Setup

After successfully deploying the infrastructure with this module, you'll need to:

  1. (Optional) Configure storage classes
  2. Install the Materialize Operator
  3. Deploy your first Materialize environment

See our Operator Installation Guide for instructions.