This directory contains a Terraform Module for deploying Knfsd on Google Cloud.

The main branch may be updated at any time with the latest changes which could be breaking. You should always configure your module to use a release. This can be configured in the modules Terraform Configuration block.

source = "github.com/GoogleCloudPlatform/knfsd-cache-utils//deployment/terraform-module-knfsd?ref=v1.0.0-beta8"

Before continuing with the deployment and configuration of Knfsd you should review the deployment prerequisites.

There are a number of optional features and special configurations that can be enabled and configured for Knfsd. If you are planning on using any of these features/configurations then please review the appropriate documentation section.

• Metrics - System and proxy metrics for monitoring and observing Knfsd
• Autoscaling - Automatic scale up of Knfsd in response to the number of connected clients
• Agent - A lightweight API that provides information on Knfsd nodes
• Fanout Architecture - Documentation on how to deploy Knfsd in the fanout approach

Basic usage of this module is as follows:

module "nfs_proxy" {

    source = "github.com/GoogleCloudPlatform/knfsd-cache-utils//deployment/terraform-module-knfsd?ref=v1.0.0-beta8"

    # Google Cloud Project Configuration
    PROJECT                        = "my-gcp-project"
    REGION                         = "us-west1"
    ZONE                           = "us-west1-a"

    # Network Configuration
    NETWORK                        = "my-vpc"
    SUBNETWORK                     = "my-subnet"
    TRAFFIC_DISTRIBUTION_MODE      = "dns_round_robin"
    DNS_NAME                       = "rendercluster1.gcp.example.com."
    ASSIGN_STATIC_IPS              = true

    # Automatically create the firewall rules for the example to provide a
    # simple quick start when testing the knfsd proxy. Best practice is to
    # create these rules yourself as part of configuring the VPC network.
    AUTO_CREATE_FIREWALL_RULES     = true

    # Knfsd Proxy Configuration
    PROXY_IMAGENAME                = "knfsd-base-image"
    EXPORT_MAP                     = "10.0.5.5;/remoteexport;/remoteexport"
    PROXY_BASENAME                 = "rendercluster1"
    KNFSD_NODES                    = 3

}

# Prints the DNS name of KNFSD proxy
output "dns_name" {
    value = module.nfs_proxy.dns_name
}

Edit the above configuration variables to match your desired configuration.

Once you have created your deploy.tf and created you can deploy Knfsd with:

terraform init
terraform apply

NOTE: HEALTHCHECK_INITIAL_DELAY_SECONDS only applies to the first time the proxy starts up. If you reboot the proxy the standard health checks intervals will apply. The time allowed for a reboot is HEALTHCHECK_INTERVAL_SECONDS * (HEALTHCHECK_UNHEALTHY_THRESHOLD - 1) + HEALTHCHECK_TIMEOUT_SECONDS, with the default values this is 60 seconds * (3 probes - 1) + 2 seconds = 122 seconds (effectively 2 minutes).

Increasing HEALTHCHECK_INTERVAL_SECONDS and/or HEALTHCHECK_UNHEALTHY_THRESHOLD will allow more time to reboot a proxy instance. However, it will also delay the system from detecting unhealthy instances.

If using the NetApp Exports Auto-Discovery feature, please also read the NetApp specific docs.

The default MIG_REPLACEMENT_METHOD depends on ASSIGN_STATIC_IPS:

• When ASSIGN_STATIC_IPS = false then the default MIG_REPLACEMENT_METHOD is SUBSTITUTE.
• When ASSIGN_STATIC_IPS = true then the default MIG_REPLACEMENT_METHOD is RECREATE.

The default SERVICE_ACCOUNT depends on ENABLE_STACKDRIVER_METRICS.

• When false the NFS proxy instances will not have a service account
• When true the NFS proxy instances will use the default compute service account with the following scopes:
  • https://www.googleapis.com/auth/logging.write
  • https://www.googleapis.com/auth/monitoring.write

If a custom service account is assigned the compute instances will use the https://www.googleapis.com/auth/cloud-platform scope. This allows the proxy instances access to any GCP API permitted by IAM.

The service account will need the following project level IAM permissions:

• Logs Writer (roles/logging.logWriter)
• Monitoring Metric Writer (roles/monitoring.metricWriter)

These mount options are for the proxy to the source server.

NOTE: When using NFS v4, it is recommended that you use NFS v4.1 or greater. NFS v4.1 has many improvements to fix limitations of the NFS v4.0 protocol.

Use of AUTO_REEXPORT requires that FSID_MODE is local or external. external is recommended. See Auto Re-export for more detail.

The recommended FSID_MODE is to always use external. For more details on FSID_MODE and FSID_DATABASE_CONFIG see Filesystem Identifiers.

FSID_DATABASE_PRIVATE_IP is required when FSID_MODE="external", and FSID_DATABASE_DEPLOY=true.

Older knfsd versions that do not support FSID_MODE always used static to assign FSIDs. If you're upgrading from an earlier knfsd version it's advised you switch to external when possible, though you can use static to keep the existing behaviour.

The FSID_MODE option supports:

• static - Each export is explicitly allocated an incrementing FSID number on start-up. This requires all the exports to be known at start-up and is not compatible with AUTO_REEXPORT=true. static is only recommended if using an explicit EXPORT_MAP.

• local - Each export is automatically allocated an FSID number by the mountd using using the standard NFS fsidd service. This uses a local sqlite database to store FSID mappings. This is not recommended for production and should only be used for single instance proxy clusters.

  If multiple proxy instances in a cluster allocate a different FSID to the same export then I/O errors or data corruption may occur if a client changes instance.

  Also, even with a single proxy instance, if the instance is rebooted

• external - Each export is automatically allocated an FSID number by the mountd using using the knfsd-fsidd service. This uses a Cloud SQL Postgres instance to store the FSID mappings. This ensures that all the instances in cluster allocate the same FSID to each export.

If you exclude a nested mount but still export the parent mount you may get I/O errors when accessing the nested mount.

The exact behaviour will depend on how the source server has exported the nested mount.

If the source server exports the mount with the crossmnt, or nohide options then trying to access the nested mount, or list the directory containing the nested mount will result in I/O errors.

If the source server exports the mount without crossmnt, or hide options then the directory for the nested mount will be visible, but empty.

It is advised that if you exclude a nested mount, you also exclude the parent mount. You may however exclude a parent mount but include a nested mount.

For example, if you have the following mounts:

/assets
/assets/common
/assets/common/textures

You could exclude /assets, but still export /assets/common and /assets/common/textures. You could also export only /assets/common/textures.

However, exporting /assets but excluding /assets/common could cause errors.

Exporting /assets and /assets/common/textures, but excluding /assets/common will likely fail, and can have unintended side-effects as the proxy will try to create the directory /assets/common.

While auto-discovery and explicit mounts can be combined the system does not have any special handling for duplicate paths.

As such it is not recommended to combine multiple auto-discovery methods, or explicit (EXPORT_MAP).

The behaviour of duplicates is undefined. The system might overwrite one mount with another, or it may error.

The proxy cannot re-export any path that matches a symlink on the local server.

The most likely symlinks that will cause conflicts are:

• /bin
• /lib
• /lib32
• /lib64
• /libx32
• /sbin

The proxy will fail to start if it attempts to export a path that matches a symlink. Check the logs for errors such as:

ERROR: Cannot mount 10.0.0.2:/bin because /bin matches a symlink

If you are providing a manual export list, specify a different path for the export, such as 10.0.0.2;/bin;/binaries.

If you're using auto-discovery add the path to the list of excluded exports, for example EXCLUDED_EXPORTS = ["/bin"]

For a full list of symlinks, start a compute instance using the proxy image (without the standard startup script) and run the command:

find / -type l

Most of the symlinks listed are unlikely to cause issues, such as /usr/lib/x86_64-linux-gnu/libc.so.6.