{"templateId":"markdown","sharedDataIds":{"sidebar":"sidebar-sidebars.yaml"},"props":{"codeGuideFiles":[],"metadata":{"markdoc":{"tagList":[]},"type":"markdown"},"seo":{"title":"Troubleshooting and Fault Recovery","description":"Catena Tools - Your game backend tool kit. Built by game developers, for game developers to simplify your game's backend workflow."},"dynamicMarkdocComponents":[],"ast":{"$$mdtype":"Tag","name":"article","attributes":{},"children":[{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"troubleshooting-and-fault-recovery"},"children":["Troubleshooting and Fault Recovery"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This page covers common failure modes for an AWS EC2 deployment of Catena, how to diagnose, and how to resolve them."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"interrupted-deploys"},"children":["Interrupted Deploys"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["git push dokku main"]}," is interrupted mid-transfer (a cancelled command, a dropped connection), Dokku leaves the app locked to prevent a second deploy from starting concurrently and corrupting the running state. A subsequent push will fail with a message telling you to run ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apps:unlock"]},"."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":""},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <path-to-your-ssh-key> dokku@<your-domain> apps:unlock <app-name>\nssh -i <path-to-your-ssh-key> dokku@<your-domain> ps:report <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Check ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ps:report"]}," to see whether the app is still running the previous, working build or is in a partially-deployed state. Either way, redeploy cleanly and let it complete without interruption this time:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"git push dokku main\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This rebuilds and restarts the app from your current branch, replacing whatever partial state the interruption left behind."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"app-crash-looping-or-failing-to-start"},"children":["App Crash-Looping or Failing to Start"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"-1"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <path-to-your-ssh-key> dokku@<your-domain> ps:report <app-name>\nssh -i <path-to-your-ssh-key> dokku@<your-domain> logs <app-name> --num 500\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Look for an exception or stack trace near the point the app attempts to start."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Common causes and their fixes:"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"a-required-configuration-value-is-missing-or-empty."},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Cause:"]}," A required configuration value is missing or empty."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["System.ArgumentException"]}," or similar referencing a null/empty config path is a strong signal here."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["First, confirm what's actually set before assuming a value:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"  ssh -i <path-to-your-ssh-key> dokku@<your-domain> config:show <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["It's also worth confirming the value is actually visible inside the running container, not just recorded in Dokku's config store. A value can be set but might not be picked up by the app yet if it hasn't been restarted since it was set:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"  ssh -t -i <path-to-your-ssh-key> dokku@<your-domain> enter <app-name> web\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"set-the-missing-value-via-your-secrets-script"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}," Set the missing value via your secrets script"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["See ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/installation/aws-secret-management"},"children":["Managing Application Secrets"]}," for more details:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"  ./set-dokku-secrets.sh <your-domain>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This restarts the app automatically with the corrected value — no redeploy needed, since environment variables aren't part of the deployed image."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"a-configuration-key-is-set-via-environment-variable-but-the-key-path-doesnt-exactly-match-what-the-application-expects."},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Cause:"]}," A configuration key is set via environment variable, but the key path doesn't exactly match what the application expects."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Nested configuration sections require every level of the path to be present in the environment variable name, including any intermediate segment that isn't obvious from the equivalent ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["appsettings.json"]}," structure. A value that looks like it should map correctly can silently fail to bind if one segment is missing. The property simply keeps its default (often empty value)."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"verify-the-expected-config-value-and-update"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Fix:"]}," Verify the expected config value and update"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Validate the value in the actual C# configuration class that drives ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["appsettings.json"]},". Confirm the full key path, and correct the environment variable name via your secrets script. (If you can't confirm from ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["appsettings.json"]}," itself)"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["It will be a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["*config.cs"]}," class that is associated with the service you are having issues with. For example, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Services\\CatenaMatchmaking\\Config\\MatchmakerConfig.cs"]}," associated with matchmaking configurations."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Once resolved and values are set in your env file, run the script:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"./set-dokku-secrets.sh <your-domain>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["No redeploy needed. This restarts the app with the corrected variable name."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["See ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/installation/aws-secret-management"},"children":["Managing Application Secrets"]}," for more details"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"an-aws.accessdenied-or-amazonserviceexception-at-startup"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Cause:"]}," An ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AWS.AccessDenied"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AmazonServiceException"]}," at startup"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"update-iam-policies"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}," Update IAM policies"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["See ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"#permission-and-credential-failures"},"children":["Permission and Credential Failures"]}," below for diagnosing and fixing the underlying IAM issue. Once the policy is corrected, restart the app to pick it up. No redeploy needed, since this is an infrastructure-side fix, not a code or config change:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <path-to-your-ssh-key> dokku@<your-domain> ps:restart <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["In general:"]}," a fix that changes IAM policy, security groups, or other AWS-side infrastructure only requires a restart. A fix that changes application code or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["appsettings.json"]}," requires a commit and redeploy (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["git push dokku main"]},"). A fix that only changes an environment variable's value can be applied via the secrets script alone, with no redeploy."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"ssh-connection-dropped-mid-provisioning"},"children":["SSH Connection Dropped Mid-Provisioning"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["During ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]},", the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["remote-exec"]}," provisioner (which SSHes into a freshly-launched instance to run setup steps) can fail with:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"Error: remote-exec provisioner error\nerror executing \"/tmp/terraform_<id>.sh\": wait: remote command exited without exit status or exit signal\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This means the SSH session was cut off before the script finished, it doesn't necessarily mean that the script itself failed."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"debug-your-deployment-machine-and-ec2-machine"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}," Debug your deployment machine and EC2 machine"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Check your deployment machines sleep settings. Deployment here could take upwards of 20-30 min. Make sure your machine doesn't go to sleep in that time period or this deployment will fail."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If your machine isn't causing the issue, you can debug the remote machine."]},{"$$mdtype":"Tag","name":"ol","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":["View your ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/ec2/home"},"children":["AWS EC2 instances"]}]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Find the Catena instance and Connect via Session Manager."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":["Once in, check whether the pushed script is still present:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ls -la /tmp/terraform_*.sh\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If it exists, run it directly to see the real failure without Terraform's SSH session as an intermediary:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"sudo bash /tmp/terraform_<id>.sh\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Also check whether the instance rebooted unexpectedly during setup (a common cause of a dropped session e.g. if the install process restarts networking):"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"uptime\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"-2"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If the manual run surfaces the actual error (out-of-memory, a package install failure, a networking restart mid-script), address that specific cause. For an undersized instance, increase ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["catena_ec2_instance_size"]}," in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["vars.tfvars"]}," before reapplying. For a transient issue, simply re-running ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]}," (which re-triggers the provisioner) is often sufficient, since a dropped connection doesn't necessarily indicate a repeatable failure:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform apply -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If the instance is in a partially-provisioned state and reapplying doesn't cleanly recover it, taint the resource to force a clean recreation:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform taint module.ec2.null_resource.setup_catena\nterraform apply -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"instance-or-infrastructure-failure"},"children":["Instance or Infrastructure Failure"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If the EC2 instance itself is unresponsive (health check failing, SSH/Session Manager unreachable):"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"check-ec2-status"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}," Check EC2 status"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws ec2 describe-instances --instance-ids <your-instance-id> --profile <your-deploy-profile> --query \"Reservations[].Instances[].State\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Confirm the instance is actually ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["running"]},". Check for drift between Terraform's state and the real infrastructure:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform plan -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A plan showing unexpected changes (especially destroy/recreate on resources you didn't touch) indicates the instance was modified manually or by something outside Terraform's awareness."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"reboot-or-re-deploy"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}," Reboot or Re-deploy"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["if the instance is unresponsive but Terraform sees no drift, a reboot via the AWS console or CLI is the least disruptive first step:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws ec2 reboot-instances --instance-ids <your-instance-id> --profile <your-profile>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If that doesn't resolve it, or the instance is genuinely unrecoverable, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]}," will recreate it from the existing configuration:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform apply -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Be aware this replaces the SQLite database on the root volume — see Backup and Recovery documentation before doing this on any deployment holding real data. After recreation, you'll also need to redeploy the app itself and re-set application secrets, since neither is tracked by Terraform state:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh-keygen -R <your-domain>\ngit push dokku main\n./set-dokku-secrets.sh <your-domain>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"state-save-failures-during-apply-or-destroy"},"children":["State Save Failures During Apply or Destroy"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Occasionally ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]}," or ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform destroy"]}," will fail partway through with something like:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"Error: Failed to save state\nError saving state: failed to upload state: operation error S3: PutObject, https response error\nStatusCode: 0, RequestID: , HostID: , request send failed, Put\n\"https://catena-terraform-state.s3.us-east-1.amazonaws.com/...\": dial tcp: lookup catena-terraform-state.s3.us-east-1.amazonaws.com: no such host\n"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This is a DNS resolution failure — Terraform couldn't reach S3 at all to persist its state, typically due to a transient network issue, a VPN or corporate DNS interfering with resolution, or a brief AWS-side disruption. This is distinct from an ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AccessDenied"]}," error (see ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"#permission-and-credential-failures"},"children":["Permission and Credential Failures"]}," below): here, Terraform never got far enough to even attempt authorization."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["The immediate risk:"]}," if the underlying ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apply"]},"/",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["destroy"]}," operation itself succeeded or partially succeeded against AWS, but the state save failed, Terraform's local understanding of your infrastructure is now out of sync with what's actually deployed. Terraform handles this by writing the current state to a local ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["errored.tfstate"]}," file and telling you to push it back manually. ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Do NOT ignore this message or delete the file"]},", since it may be the only record of changes that already happened in AWS but were never recorded in the real state backend."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"confirm-basic-connectivity-to-s3-before-assuming-anything-is-broken-on-the-aws-side"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}," confirm basic connectivity to S3 before assuming anything is broken on the AWS side:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"nslookup catena-terraform-state.s3.us-east-1.amazonaws.com\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If this fails to resolve, the problem is local network/DNS, not AWS or your credentials — check VPN status, try a different network, or wait and retry, since this class of failure is usually transient."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"push-the-locally-saved-error-state-back-to-the-real-backend"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}," Push the locally-saved error state back to the real backend:"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["once connectivity is confirmed working again, push the locally-saved error state back to the real backend, exactly as Terraform's own error message instructs:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform state push errored.tfstate\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This reconciles the backend with whatever Terraform's local process actually knew at the moment of failure. After it succeeds, run a plan to confirm state and reality now agree before making any further changes:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform plan -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A clean plan (no unexpected changes) confirms the push fully resolved the drift. If the plan still shows unexpected differences, treat it the same as ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Instance or Infrastructure Failure"]}," below — verify the real state of resources in the AWS console directly rather than trusting either local state blindly."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"permission-and-credential-failures"},"children":["Permission and Credential Failures"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["An ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AccessDenied"]}," error, whether from Terraform during ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["plan"]},"/",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apply"]},", or from the running application in its logs, always follows the same shape:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"header":{"controls":{"copy":{}}},"source":"User: arn:aws:iam::<account>:<user-or-role> is not authorized to perform: <action> on resource: <resource> because no identity-based policy allows the <action> action\n"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"confirm-expected-principal/action-involved-and-update-policy."},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}," Confirm expected principal/action involved and update policy."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The error tells you exactly which principal, action, and resource are involved."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Confirm what's currently attached to that principal:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws iam list-attached-user-policies --user-name <user> --profile <profile>\naws iam list-attached-role-policies --role-name <role> --profile <profile>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Then inspect the specific policy for the missing action:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws iam get-policy-version --policy-arn <policy-arn> --version-id <version> --profile <profile>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"update-the-iam-policy-with-the-requested-permission/resource"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery:"]}," Update the IAM policy with the requested permission/resource"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Naivagate to the AWS web page and go to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["IAM"]}," -> ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Policies"]},"."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Find ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["CatenaDeploymentPolicy"]},", add the missing action to the relevant statement, scoped to the specific resource named in the error."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Do not widen resource to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["\"Resource\": \"*\""]}," unless the action genuinely doesn't support resource-level scoping (as is the case for most GameLift actions)."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Retry the original ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["plan"]},", or application action that triggered the error."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"matchmaking-ticket-failures"},"children":["Matchmaking Ticket Failures"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A ticket that never leaves ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["SEARCHING"]}," has typically exceeded its ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["request_timeout_seconds"]}," without finding a match, and will move to ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["TIMED_OUT"]},". This is expected behavior when there aren't enough compatible players — the documented recovery is resubmission, not a permission or configuration problem."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"check-status-of-current-matchmaking-configs"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Diagnosis:"]}," Check status of current matchmaking configs"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws gamelift describe-matchmaking --ticket-ids <ticket-id> --profile <profile> --region <region>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Check the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Status"]}," and ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["StatusMessage"]}," fields directly rather than relying solely on what the application surfaces, since an underlying ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AccessDeniedException"]}," from GameLift can sometimes be caught and re-presented by the application as a more generic failure (such as \"queue does not exist\") rather than propagated as-is."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"-3"},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Recovery, if tickets fail immediately rather than timing out normally:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Confirm the runtime role or user has all four required GameLift actions (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["StartMatchmaking"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["StopMatchmaking"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["DescribeMatchmaking"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["DescribeMatchmakingConfigurations"]},") and that the SQS notification queue permissions (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ReceiveMessage"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["DeleteMessage"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["GetQueueAttributes"]},") are present. See ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/features/matchmaking/aws-flex-match"},"children":["Setting Up AWS FlexMatch"]}," and ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"#permission-and-credential-failures"},"children":["Permission and Credential Failures"]}," above for the expected values here. Once corrected, no redeploy or restart is needed, IAM changes take effect on the next API call."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Confirming the notification pipeline is working"]},", independent of the application's own logs:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws sqs get-queue-attributes --queue-url <your_sqs_queue_url> --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible --profile <profile>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["A message stuck in the queue (nonzero count that doesn't drain over time) points at a problem reading from SQS specifically, distinct from the GameLift-side matchmaking calls. Check the application's logs for an ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["AccessDenied"]}," on ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["sqs:ReceiveMessage"]}," at the time a match completes."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"references-where-to-look-and-what-to-run"},"children":["References: Where to look and What to Run"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The scenarios above cover specific failures. This section is a general-purpose reference for direct debugging, independent of any particular error."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"finding-things-in-the-aws-console"},"children":["Finding Things in the AWS Console"]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"What you need"},"children":["What you need"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Where to find it"},"children":["Where to find it"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Instance state, public IP, instance ID"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Instances:"},"children":["EC2 → Instances"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["A shell on the instance, without SSH"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Select the instance → ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Connect"]}," → ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Session Manager"]}," tab → ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Connect"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["IAM role/user policies"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/iamv2/home#/roles"},"children":["IAM → Roles"]}," or ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/iam/home#/users"},"children":["IAM → Users"]}," → select the role/user → ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Permissions"]}," tab"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["A role's trust policy"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["IAM → Roles → select the role → ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Trust relationships"]}," tab"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["DNS records"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones"},"children":["Route53 → Hosted zones"]}," → select your zone"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Matchmaking configurations and rule sets"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/gamelift/home?region=us-east-1#/r/matchmaking/configurations"},"children":["GameLift → Matchmaking configurations"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["SQS queue contents/attributes"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/sqs/v2/home?region=us-east-1#/queues"},"children":["SQS → Queues"]}," → select your queue"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Current quota usage and limits"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/servicequotas/home"},"children":["Service Quotas"]}," → select the relevant service"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Terraform state file (raw)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://s3.console.aws.amazon.com/s3/home?region=us-east-1"},"children":["S3 → your state bucket"]}," → browse to the workspace/module path"]}]}]}]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"where-things-are-defined-for-deployment"},"children":["Where Things Are Defined for deployment"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The table above is for finding a resource's ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["current, live"]}," state in the console. This table below is for finding where that resource is ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["defined"]}," in Terraform during deployment. These are the value you need to know if you want to actually make lasting changes to the infrastructure."]},{"$$mdtype":"Tag","name":"div","attributes":{"className":"md-table-wrapper"},"children":[{"$$mdtype":"Tag","name":"table","attributes":{"className":"md"},"children":[{"$$mdtype":"Tag","name":"thead","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Concept"},"children":["Concept"]},{"$$mdtype":"Tag","name":"th","attributes":{"data-label":"Defined in code"},"children":["Defined in code"]}]}]},{"$$mdtype":"Tag","name":"tbody","attributes":{},"children":[{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["EC2 instance (size, AMI, root volume)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/ec2/main.tf"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["EC2 runtime IAM role (SSM, FlexMatch permissions)"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/iam/main.tf"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["EC2 runtime role's trust policy"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/iam/main.tf"]},", the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["assume_role_policy"]}," block"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Deploy-time user policy (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["CatenaDeploymentPolicy"]},")"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Not managed by Terraform — created and edited manually in the console per ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/installation/aws-ec2"},"children":["the AWS EC2 guide"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["VPC, subnet, internet gateway, route table"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/network/"]}," (or the equivalent path under ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/ec2-gameserver/"]},")"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Security group / open ports"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/network/security_groups/main.tf"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Elastic IP"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/main.tf"]},", the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["eip"]}," module"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Route53 domain / DNS records"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/dns/main.tf"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["SQS notification queue"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/flex-match/main.tf"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws_sqs_queue.notifications_queue"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["SNS notification topic"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/flex-match/main.tf"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws_sns_topic.catena_flex_match_sns"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["GameLift matchmaking configurations / rule sets"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/flex-match/main.tf"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["awscc_gamelift_matchmaking_configuration"]}," / ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["_rule_set"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["FlexMatch runtime policy"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/flex-match/main.tf"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws_iam_policy.catena_flexmatch_runtime_policy"]}]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Backup S3 bucket"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/main.tf"]}," (or wherever the backup bucket resource is defined)"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Terraform state bucket itself"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Not managed by this Terraform config — created once, manually, as a prerequisite"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["SSH keypair"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/ec2/main.tf"]},", ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ssh_public_key_path"]}," / ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ssh_private_key_path"]}," variables"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["IMDSv2 enforcement"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/ec2/main.tf"]},", the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["metadata_options"]}," block"]}]},{"$$mdtype":"Tag","name":"tr","attributes":{},"children":[{"$$mdtype":"Tag","name":"td","attributes":{},"children":["Which Terraform workspace maps to which resource names"]},{"$$mdtype":"Tag","name":"td","attributes":{},"children":[{"$$mdtype":"Tag","name":"code","attributes":{},"children":["aws/catena-core/main.tf"]}," (and equivalent in ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["flex-match"]},"/",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["ec2-gameserver"]},"), the ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["workspace_prepend"]}," local"]}]}]}]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["If you edit any of the files, you will need to run ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply --var-file=\"vars.tfvars\""]}," to apply the changes"]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"helpful-commands-and-tools"},"children":["Helpful commands and tools"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Here are some helpful processes to get you insight on your current deployment for debugging."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"accessing-the-instance-directly"},"children":["Accessing the Instance Directly"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Session Manager"]}," (no SSH key needed, works even if SSH itself is broken):"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Go to AWS and access via the ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"https://us-east-1.console.aws.amazon.com/console/home?region=us-east-1#"},"children":["AWS Console"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Via the console, ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["EC2"]}," → ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["<your instance>"]}," → ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Connect"]}," → ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Session Manager tab"]}," → ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["Connect"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["SSH via Dokku's restricted user"]}," (runs a single Dokku command, does not give a shell) Lets you run comands as the dokku instance:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> <dokku-command>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["A shell inside the running app's container"]},":"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Goin one deeper, actually connecting the the running app container. Opens a shell to run commands directly."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -t -i <your-key> dokku@<your-domain> enter <app-name> web\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":3,"id":"common-commands-from-your-machine"},"children":["Common Commands From Your Machine"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Examples of some commands to run from your machine for debugging or common gotchas."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["App status and health:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Prints a report on the app status and health"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> ps:report <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Logs — recent history:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This will print the recent logs in the shell, 500 lines worth"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> logs <app-name> --num 500\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Logs — live tail:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Starts showing the live tail of logs from the active instance"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> logs <app-name> --tail\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Logs — save to a local file for searching:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Save the logs to a local file called ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["catena-logs.txt"]},". Good for text search"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> logs <app-name> --num 5000 > catena-logs.txt\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["All environment variables currently set:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Prints all environment variables currently set on the machine."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> config:show <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["A single environment variable:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Print a specific environment variable"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> config:get <app-name> <KEY_NAME>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Restart the app"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["picks up new environment variables or IAM changes; ",{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["does not"]}," redeploy code."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> ps:restart <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Clear a stale SSH host key"]}," — needed any time an instance is recreated (",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform destroy"]}," + ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["apply"]},", or a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["terraform apply"]}," that replaces the instance)."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["The new instance has a different underlying SSH host key even if the domain stays the same, and your local machine will refuse to connect with a ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["REMOTE HOST IDENTIFICATION HAS CHANGED"]}," warning until the stale entry is cleared:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh-keygen -R <your-domain>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This removes the cached entry for that hostname from your local ~/.ssh/known_hosts file. On a new deployment the new isntance generates a brand new SSH host key at boot but keeps the domain the same. So domain and key are legitimate but it doesnt match the ",{"$$mdtype":"Tag","name":"em","attributes":{},"children":["old"]}," key our machine knew. Your machine will cache the new version after clearing the old values."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Domain and TLS configuration:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> domains:report <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Confirm which AWS identity a command is running as:"]}]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"aws sts get-caller-identity --profile <profile>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Confirm Terraform's state matches real infrastructure:"]}]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Validate infrastructure hasn't drifted from whats deployed."]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform plan -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":1,"id":"backup-and-recovering-catena"},"children":["Backup and Recovering Catena"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["This guide walks through backing up, tearing down and rebuilding a Catena deployment, restoring your data from an existing backup afterward. Assuming some fault or error state that occured."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"1.-create-or-confirm-you-have-a-backup"},"children":["1. Create or Confirm You Have a Backup"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Before tearing anything down, confirm or create a backup. See ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/installation/aws-backup-management"},"children":["Backup and Recovery"]}," for how backups are created and managed (manual ",{"$$mdtype":"Tag","name":"code","attributes":{},"children":["sqlite3 .backup"]},", or automatic EBS snapshots)."]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"2.-tear-down-the-existing-deployment"},"children":["2. Tear Down the Existing Deployment"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"cd aws/catena-core\nterraform destroy -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"3.-redeploy"},"children":["3. Redeploy"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"terraform apply -var-file=\"vars.tfvars\"\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Follow the standard post-deploy steps:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh-keygen -R <your-domain>\ngit push dokku main\n./set-dokku-secrets.sh <your-domain>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"4.-restore-your-backup"},"children":["4. Restore Your Backup"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Follow the restore steps for whichever backup method you used — see ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"/installation/aws-backup-management"},"children":["Backup and Recovery"]},":"]},{"$$mdtype":"Tag","name":"ul","attributes":{},"children":[{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["Manual backup"]},": restore the SQLite database file to the new instance."]},{"$$mdtype":"Tag","name":"li","attributes":{},"children":[{"$$mdtype":"Tag","name":"strong","attributes":{},"children":["EBS snapshot"]},": attach the snapshot as the instance's data volume."]}]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"5.-confirm-recovery"},"children":["5. Confirm Recovery"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Restart the app and confirm it starts cleanly with the restored data:"]},{"$$mdtype":"Tag","name":"CodeBlock","attributes":{"data-language":"bash","header":{"controls":{"copy":{}}},"source":"ssh -i <your-key> dokku@<your-domain> ps:restart <app-name>\nssh -i <your-key> dokku@<your-domain> ps:report <app-name>\n","lang":"bash"},"children":[]},{"$$mdtype":"Tag","name":"Heading","attributes":{"level":2,"id":"support"},"children":["Support"]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["Catena does not currently offer set technical support tiers or standard Service Level Agreements (SLAs). Support, response times, and any associated terms are arranged on a per-contract basis."]},{"$$mdtype":"Tag","name":"p","attributes":{},"children":["To request support, contact ",{"$$mdtype":"Tag","name":"a","attributes":{"href":"mailto:it@wolfjawstudios.com"},"children":["it@wolfjawstudios.com"]},"."]}]},"headings":[{"value":"Troubleshooting and Fault Recovery","id":"troubleshooting-and-fault-recovery","depth":1},{"value":"Interrupted Deploys","id":"interrupted-deploys","depth":2},{"value":"","id":null,"depth":3},{"value":"App Crash-Looping or Failing to Start","id":"app-crash-looping-or-failing-to-start","depth":2},{"value":"","id":"-1","depth":3},{"value":"A required configuration value is missing or empty.","id":"a-required-configuration-value-is-missing-or-empty.","depth":3},{"value":"Set the missing value via your secrets script","id":"set-the-missing-value-via-your-secrets-script","depth":3},{"value":"A configuration key is set via environment variable, but the key path doesn't exactly match what the application expects.","id":"a-configuration-key-is-set-via-environment-variable-but-the-key-path-doesnt-exactly-match-what-the-application-expects.","depth":3},{"value":"Verify the expected config value and update","id":"verify-the-expected-config-value-and-update","depth":3},{"value":"An AWS.AccessDenied or AmazonServiceException at startup","id":"an-aws.accessdenied-or-amazonserviceexception-at-startup","depth":3},{"value":"Update IAM policies","id":"update-iam-policies","depth":3},{"value":"SSH Connection Dropped Mid-Provisioning","id":"ssh-connection-dropped-mid-provisioning","depth":2},{"value":"Debug your deployment machine and EC2 machine","id":"debug-your-deployment-machine-and-ec2-machine","depth":3},{"value":"","id":"-2","depth":3},{"value":"Instance or Infrastructure Failure","id":"instance-or-infrastructure-failure","depth":2},{"value":"Check EC2 status","id":"check-ec2-status","depth":3},{"value":"Reboot or Re-deploy","id":"reboot-or-re-deploy","depth":3},{"value":"State Save Failures During Apply or Destroy","id":"state-save-failures-during-apply-or-destroy","depth":2},{"value":"confirm basic connectivity to S3 before assuming anything is broken on the AWS side:","id":"confirm-basic-connectivity-to-s3-before-assuming-anything-is-broken-on-the-aws-side","depth":3},{"value":"Push the locally-saved error state back to the real backend:","id":"push-the-locally-saved-error-state-back-to-the-real-backend","depth":3},{"value":"Permission and Credential Failures","id":"permission-and-credential-failures","depth":2},{"value":"Confirm expected principal/action involved and update policy.","id":"confirm-expected-principal/action-involved-and-update-policy.","depth":3},{"value":"Update the IAM policy with the requested permission/resource","id":"update-the-iam-policy-with-the-requested-permission/resource","depth":3},{"value":"Matchmaking Ticket Failures","id":"matchmaking-ticket-failures","depth":2},{"value":"Check status of current matchmaking configs","id":"check-status-of-current-matchmaking-configs","depth":3},{"value":"","id":"-3","depth":3},{"value":"References: Where to look and What to Run","id":"references-where-to-look-and-what-to-run","depth":2},{"value":"Finding Things in the AWS Console","id":"finding-things-in-the-aws-console","depth":3},{"value":"Where Things Are Defined for deployment","id":"where-things-are-defined-for-deployment","depth":3},{"value":"Helpful commands and tools","id":"helpful-commands-and-tools","depth":2},{"value":"Accessing the Instance Directly","id":"accessing-the-instance-directly","depth":3},{"value":"Common Commands From Your Machine","id":"common-commands-from-your-machine","depth":3},{"value":"Backup and Recovering Catena","id":"backup-and-recovering-catena","depth":1},{"value":"1. Create or Confirm You Have a Backup","id":"1.-create-or-confirm-you-have-a-backup","depth":2},{"value":"2. Tear Down the Existing Deployment","id":"2.-tear-down-the-existing-deployment","depth":2},{"value":"3. Redeploy","id":"3.-redeploy","depth":2},{"value":"4. Restore Your Backup","id":"4.-restore-your-backup","depth":2},{"value":"5. Confirm Recovery","id":"5.-confirm-recovery","depth":2},{"value":"Support","id":"support","depth":2}],"frontmatter":{"seo":{"title":"Troubleshooting and Fault Recovery"}},"lastModified":"2026-07-31T21:05:51.000Z"},"slug":"/installation/troubleshooting-common-issues","userData":{"isAuthenticated":false,"teams":["anonymous"]}}