Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OP-517: Add deploy pipelines for livepoll and pandoc #1

Open
wants to merge 27 commits into
base: dockerize-pandoc-livepoll
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
65e9723
OP-517: Add deploy pipelines for livepoll and pandoc
jeff-dewan-lumen Oct 15, 2024
d29e2e7
OP-517: Need to add github action files to repository main branch
jeff-dewan-lumen Oct 15, 2024
2f70d36
Rename
jeff-dewan-lumen Oct 15, 2024
95c1d18
Remove ECR public
jeff-dewan-lumen Oct 15, 2024
6a6826d
Remove ECR public
jeff-dewan-lumen Oct 15, 2024
e183b92
Remove ECR public
jeff-dewan-lumen Oct 15, 2024
576a43d
Fix project path
jeff-dewan-lumen Oct 15, 2024
8765826
fmt
jeff-dewan-lumen Oct 15, 2024
038a133
fix terraform
jeff-dewan-lumen Oct 15, 2024
b824090
fix terraform
jeff-dewan-lumen Oct 15, 2024
5f566c5
Remove public ecr
jeff-dewan-lumen Oct 15, 2024
e429f8b
Add docker entrypoint
jeff-dewan-lumen Oct 15, 2024
b629b3a
Install awscli and jq
jeff-dewan-lumen Oct 15, 2024
a2ff1f5
Split entrypoint and fix pandoc
jeff-dewan-lumen Oct 17, 2024
7cb28ba
Fix entrypoint
jeff-dewan-lumen Oct 17, 2024
78bc3b4
Add health.html, update docker-compose
jeff-dewan-lumen Oct 17, 2024
bf1665a
Add health.html, update docker-compose
jeff-dewan-lumen Oct 17, 2024
29dcdad
Deploy cron and remove terraform and entrypoint for pandoc
jeff-dewan-lumen Oct 17, 2024
9ba0277
Remove pandoc terraform
jeff-dewan-lumen Oct 17, 2024
cd53ac0
Remove pandoc terraform
jeff-dewan-lumen Oct 17, 2024
1d7ebbe
Run empty terraform
jeff-dewan-lumen Oct 17, 2024
861f753
Remove cron deploy, chown files
jeff-dewan-lumen Oct 17, 2024
0efdeca
Cleanup
jeff-dewan-lumen Oct 17, 2024
cf5f618
Remove pandoc cron container and rely on EFS lifecycle rules to archi…
jeff-dewan-lumen Oct 22, 2024
2910a75
Merge remote-tracking branch 'origin/master' into OP-517/livepoll-pan…
jeff-dewan-lumen Oct 22, 2024
6c1c73f
Fix build
jeff-dewan-lumen Oct 22, 2024
8e9cc1f
Bump PHP to latest and use production PHP defaults for pandoc
jeff-dewan-lumen Oct 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/actions/private-ecr-login/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: 'Log into Private ECR'
description: 'Github OIDC auth and assume role into account, then use AWS ECR Login action'
inputs:
aws_ci_account:
description: 'AWS Account ID for CI'
required: false
default: 824635284302
aws_user_account:
description: 'AWS Account ID for users'
required: false
default: 265299512749
outputs:
registry:
description: "ECR Registry"
value: ${{ steps.login-ecr.outputs.registry }}
runs:
using: "composite"
steps:
- name: assume oidc role
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: us-west-2
role-to-assume: arn:aws:iam::${{ inputs.aws_user_account }}:role/ci-oidc-role
role-session-name: github-actions-oidc
role-duration-seconds: 900
- name: assume target role
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
aws-session-token: ${{ env.AWS_SESSION_TOKEN }}
aws-region: us-west-2
role-to-assume: arn:aws:iam::${{ inputs.aws_ci_account }}:role/ci-role
role-session-name: github-actions-private-ecr
role-duration-seconds: 900

- name: Login to Private ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
172 changes: 172 additions & 0 deletions .github/workflows/build-and-push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
name: build-and-push
env:
LIVEPOLL_ECR_REPOSITORY: livepoll
PANDOC_ECR_REPOSITORY: pandoc
on:
pull_request:
types: [ opened, synchronize, reopened ]
push:
branches:
- main
- rc/**

jobs:
validate-terraform:
runs-on: ubuntu-latest
name: Validate Terraform
timeout-minutes: 5
permissions:
id-token: write # Used for AWS OIDC auth
contents: read
actions: read
steps:
- uses: actions/checkout@v4
name: Checkout project

- name: OIDC Auth to AWS
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: us-west-2
role-to-assume: arn:aws:iam::${{ secrets.aws_user_account }}:role/ci-oidc-role
role-session-name: github-actions
role-duration-seconds: 900

- name: Setup Terraform
uses: hashicorp/setup-terraform@v3

- name: Livepoll Terraform Init
run: terraform init -backend=false
working-directory: deploy/terraform/livepoll

- name: Livepoll Terraform Format
run: terraform fmt -check
working-directory: deploy/terraform/livepoll

- name: Livepoll Terraform Validate
id: validate
run: terraform validate -no-color
working-directory: deploy/terraform/livepoll

- name: Pandoc Terraform Init
run: terraform init -backend=false
working-directory: deploy/terraform/pandoc

- name: Pandoc Terraform Format
run: terraform fmt -check
working-directory: deploy/terraform/pandoc

- name: Pandoc Terraform Validate
run: terraform validate -no-color
working-directory: deploy/terraform/pandoc

build-and-push-livepoll:
runs-on: ubuntu-latest
name: Build and Push Livepoll Docker Image
timeout-minutes: 30
needs: [validate-terraform]
permissions:
id-token: write # Used for AWS OIDC auth
contents: read
actions: read
steps:
- uses: actions/checkout@v4
name: Checkout project

- id: read_tree_hash
name: Read git tree hash
run: |
tree_hash=$(git rev-parse HEAD:)
echo "tree_hash=$tree_hash" >> $GITHUB_OUTPUT

- id: set_branch_name
name: Read git branch name
run: |
branch_name=${GITHUB_REF##*/}
echo "branch_name=$branch_name" >> $GITHUB_OUTPUT

- uses: ./.github/actions/private-ecr-login
name: Login to Private ECR
id: login-ecr

- name: Check for prebuilt image
id: prebuilt_check
run: |
docker manifest inspect ${{ steps.login-ecr.outputs.registry }}/${{ env.LIVEPOLL_ECR_REPOSITORY }}:gha-tree-${{ steps.read_tree_hash.outputs.tree_hash }} || echo "image_exists=$?" >> $GITHUB_OUTPUT

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
if: steps.prebuilt_check.outputs.image_exists != 0

- name: Docker build and push
uses: docker/build-push-action@v5
if: steps.prebuilt_check.outputs.image_exists != 0
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
with:
context: .
file: docker/Dockerfile-livepoll
platforms: linux/amd64
push: true
tags: |
${{ steps.login-ecr.outputs.registry }}/${{ env.LIVEPOLL_ECR_REPOSITORY }}:gha-commit-${{ github.sha }}
${{ steps.login-ecr.outputs.registry }}/${{ env.LIVEPOLL_ECR_REPOSITORY }}:gha-tree-${{ steps.read_tree_hash.outputs.tree_hash }}
${{ steps.login-ecr.outputs.registry }}/${{ env.LIVEPOLL_ECR_REPOSITORY }}:gha-build-${{ steps.set_branch_name.outputs.branch_name }}-${{ github.run_number }}-${{ github.run_attempt }}
${{ steps.login-ecr.outputs.registry }}/${{ env.LIVEPOLL_ECR_REPOSITORY }}:gha-build-${{ steps.set_branch_name.outputs.branch_name }}-latest
cache-from: type=gha
cache-to: type=gha,mode=max

build-and-push-pandoc:
runs-on: ubuntu-latest
name: Build and Push Pandoc Docker Image
timeout-minutes: 30
needs: [validate-terraform]
permissions:
id-token: write # Used for AWS OIDC auth
contents: read
actions: read
steps:
- uses: actions/checkout@v4
name: Checkout project

- id: read_tree_hash
name: Read git tree hash
run: |
tree_hash=$(git rev-parse HEAD:)
echo "tree_hash=$tree_hash" >> $GITHUB_OUTPUT

- id: set_branch_name
name: Read git branch name
run: |
branch_name=${GITHUB_REF##*/}
echo "branch_name=$branch_name" >> $GITHUB_OUTPUT

- uses: ./.github/actions/private-ecr-login
name: Login to Private ECR
id: login-ecr

- name: Check for prebuilt image
id: prebuilt_check
run: |
docker manifest inspect ${{ steps.login-ecr.outputs.registry }}/${{ env.PANDOC_ECR_REPOSITORY }}:gha-tree-${{ steps.read_tree_hash.outputs.tree_hash }} || echo "image_exists=$?" >> $GITHUB_OUTPUT

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
if: steps.prebuilt_check.outputs.image_exists != 0

- name: Docker build and push
uses: docker/build-push-action@v5
if: steps.prebuilt_check.outputs.image_exists != 0
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
with:
context: .
file: docker/Dockerfile-pandoc
platforms: linux/amd64
push: true
tags: |
${{ steps.login-ecr.outputs.registry }}/${{ env.PANDOC_ECR_REPOSITORY }}:gha-commit-${{ github.sha }}
${{ steps.login-ecr.outputs.registry }}/${{ env.PANDOC_ECR_REPOSITORY }}:gha-tree-${{ steps.read_tree_hash.outputs.tree_hash }}
${{ steps.login-ecr.outputs.registry }}/${{ env.PANDOC_ECR_REPOSITORY }}:gha-build-${{ steps.set_branch_name.outputs.branch_name }}-${{ github.run_number }}-${{ github.run_attempt }}
${{ steps.login-ecr.outputs.registry }}/${{ env.PANDOC_ECR_REPOSITORY }}:gha-build-${{ steps.set_branch_name.outputs.branch_name }}-latest
cache-from: type=gha
cache-to: type=gha,mode=max
30 changes: 30 additions & 0 deletions .github/workflows/manual-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: manual-deploy
on:
workflow_dispatch:
inputs:
environment:
description: 'Environment'
required: true
default: 'dev'
type: choice
options:
- dev
- staging
project:
description: 'Project'
required: true
default: 'livepoll'
type: choice
options:
- livepoll
- pandoc
jobs:
ecs-deploy:
name: ECS Deployment
uses: ./.github/workflows/shared-deploy.yml
with:
environment: ${{ github.event.inputs.environment }}
project: ${{ github.event.inputs.project }}
secrets:
aws_user_account: ${{ secrets.AWS_USER_ACCOUNT }}
aws_ci_account: ${{ secrets.AWS_CI_ACCOUNT }}
100 changes: 100 additions & 0 deletions .github/workflows/shared-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
name: shared-deployment-workflow
permissions:
id-token: write # Used for AWS OIDC auth
contents: read # This is required for actions/checkout
on:
workflow_call:
inputs:
environment:
description: 'Environment name passed from the caller workflow'
required: true
type: string
project:
description: 'Project name passed from the caller workflow'
required: true
type: string
secrets:
aws_user_account:
description: 'AWS Account ID for IAM users'
required: true
aws_ci_account:
description: 'AWS Account ID for IAM users'
required: true

jobs:
terraform-deploy:
name: Terraform Deployment
uses: ./.github/workflows/terraform-deploy.yml
with:
environment: ${{ inputs.environment }}
project: ${{ inputs.project }}
secrets:
aws_user_account: ${{ secrets.AWS_USER_ACCOUNT }}
aws_ci_account: ${{ secrets.AWS_CI_ACCOUNT }}

ecs-deploy:
name: ECS Deployment
timeout-minutes: 30
runs-on: ubuntu-latest
needs: [terraform-deploy]
steps:
- uses: actions/checkout@v4
name: Checkout project

- uses: ./.github/actions/private-ecr-login
name: Login to Private ECR
id: login-ecr

- id: read_env_json
name: Read Environment JSON
run: |
env_json=$(jq -c '.environments[] | select(.environment_label=="${{ inputs.environment }}")' ./deploy/${{ inputs.project }}-environments.json)
echo "env_json=$env_json" >> $GITHUB_OUTPUT
- id: set_env_metadata
name: Set Environment Metadata
run: |
echo "account=${{ fromJSON(steps.read_env_json.outputs.env_json).account }}" >> $GITHUB_OUTPUT
echo "ssm_prefix=${{ fromJSON(steps.read_env_json.outputs.env_json).ssm_prefix }}" >> $GITHUB_OUTPUT
echo "webapp_ssm_prefix=${{ fromJSON(steps.read_env_json.outputs.env_json).webapp.ssm_prefix }}" >> $GITHUB_OUTPUT
echo "shoryuken_ssm_prefix=${{ fromJSON(steps.read_env_json.outputs.env_json).shoryuken.ssm_prefix }}" >> $GITHUB_OUTPUT
echo "cluster_name=${{ fromJSON(steps.read_env_json.outputs.env_json).fargate.cluster_name }}" >> $GITHUB_OUTPUT

- name: OIDC Auth to AWS
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: us-west-2
role-to-assume: arn:aws:iam::${{ secrets.aws_user_account }}:role/ci-oidc-role
role-session-name: github-actions
role-duration-seconds: 900

- name: Assume role in target account
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
aws-session-token: ${{ env.AWS_SESSION_TOKEN }}
aws-region: us-west-2
role-to-assume: arn:aws:iam::${{ steps.set_env_metadata.outputs.account }}:role/ci-role
role-session-name: github-actions
role-duration-seconds: 1200

- id: read_tree_hash
name: Read git tree hash
run: |
tree_hash=$(git rev-parse HEAD:)
echo "tree_hash=$tree_hash" >> $GITHUB_OUTPUT

- name: Verify image
run: |
if ! docker manifest inspect ${{ steps.login-ecr.outputs.registry }}/${{ inputs.project }}:gha-tree-${{ steps.read_tree_hash.outputs.tree_hash }}; then
echo "If this is a PR build, you may need to pull in changes from the target branch into your PR branch."
exit 1
fi

- name: Run deploy script
run: |
deploy/ecs_deploy.sh \
--cluster-name ${{ steps.set_env_metadata.outputs.cluster_name }} \
--ssm-prefix ${{ steps.set_env_metadata.outputs.ssm_prefix }} \
--account-number ${{ secrets.aws_ci_account }} \
--project-name ${{ inputs.project }}
16 changes: 16 additions & 0 deletions .github/workflows/tag-deploy-livepoll.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: tag-deploy
on:
push:
tags:
- release/livepoll/*

jobs:
ecs-deploy:
name: ECS Deployment
uses: ./.github/workflows/shared-deploy.yml
with:
environment: prod
project: livepoll
secrets:
aws_user_account: ${{ secrets.AWS_USER_ACCOUNT }}
aws_ci_account: ${{ secrets.AWS_CI_ACCOUNT }}
16 changes: 16 additions & 0 deletions .github/workflows/tag-deploy-pandoc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: tag-deploy
on:
push:
tags:
- release/pandoc/*

jobs:
ecs-deploy:
name: ECS Deployment
uses: ./.github/workflows/shared-deploy.yml
with:
environment: prod
project: pandoc
secrets:
aws_user_account: ${{ secrets.AWS_USER_ACCOUNT }}
aws_ci_account: ${{ secrets.AWS_CI_ACCOUNT }}
Loading