GH action to generate report

[asmacdo]

Fixes #177

Step 1: Create Skeleton

• Authenticate with AWS
• Connect to K8s cluster
• deploys our job-runner pod onto a Karpenter NodeClaim
• Creates a SPOT node as needed
• Run dummy job
• Delete Pod
• Scale Down

I've verified that when a user-node is available (created by running a tiny jupyterhub), the job pod schedules on that node. I then shut down my jupyterhub and all user-nodes scaled down. I reran this job, and Karpenter successfully scaled up a new spot node, the pod was scheduled on it, ran successfully, was deleted, and the node cleaned up. Step 1 complete!

Step 2 Generate Report

• Connect Pod to EFS
• List users
• du each user
• du shared
• collate data into report
• Double Check that nodes come up and down successfully
• Run job several times in 1 day, check next day for EFS usage spike (IIUC we should be fine because EFS is Bursting mode)

Step 3 Push Report

• Create private GitHub repository to store reports
• Configure bot permission to push to repo
• push report to repo on complete

Questions to answer:

• If a SPOT node is preempted, can we redeploy again later?

[kabilar]

Should we also run this on a weekly basis?

[asmacdo]

It's on PR push just so I can test easily, but yes, 1/week sounds good to me. @kabilar Do you have a preference for what day/time?

[kabilar]

Great, thank you. How about Mondays at 6am EST? We can then review the report on Monday mornings.