Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete stale servers after N hours #27

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bastelfreak
Copy link
Member

No description provided.

@codecov
Copy link

codecov bot commented Sep 22, 2023

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (41229a0) 92.30% compared to head (8acead4) 91.20%.

❗ Current head 8acead4 differs from pull request most recent head f5bd909. Consider uploading reports for the commit f5bd909 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #27      +/-   ##
==========================================
- Coverage   92.30%   91.20%   -1.10%     
==========================================
  Files           3        3              
  Lines          78       91      +13     
==========================================
+ Hits           72       83      +11     
- Misses          6        8       +2     
Files Coverage Δ
lib/beaker/hypervisor/hcloud.rb 92.53% <84.61%> (-1.91%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bastelfreak
Copy link
Member Author

There's no way to automatically delete VMs that are older than X, so we've to implement this in beaker-hcloud. I did some manual poking in the IRB and that worked fine. I didn't test this code yet.

Copy link
Member

@ekohl ekohl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any thoughts how concurrency would work? What if 2 CI jobs run a the same time and both try to delete the server?

# "alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between."
# https://docs.hetzner.cloud/#labels
def vm_deletion_date
(DateTime.current + @delete_vm_after.hours).to_i.to_s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also use native Ruby to avoid the ActiveSupport dependency

Suggested change
(DateTime.current + @delete_vm_after.hours).to_i.to_s
(DateTime.now + (@delete_vm_after /24)).to_i.to_s

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah right, that works as well. We pull in activesupport anyways because it's a dependency for hcloud. So should we use it here?

Comment on lines +90 to +91
server.destroy if server.labels['delete_vm_after'] && Time.now > Time.at(server.labels['delete_vm_after'].to_i)
server.destroy unless server.labels['delete_vm_after']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about a more traditional condition:

Suggested change
server.destroy if server.labels['delete_vm_after'] && Time.now > Time.at(server.labels['delete_vm_after'].to_i)
server.destroy unless server.labels['delete_vm_after']
if !server.labels['delete_vm_after'] || Time.now > Time.at(server.labels['delete_vm_after'].to_i)
server.destroy
end

raise 'You need to pass a token as BEAKER_HCLOUD_TOKEN environment variable' unless ENV['BEAKER_HCLOUD_TOKEN']

@client = ::Hcloud::Client.new(token: ENV.fetch('BEAKER_HCLOUD_TOKEN'))
delete_servers_if_required
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing this in the constructor is dangerous. It means you can't really write unit tests.

Why don't you add this to the cleanup? There is a risk that if things crash you never get to the cleanup, but you can also consider writing a specific clean up only task that runs X times a day in some common repository.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that. maybe we just run a github action as cron somewhere that does the cleanup. That also fixes the problems of cuncurrent cleanup jobs

@@ -45,7 +45,8 @@
'dns_ptr' => 'server1.example.com',
},
},
destroy: true)
destroy: true,
labels: { vm_delete_after: '1695385549' })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be delete_vm_after?

@@ -56,7 +57,8 @@
'dns_ptr' => 'server2.example.com',
},
},
destroy: true)
destroy: true,
labels: { vm_delete_after: '1695385549' })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be delete_vm_after?

Look for servers in your project named exactly as the ones in your beaker host configuration and SSH keys with names beginning with `Beaker-`.

The gem will try to automatically delete old virtual machines. Every created
cloud instance gets a label `delete_vm_after: 'Fri, 22 Sep 2023 15:39:17 +0200'`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timestamp formats here (human-readable with TZ) and in the unit tests below (unix time w/o TZ) are different. Is it expected?

@bastelfreak
Copy link
Member Author

I raised a new PR that just adds the label, without any VM deletion: #29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants