Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

monitor runner custom service returns 404 not found for the particular url #1246

Open
tonystaark opened this issue Feb 19, 2025 · 1 comment

Comments

@tonystaark
Copy link

This returns 404 since the instance metadata url does not have autoscaling option as its path. Therefore will never execute this logic.

Image Image

Any idea how this was implemented at the first place?

@kayman-mk
Copy link
Collaborator

Hm, not sure why you see this. Checked it on my agent and all urls are working as expected. Which one does not work on your instance?

imds_token=$(curl -sSf -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
instance_id=$(curl -sSf -H "X-aws-ec2-metadata-token: $imds_token" "http://169.254.169.254/latest/meta-data/instance-id")
region=$(curl -sSf -H "X-aws-ec2-metadata-token: $imds_token" "http://169.254.169.254/latest/meta-data/placement/region")
# Function to send complete-lifecycle-action
send_complete_lifecycle_action() {
echo "Sending complete-lifecycle-action for instance $instance_id..."
if [ -z "$region" ]; then
echo "Failed to retrieve AWS region."
exit 1
fi
echo "Current AWS Region: $region"
# Retrieve the Auto Scaling Group name associated with the instance
asg_name=$(aws --region $region ec2 describe-tags --filters "Name=resource-id,Values=$instance_id" "Name=key,Values=aws:autoscaling:groupName" --query 'Tags[].Value' --output text)
if [ -z "$asg_name" ]; then
echo "Failed to retrieve Auto Scaling Group name."
exit 1
fi
echo "Auto Scaling Group Name: $asg_name"
# Retrieve the lifecycle hook names associated with the Auto Scaling Group
lifecycle_hooks=$(aws --region $region autoscaling describe-lifecycle-hooks --auto-scaling-group-name $asg_name --query "LifecycleHooks[?LifecycleTransition=='autoscaling:EC2_INSTANCE_TERMINATING'].LifecycleHookName" --output text)
if [ -z "$lifecycle_hooks" ]; then
echo "No lifecycle hooks found for Auto Scaling Group: $asg_name"
exit 1
fi
echo "Lifecycle Hooks: $lifecycle_hooks"
# This is needed to let CloudWatch Logs catch up.
echo "Will send complete actions in 10 seconds..."
sleep 10
for hook_name in $lifecycle_hooks; do
echo "Completing lifecycle action for hook: $hook_name"
aws autoscaling complete-lifecycle-action \
--lifecycle-hook-name $hook_name \
--auto-scaling-group-name $asg_name \
--lifecycle-action-result CONTINUE \
--instance-id $instance_id \
--region $region
if [ $? -eq 0 ]; then
echo "Lifecycle action completed successfully for hook: $hook_name"
else
echo "Failed to complete lifecycle action for hook: $hook_name"
fi
done
}
# Function to monitor the existence of the Runner process
monitor_process() {
systemctl --no-block stop gitlab-runner.service
sleep 5 # grace period to allow GitLab Runner to shutdown
while true; do
status=$(systemctl is-active gitlab-runner.service)
if [ "$status" = "inactive" ] || [ "$status" = "failed" ]; then
echo "GitLab Runner service $status. Proceeding with cleanup."
send_complete_lifecycle_action
else
echo "GitLab Runner Service still running, sleeping..."
fi
# it usually takes a while to complete the jobs, so don't poll too often
sleep 30
done
}
# Main function
main() {
# Poll whether the instance is in the terminating state using instance metadata API
imds_token=$(curl -sSf -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
target_lifecycle_state=$(curl -sSf -H "X-aws-ec2-metadata-token: $imds_token" "http://169.254.169.254/latest/meta-data/autoscaling/target-lifecycle-state")
if [ "$target_lifecycle_state" = "Terminated" ]; then
# Monitor the existence of the Runner process
monitor_process
else
if [ "${user_data_trace_log}" = "true" ]; then
echo "Instance target lifecycle state is $target_lifecycle_state. No action required."
fi
fi
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants