| 4719 |
marundel |
1 |
Sleep Tight
|
|
|
2 |
===========
|
|
|
3 |
|
|
|
4 |
Overview
|
|
|
5 |
--------
|
|
|
6 |
Sleep tight is a python script _sleep_tight.py_ which interacts with AWS EC2 RESTful API to start and stop machines. Generally it would run it under a cron or scheduled task, though it can be run manually. Sleep tight queries tags on an instances to determine the actions it should perform.
|
|
|
7 |
|
|
|
8 |
|
|
|
9 |
Running
|
|
|
10 |
-------
|
|
|
11 |
Sleep tight can be run from the command line, or as a cron job.
|
|
|
12 |
There are currently no command line options, its operation is controlled by tags on instances.
|
|
|
13 |
|
|
|
14 |
Command line:
|
|
|
15 |
`python sleep_tight.py`
|
|
|
16 |
|
|
|
17 |
Suggested crontab
|
|
|
18 |
`*/3 * * * * python /home/tshtstsync/bin/sleep_tight.py >> /home/tshtstsync/sleep_tight.log 2>&1`
|
|
|
19 |
|
|
|
20 |
This will three times per hour, every day, all day.
|
|
|
21 |
|
|
|
22 |
|
|
|
23 |
Basic Logic Flow
|
|
|
24 |
----------------
|
|
|
25 |
Simply sleep tight will perform the following logic:
|
|
|
26 |
|
|
|
27 |
```flow
|
|
|
28 |
st=>start: Start
|
|
|
29 |
e=>end
|
|
|
30 |
|
|
|
31 |
firstInstance=>operation: Get first instance
|
|
|
32 |
getTags=>operation: Get tags from instance
|
|
|
33 |
isActionDay=>condition: Is today in ACTION_DAYS?
|
|
|
34 |
noKill=>condition: Is NO_KILL_UNTIL >= now?
|
|
|
35 |
lastStop=>condition: Haven't stopped this hour?
|
|
|
36 |
hourStop=>condition: STOP_HOUR_GMT = now?
|
|
|
37 |
stateStop=>condition: Is in running state?
|
|
|
38 |
stop=>operation: Stop instance
|
|
|
39 |
noStart=>condition: Is NO_BOOT_UNTIL >= now?
|
|
|
40 |
lastStart=>condition: Haven't started this hour?
|
|
|
41 |
hourStart=>condition: START_HOUR_GMT = now?
|
|
|
42 |
stateStart=>condition: Is in stopped state?
|
|
|
43 |
startUp=>operation: Start instance
|
|
|
44 |
moreInstances=>operation: Get next instance
|
|
|
45 |
hasInstance=>condition: End of Instances?
|
|
|
46 |
|
|
|
47 |
st->firstInstance->getTags->isActionDay
|
|
|
48 |
noKill->lastStop->hourStop->stateStop->stop
|
|
|
49 |
noStart->lastStart->hourStart->stateStart->startUp
|
|
|
50 |
moreInstances->hasInstance->getTags
|
|
|
51 |
|
|
|
52 |
|
|
|
53 |
getTags->isActionDay
|
|
|
54 |
isActionDay(yes)->noKill
|
|
|
55 |
isActionDay(no)->moreInstances
|
|
|
56 |
noKill(yes)->lastStop
|
|
|
57 |
noKill(no)->noStart
|
|
|
58 |
lastStop(yes)->hourStop
|
|
|
59 |
lastStop(no)->noStart
|
|
|
60 |
hourStop(yes)->stateStop
|
|
|
61 |
hourStop(no)->noStart
|
|
|
62 |
stateStop(yes)->stop
|
|
|
63 |
stateStop(no)->noStart
|
|
|
64 |
stop->noStart
|
|
|
65 |
noStart(yes)->lastStart
|
|
|
66 |
noStart(no)->moreInstances
|
|
|
67 |
lastStart(yes)->hourStart
|
|
|
68 |
lastStart(no)->moreInstances
|
|
|
69 |
hourStart(yes)->stateStart
|
|
|
70 |
hourStart(no)->moreInstances
|
|
|
71 |
stateStart(yes)->startUp
|
|
|
72 |
stateStart(no)->moreInstances
|
|
|
73 |
startUp->moreInstances
|
|
|
74 |
moreInstances->hasInstance
|
|
|
75 |
hasInstance(no)->getTags
|
|
|
76 |
hasInstance(yes)->e
|
|
|
77 |
```
|
|
|
78 |
|
|
|
79 |
Or in ASCII
|
|
|
80 |
```
|
|
|
81 |
|
|
|
82 |
|
|
|
83 |
(START)
|
|
|
84 |
|
|
|
|
85 |
|--------<--------
|
|
|
86 |
| |
|
|
|
87 |
/ \ |
|
|
|
88 |
/ \ |
|
|
|
89 |
(END) ---No- / Next \ |
|
|
|
90 |
\Instance?/ |
|
|
|
91 |
\ / |
|
|
|
92 |
\ / |
|
|
|
93 |
| ^
|
|
|
94 |
Yes |
|
|
|
95 |
| |
|
|
|
96 |
------------ |
|
|
|
97 |
| Get Tags | |
|
|
|
98 |
------------ |
|
|
|
99 |
| |
|
|
|
100 |
/ \ |
|
|
|
101 |
/ Is \ ^
|
|
|
102 |
/ Action \ -No----->--|
|
|
|
103 |
\ Day? / |
|
|
|
104 |
\ / |
|
|
|
105 |
\ / |
|
|
|
106 |
| |
|
|
|
107 |
Yes |
|
|
|
108 |
| |
|
|
|
109 |
/ \ ^
|
|
|
110 |
/ >= \ |
|
|
|
111 |
/ NO KILL \ -No-->- |
|
|
|
112 |
\ UNITL? / | |
|
|
|
113 |
\ / | |
|
|
|
114 |
\ / | |
|
|
|
115 |
| | |
|
|
|
116 |
Yes | |
|
|
|
117 |
| | ^
|
|
|
118 |
| | |
|
|
|
119 |
/ \ v |
|
|
|
120 |
/ Have\ | |
|
|
|
121 |
/ Stopped \ -Yes->-| |
|
|
|
122 |
\ this / | |
|
|
|
123 |
\ hour/ | |
|
|
|
124 |
\ / | |
|
|
|
125 |
| | ^
|
|
|
126 |
No | |
|
|
|
127 |
| v |
|
|
|
128 |
| | |
|
|
|
129 |
/ \ | |
|
|
|
130 |
/ \ | |
|
|
|
131 |
/STOP_HOUR\ -No-->-| |
|
|
|
132 |
\ now? / | |
|
|
|
133 |
\ / | ^
|
|
|
134 |
\ / | |
|
|
|
135 |
| v |
|
|
|
136 |
Yes | |
|
|
|
137 |
| | |
|
|
|
138 |
| | |
|
|
|
139 |
/ \ | |
|
|
|
140 |
/ \ | |
|
|
|
141 |
/ State \ -No-->-| ^
|
|
|
142 |
\ Running?/ | |
|
|
|
143 |
\ / v |
|
|
|
144 |
\ / | |
|
|
|
145 |
| | |
|
|
|
146 |
Yes | |
|
|
|
147 |
| | |
|
|
|
148 |
| | |
|
|
|
149 |
-------- | ^
|
|
|
150 |
| STOP | | |
|
|
|
151 |
-------- v |
|
|
|
152 |
| | |
|
|
|
153 |
|------<------ |
|
|
|
154 |
| |
|
|
|
155 |
/ \ |
|
|
|
156 |
/ >= \ |
|
|
|
157 |
/ NO STRT \ -No----->--^
|
|
|
158 |
\ UNITL? / |
|
|
|
159 |
\ / |
|
|
|
160 |
\ / |
|
|
|
161 |
| |
|
|
|
162 |
Yes |
|
|
|
163 |
| |
|
|
|
164 |
| |
|
|
|
165 |
/ \ ^
|
|
|
166 |
/ Have\ |
|
|
|
167 |
/ Started \ -Yes---->--|
|
|
|
168 |
\ this / |
|
|
|
169 |
\ hour/ |
|
|
|
170 |
\ / |
|
|
|
171 |
| |
|
|
|
172 |
No |
|
|
|
173 |
| ^
|
|
|
174 |
| |
|
|
|
175 |
/ \ |
|
|
|
176 |
/ \ |
|
|
|
177 |
/STRT_HOUR\ -No----->--|
|
|
|
178 |
\ now? / |
|
|
|
179 |
\ / |
|
|
|
180 |
\ / |
|
|
|
181 |
| ^
|
|
|
182 |
Yes |
|
|
|
183 |
| |
|
|
|
184 |
| |
|
|
|
185 |
/ \ |
|
|
|
186 |
/ \ |
|
|
|
187 |
/ State \ -No----->--|
|
|
|
188 |
\ Stopped?/ |
|
|
|
189 |
\ / ^
|
|
|
190 |
\ / |
|
|
|
191 |
| |
|
|
|
192 |
Yes |
|
|
|
193 |
| |
|
|
|
194 |
| |
|
|
|
195 |
---------- |
|
|
|
196 |
| START UP | |
|
|
|
197 |
---------- ^
|
|
|
198 |
| |
|
|
|
199 |
-------------->--
|
|
|
200 |
|
|
|
201 |
```
|
|
|
202 |
|
|
|
203 |
Tags
|
|
|
204 |
----
|
|
|
205 |
USER tags are those configurable by the user.
|
|
|
206 |
APP tags, or application tags are those used by sleep tight to store state
|
|
|
207 |
Any non optional tags are created and set to the default values if they are not present on the instance.
|
|
|
208 |
|
|
|
209 |
Tag Type | Tag | Default | Description
|
|
|
210 |
-------- | --------------- | ---------- | --------------
|
|
|
211 |
USER | Name | | (Optional) Used in output messages to help identify an instance.
|
|
|
212 |
USER | START_HOUR_GMT | 99 | The hour in GMT/UTC which the instance is to be started.
|
|
|
213 |
USER | STOP_HOUR_GMT | 10 | The hour in GMT/UTC which the instance is to be stopped.
|
|
|
214 |
USER | NO_BOOT_UNTIL | 2030-01-01 | Only attempt to start the instance after this date.
|
|
|
215 |
USER | NO_KILL_UNTIL | 2000-12-31 | Only attempt to stop the instance after this date.
|
|
|
216 |
USER | ACTION_DAYS | MTWHF__ | The days of the week to start or stop.
|
|
|
217 |
APP | LAST_AUTO_START | | The date and hour GMT which sleep tight started the instance
|
|
|
218 |
APP | LAST_AUTO_STOP | | The date and hour GMT which sleep tight stopped the instance
|
|
|
219 |
|
|
|
220 |
#### Name
|
|
|
221 |
The name is an optional tag which is queried from the instance for use in log messages, if this tag is not present the instance id is used.
|
|
|
222 |
e.g.
|
|
|
223 |
2014-11-28 07 : i-18592226 80 99 10 --- Do Nothing for TSH-TST-EPG-SERVER
|
|
|
224 |
^ instance id used if name tag is missing ^ name is append to end of message.
|
|
|
225 |
|
|
|
226 |
#### START_HOUR_GMT
|
|
|
227 |
If the current GMT hour is the same as this value sleep tight may attempt to start the instance, provided other conditions are met. In order to stop sleep tight from starting the instance set it to a number larger than 24.
|
|
|
228 |
**Default:** 99*
|
|
|
229 |
> **Note:** This value must be an integer.
|
|
|
230 |
> **Note:** START_HOUR_GMT and STOP_HOUR_GMT cannot be the same value even if _disabled_ with a larger value.
|
|
|
231 |
|
|
|
232 |
#### STOP_HOUR_GMT
|
|
|
233 |
If the current GMT hour is the same as this value sleep tight may attempt to stop the instance, provided other conditions are met. In order to prevent sleep tight from stopping the instance set it to a number larger than 24.
|
|
|
234 |
**Default:** *10*
|
|
|
235 |
> **Note:** This value must be an integer.
|
|
|
236 |
> **Note:** START_HOUR_GMT and STOP_HOUR_GMT cannot be the same value even if _disabled_ with a larger value.
|
|
|
237 |
|
|
|
238 |
#### NO_BOOT_UNTIL
|
|
|
239 |
Sleep tight will only attempt to start the instance if the current GMT hour is past midnight GMT of the supplied date.
|
|
|
240 |
The date must be in the format YYYY-MM-DD where YYYY is a 4 digit year, MM is a two digit month padded with zeros for months Jan-Sept, DD is a two digit date padded with zeros for days 1-9.
|
|
|
241 |
The following are valid values:
|
|
|
242 |
: 2014-01-12
|
|
|
243 |
: 2014-09-09
|
|
|
244 |
: 2014-12-31
|
|
|
245 |
**Default:** *2030-01-01*
|
|
|
246 |
> **Note:** If the date supplied is not a valid date sleep tight will apply the default NO_BOOT_UNTIL date to logic processing.
|
|
|
247 |
|
|
|
248 |
#### NO_KILL_UNTIL
|
|
|
249 |
Sleep tight will only attempt to stop the instance if the current GMT hour is past midnight GMT of the supplied date.
|
|
|
250 |
The date must be in the format YYYY-MM-DD where YYYY is a 4 digit year, MM is a two digit month padded with zeros for months Jan-Sept, DD is a two digit date padded with zeros for days 1-9.
|
|
|
251 |
The following are valid values:
|
|
|
252 |
: 2014-01-12
|
|
|
253 |
: 2014-09-09
|
|
|
254 |
: 2014-12-31
|
|
|
255 |
**Default:** *2030-01-01*
|
|
|
256 |
> **Note:** If the date supplied is not a valid date sleep tight will apply the default NO_KILL_UNTIL date to logic processing.
|
|
|
257 |
|
|
|
258 |
#### ACTION_DAYS
|
|
|
259 |
Specify the day of the week in which to apply start or stop actions. Sleep tight will attempt to lookup the current day in the supplied tag, if that character is present action processing will continue. Any characters other than those listed have no affect, and the list of days may be in any order.
|
|
|
260 |
|
|
|
261 |
Character | Day of Week
|
|
|
262 |
--------- | ------------
|
|
|
263 |
M | Monday
|
|
|
264 |
T | Tuesday
|
|
|
265 |
W | Wednesday
|
|
|
266 |
H | Thursday
|
|
|
267 |
F | Friday
|
|
|
268 |
S | Saturday
|
|
|
269 |
U | Sunday
|
|
|
270 |
|
|
|
271 |
The following are valid values:
|
|
|
272 |
: M
|
|
|
273 |
: MTWHFSU
|
|
|
274 |
: WMFUTHS
|
|
|
275 |
: __--++U++__--ZZZ
|
|
|
276 |
**Default:** *MTWHF__*
|
|
|
277 |
> **Note:** Thursday, and Sunday are not the first letter of their name, the value represent Thursday is **H**, while Sunday is **U**
|
|
|
278 |
> **Note:** Currently only uppercase letters are searched for. Lowercase letters will be ignored.
|
|
|
279 |
|
|
|
280 |
|
|
|
281 |
Logging
|
|
|
282 |
-------
|
|
|
283 |
Sleep tight does not log to a file. All messages are printed to standard out. It is recommended if using a cron job you redirect this to a file.
|
|
|
284 |
The messages are generally in the form.
|
|
|
285 |
DATE HOUR : INSTANCE_ID INSTANCE_STATE START_HOUR STOP_HOUR --- ACTION_MESSAGE for INSTANCE_NAME
|
|
|
286 |
e.g.
|
|
|
287 |
2014-11-27 07 : i-09eca937 80 99 10 --- Stopping - no action already shutting/shutdown for EBRIO-Bastion
|
|
|
288 |
2014-11-27 10 : i-18592226 16 99 10 --- Stopping instance for TSH-TST-EPG-SERVER
|
|
|
289 |
2014-11-28 07 : i-5bb72794 80 99 10 --- Do Nothing for COCT-EPG-TEST
|
|
|
290 |
2014-11-28 07 : i-def4b1e0 16 98 99 --- (boot:2030-01-01) (kill:2030-01-01) no kill until for TSH-Sync-Server
|
|
|
291 |
2014-11-28 07 : i-c1ac300e 16 23 10 --- (boot:2000-01-01) (kill:2030-01-01) no kill until(boot:2000-01-01) (kill:2030-01-01) no boot until for pfsense_VPN
|
|
|
292 |
|
|
|
293 |
> **Note:** Exceptions from python or Amazon may be present in the logs in their standard representation.
|
|
|
294 |
|
|
|
295 |
|
|
|
296 |
Python Libraries
|
|
|
297 |
----------------
|
|
|
298 |
Sleep tight uses the following python libraries:
|
|
|
299 |
boto.ec2
|
|
|
300 |
datetime
|
|
|
301 |
sys
|
|
|
302 |
time
|
|
|
303 |
|
|
|
304 |
|
|
|
305 |
AWS Configuration
|
|
|
306 |
-----------------
|
| 5800 |
marundel |
307 |
Sleep tight looks for a configuration file ~/.boto
|
| 4719 |
marundel |
308 |
The file should contain a [sleep_tight] record header with the following information:
|
|
|
309 |
|
|
|
310 |
[sleep_tight]
|
|
|
311 |
region = ap-southeast-2
|
|
|
312 |
aws_access_key_id = <replace-with-access-key-id>
|
|
|
313 |
aws_secret_access_key = <replace-with-secret-access-key>
|
|
|
314 |
|
|
|
315 |
|
|
|
316 |
AWS IAM Permissions
|
|
|
317 |
-------------------
|
| 5800 |
marundel |
318 |
Create a new IAM user or modify an existing user.
|
| 4719 |
marundel |
319 |
Sleep tight at a minimum requires ec2 Describe anything, StartInstances, StopInstances permissions
|
|
|
320 |
|
|
|
321 |
Describe any permissions
|
|
|
322 |
{
|
|
|
323 |
"Version": "2012-10-17",
|
|
|
324 |
"Statement": [
|
|
|
325 |
{
|
|
|
326 |
"Effect": "Allow",
|
|
|
327 |
"Action": "ec2:Describe*",
|
|
|
328 |
"Resource": "*"
|
|
|
329 |
}
|
|
|
330 |
]
|
|
|
331 |
}
|
|
|
332 |
|
|
|
333 |
Start and Stop Instances permissions
|
|
|
334 |
{
|
|
|
335 |
"Version": "2012-10-17",
|
|
|
336 |
"Statement": [
|
|
|
337 |
{
|
|
|
338 |
"Effect": "Allow",
|
|
|
339 |
"Action": [
|
|
|
340 |
"ec2:StartInstances",
|
|
|
341 |
"ec2:StopInstances"
|
|
|
342 |
],
|
|
|
343 |
"Resource": [
|
|
|
344 |
"arn:aws:ec2:ap-southeast-2:786476282193:instance/*"
|
|
|
345 |
],
|
|
|
346 |
}
|
|
|
347 |
]
|
|
|
348 |
}
|
|
|
349 |
|
|
|
350 |
|
|
|
351 |
Known Bugs and Limitations
|
|
|
352 |
--------------------------
|
|
|
353 |
Multiple regions has not yet been tested.
|
|
|
354 |
Currently will not assign an elastic IP address on startup to an instance.
|
|
|
355 |
Defaults to stopping instances at 10am GMT time, 6pm Perth time, there is no configuration file or similar to change this default without a variable in the code.
|
|
|
356 |
No support for timezones other than GMT.
|
|
|
357 |
Defaults NO_BOOT_UNTIL and NO_KILL_UNTIL if the dates are not in the correct format..
|
|
|
358 |
Currently instance appears to need a public/elastic IP address to talk to AWS, though this could be VPC configuration.
|
|
|
359 |
|
|
|
360 |
|
|
|
361 |
> Good night, sleep tight, don't let the bed bugs bite.
|
|
|
362 |
|