On my web server on Amazon Web Services (AWS) I had a T2.micro instance running Ubunto and a web server.
After all installed and up and running I started gazing at the CloudWatch Monitoring charts. Here is one with the default options:
With the help of AWS documentation I recalled the definitions that would help me interpret this graph.
On the AWS documentation at View Amazon EC2 Metrics we can read the explanation about the CPU Credit Usage metrics: “The number of CPU credits consumed during the specified period.”.
So, hovering on the original chart “CPU Credit Usage” I read that, for example, at 2014-09-29 03:19 the server consumed 0.15 credits during 5 minutes, on average. That corresponds to 1.8 credits over one hour. Looking at the graph we can say that the average is about 0.16, which corresponds to 1.92 credits. Let’s assume 2 credits spent in this hour for simplicity. A T2.micro is given 6 CPU credits each hour as we can see in Amazon EC2 T2 instances. So, over this hour, 4 CPU credits were not spent and went right into the CPU Credit Balance.
Then I changed two parameters of the chart: changed “Time Range” to the “Last 24 Hours” and “Period” to “1 Hour”. This is the chart:
Here we see that credit usage has been between 0.1 and .2 for most of the time with a spike to near 1.0, just before 14:00. If you do not pay much attention to the 5 min period chart you tend to think that the values on this last chart are the credit usage by hour. Indeed the documentation indicates “The number of CPU credits consumed during the specified period.” as already quoted. And, the specified period is 1 hour, on that last chart. But no, the “specified period” continues to be the sampling period of 5 minutes. So, to make this all come right, every time you change the “Period” selector to other than 5 minutes, you must change the “Statistic” selector to “SUM”.
And now the value agrees with the ones we saw on the first chart: just about 2 credits per hour in that 14:55 to 15:50 time period of the first chart.
For the chart with the period of 5 minutes, choosing “AVG” or “SUM” gives the same, since there is only one value to sum or average.
Wanting to confirm all this, I’ve managed to put the machine into 100% CPU load for 10 minutes with the command:
$ stress --timeout 10m --cpu 1
Here are the charts:
Having the CPU at 100% for 10 minutes, I consumed 10 credits. Since T2.micro receives 1 credit per each 10 minutes, I should have a decrease of 9 credits in the balance over that period. And in fact it lowered from 112.77 to 103.42. And note that the three points of the spike in the credit usage chart sum about 10: 1,95+4,99+3,21.
At this point I could read and make sense of the “CPU Credit Usage” chart. Time to bring in the “CPU Credit Balance”:
Earlier in this text I concluded that for the period of around 14:55 to 15:55 the server spent just about 2 credits for that hour. That would add 4 credits to the balance.
So what I did not understand is how is the Credit Balance was flat for that time period, instead of raising 4 credits. For other periods there were also other similar questions, like, for the last 24 hours, the credit usage is bellow 2 per hour, but the Credit Balance descends, although slowly.
Although I had read about the credits added to the balance would expire: “If at any moment the instance does not need the credits it receives, it stores them in its CPU Credit balance for up to 24 hours.” But it was only after I read this thread on AWS forums that things became clear.
I’ll try to explain with this scenarios. Note that for my instance type, a t2.micro, the 6 credit per hour corresponds to a constant rate of 10% cpu usage. That is because, AWS defines one credit is a full 100% CPU usage for one minute.
Scenario 1.
- start with 0 credit balance
- CPU spends 1h at 10% usage, constantly for simplicity. Earned 6 credits and spent those 6 credits.
- balance remains at 0.
Scenario 2.
- start with 0 credit balance
- 1h at 5% CPU usage. Earned 6 credits and spent 3 of those 6 credits.
- balance goes up by 3 credits. These 3 credits will be subtracted from balance after 24 hours.
Scenario 3. Just after 23 hours after scenario 2.
- start with 3 credit balance
- 1h at 5% CPU usage. Earned 6 credits and spent 3 of those 6 credits.
- balance goes up by 3 credits. Those 3 credits just earned.
- balance goes down by 3 credits. The 3 credits from scenario 2 that just expired.
- balance chart shows no change, remaining at 3 credits.
So on this scenario 3, you seem to be earning 3 credits on that hour, and at first though you expect that your balance grow by 3 units. But that really depends on the expiring credits. This is something difficult (impossible?) to see in those charts.
I’m giving up looking at charts at this point, since (I think) I’ve grasped the mechanism.
Have fun!
Thanks for clarifying how Amazon credits work! I was rather worried at the precipitous drop in my credit balance over the last few days as I migrated over a bunch of production sites from another server.
What is not clear from the Amazon documentation is that the calculation of the credit balance changes after the first 24 hours. In addition to factoring in the impact of CPU utilization on the next credit usage, it is also important to compare the current CPU with the value 24 hours in the past. As you point out, if CPU utilization is steady, any current credits will be offset by expired credits, resulting in a steady balance. If CPU utilization is increasing, then current credits will be less than the expired credits, resulting in a declining balance even if the current CPU is less than the base performance. Conversely, if CPU utilization is decreasing, current credits will be greater than expired credits, resulting in an increasing balance.
Things get a bit complicated if the average CPU 24 hours ago was above the base performance since no credits were accumulated and therefore no credits are expired. The credit balance is only affected by the net credit usage.