Are you using Memcached Distributed Memory Cache in your application? You can use CopperEgg to monitor your Memcached Installation for availability and performance. As an illustration, you can configure and receive alerts for the following scenarios:

  • Availability: alert if the Memcached instance is no longer available.
    • If the Server CopperEgg value falls to 0, it implies that the server is no longer available for monitoring.
    • If the auth_errors is abnormally high, that indicates that too many Authentication Requests are failing, possibly because the Memcached installation is under attack by a hacker.
  • Performance: alert if the Memcached instance performance is poor.
    • If the curr_connections value (Open Connections count) is significantly above its normal range, that would imply that the Memcached server is under heavy load.
    • Bytes_read and bytes_written values track the read-write performance of the Memcached installation
    • If the get_misses value increases significantly above its normal range, that is a disturbing indication that too many keys are resulting in cache misses and requires further investigation.

The full set of Memcached performance metrics that are monitored is available here

Prerequisites

CopperEgg offers a SAAS based out-of-the-box monitoring solution for Memcached that is easy to set up. If you are not familiar with the concept of custom metrics as used in CopperEgg, please go through Custom Metrics information once before continuing with this document.  More details on Custom Metrics can be found.

  1. CopperEgg account. If you don’t already have an account, you can sign up for a 14 day FREE TRIAL of CopperEgg here (no Credit Card required).
  2. Memcached up and running in your environment (this can be on your own server or using Amazon Web Services’ Elasticache.)
  3. Root access to server to install the Memcached agent.

Monitor Memcached using out-of-the-box custom metrics installer from CopperEgg

There are three steps that you need to do to start monitoring a Memcached service:

  1. Memcache Agent Installation
  2. Verify that your Memcache Dashboard is up and running
  3. Configure alerts to be notified about Memcached performance issues

1.Memcached Agent Installation

Figure 1: In your CopperEgg Account, navigate to Custom>Getting Started

Login to your CopperEgg account and navigate to Custom tab -> Getting Started. Click on Memcached icon.

Figure 2: The Memcached agent installation script

Click on icon to copy the installer script to your clipboard. Paste the command into the terminal window on the server where you want to install the agent.  Script can be run on the server running Memcached, or it can be a different server, but this server will need to be continually running in order to monitor Memcached.  (If you’re using AWS Elasticache and you don’t have direct server access to the memcached node you’ll need to use a different EC2 instance.   Make sure your security groups and any other networking configuration are setup to allow this server to access Elasticache.)

As soon as the script is run on a terminal with root access, it starts downloading the required Memcached monitoring agent on your server. You will need to configure the following items in the agent installation process. Item specified in square brackets ( [ ] ) are the default.

Figure 3: Executing the Memcached agent installation script with user as copperegg, Group as copperegg and monitoring frequency as 15 seconds

  1. User and Group for running service as default values: [copperegg][copperegg]
  2. Monitoring frequency: 15, [60], 300, 900, 3600 seconds

Figure 4: Executing the Memcached agent installation script with group name as “memcached”, group label as “Memcached Metrics” and Dashboard  as “Memcached”

3. Group label, group name and dashboard name for custom metrics [memcached][Memcached Metrics][Memcached]

Figure 5: Executing the Memcached agent installation script with unique id as local.memcached.server, server URL as localhost and port 11211. Only one Memcached server is added for monitoring.

4. Unique name for this node (good to differentiate if you are monitoring different memcached nodes). [<user>-memcached]

5. Server details : URL and Port for service. [localhost][11211]

6. After configuring one node, you’ll get an option to add more memcached nodes.

Figure 6: Executing the Memcached agent installation script with an upstart init file and the default value for log file name. The installation completes successfully!

7. Configure log file name and upstart job name (defaults present) [/usr/local/copperegg/log/memcached_metrics.log][/etc/init/revealmetrics_memcached.conf]

2.Verify that your Memcached Dashboard is up and running

Navigate to Dashboard and you will see a new Memcached Dashboard has been added automatically by the installer.

Figure 7: A new Dashboard for Memcached has been added

As the installed script executes, you will notice data samples appearing on the widgets for this Dashboard.

Figure 8: Widget inside Memcached Dashboard displaying CopperEgg. The CopperEgg value gradually increases over time as the Memcached service continues to remain operational.

Figure 9: Widget inside Memcached Dashboard displaying count of open connections.

Figure 10: Widget inside Memcached Dashboard displaying count of connection structures.

Monitored metrics

CopperEgg currently monitors 42 metrics which cover most of the important metrics for memcached from a performance and stability perspective. You can see the same by navigating to the Custom Tab > Custom Objects and selecting Details on the Custom Metrics Group Label [Memcached Metrics] that was given as installation input.

The metrics are:

  • cmd_get : Retrieval requests
  • cmd_flush : Storage requests
  • get_misses : Keys requested and found missing
  • bytes : Bytes used
  • CopperEgg: Server CopperEgg
  • rusage_user : Accumulated user time
  • rusage_system : Accumulated system time
  • curr_connections : Open connections
  • total_connections : Total connections
  • connection_structures : Connection structures
  • reserved_fds : Misc fds used internally
  • cmd_flush : Flush requests
  • cmd_touch : Touch requests
  • get_hits : Keys requested and found present
  • delete_misses : Deletion requests for missing keys
  • delete_hits : Deletion requests for existing keys
  • incr_misses : Increment requests against missing keys
  • incr_hits : Increment requests against existing keys
  • decr_misses : Decrement requests against missing keys
  • decr_hits : Decrement requests against existing keys
  • cas_misses : ‘Check and set’ requests against missing keys
  • cas_hits : ‘Check and set’ requests against existing keys
  • cas_badval : ‘Check and set’ requests for existing key & bad ‘Check and set’ value
  • touch_hits : Keys touched with new expiration time
  • touch_misses : Keys touched and not found
  • auth_cmds : Auth commands handled
  • auth_errors : Failed authentications
  • bytes_read : Bytes read
  • bytes_written : Bytes written
  • limit_maxbytes : Bytes allowed for storage
  • accepting_conns : Check if server is accepting connections
  • threads : Worker threads requested
  • conn_yields : Connections yielded to hit limit
  • hash_power_level : Current size multiplier for hash table
  • hash_bytes : Bytes used by hash tables
  • hash_is_expanding : Indicates if hash table is expanding
  • expired_unfetched : Items pulled from LRU never used before expiring
  • evicted_unfetched : Items evicted from LRU that were never touched
  • curr_items : Current items stored
  • total_items : Total items stored
  • evictions : Items removed from cache to free memory
  • reclaimed : No. of times entry stored using memory from expired entry

More details on these metrics can be obtained from the Memcached Wiki.

3.Configure Alerts to be notified about Memcached performance issues

Default Pre-Configured Alerts for the Memcached Database

As soon as your Memcached Database Monitoring Dashboard comes up, you will notice that some alert configurations have been automatically created by our Installation Script (under Alerts > Configure Alerts). These pre-configured alerts for Memcached Database include:

Sr. No. Metric Name Definition Alert Definitions Duration
1 Auth errors Number of failed authentications Alert will be triggered for all errors. 3 Min
2 Accepting Connections Whether or not server is accepting conns Warning Alert will be triggered when size becomes 60% of memory. 3 Min
3 Delete misses The number of delete requests the cache has received where the requested key was not found Warning Alert will be triggered for for every delete request for missing key 3 Min
4 Get misses The number of get requests the cache has received where the key requested was not found Warning Alert will be triggered for for every get request for missing key 3 Min
5 Incr misses The number of increment requests the cache has received where the key requested was not found Warning Alert will be triggered for for every increment request for missing key 3 Min
6 Decr misses The number of decrement requests the cache has received where the requested key was not found Warning Alert will be triggered for for every decrement request for missing key 3 Min
7 Cas misses The number of Cas requests the cache has received where the key requested was not found Warning Alert will be triggered for for every CAS request for missing key 3 Min
8 Touch misses The number of items that have been touched, but were not found Warning Alert will be triggered for for every touch request for missing key 3 Min

If the alerting condition is met for these pre-configured alerts, some alert notifications may also be triggered.

Configuring your own Alerts for the Memcached Database

Figure 11: A new alert being added that will be triggered when Memcached Open Connections metric exceeds a threshold of 2000 within a time period of 1 min

Alerts can also be configured against any of these 42 monitored metrics for Memcached

You can configure new alerts that will be triggered when performance issues with Memcached arise. You can also configure the notification mechanisms for a triggered alert.

  1. Go to Alerts Tab > Configure alerts and click on “New Alerts” button
  2. Provide values for these fields in the New Alert page:
  • Description: A description of the alert that will be easily recognized by you and your team if the alert is triggered
  • Alert me when: Select the metric of interest and the condition upon which the alert is triggered. In the Alert me when dropdown, you can prefill “Custom: <your Memcached Metric Group>” to get only the list of monitored metrics for your specific Memcached installation. From this list, you can then select the specific metric for which you want to configure an alert. As an illustration, in Figure 11, the chosen metric of interest is ‘Open Connections’ from the MemcacheNDTest Metrics group and the condition for triggering the alert is if the average value is more than 2000.
  • For at least: The duration for which the alert condition must be valid for the alert to be triggered
  • Matching tags: By default, (match everything) is chosen. If needed, you can attach tags to a particular metric object and select those tags here, which would cause only alerts on that metric object to be triggered.
  • Excluding tags: By default, (exclude nothing) is chosen. If needed, you can attach tags to a particular metric object and select those tags here, which would cause alerts on that metric object to be skipped.
  • Annotate: When enabled, an annotation is automatically created when the alert is triggered. Annotations will be visible in the custom metrics dashboard where the data stream is displayed.
  • Automatic Clear: When enabled, the alert issue is automatically cleared if the triggering condition is no longer true
  • Notify on clear: When enabled, notifications are also sent when the alert issue is cleared. Please do note that notifications are always sent when the alert issue is triggered.
  • Send Notifications To: Here you can configure the notification mechanisms by which the alert is communicated to you and others in your team.

Alerts can also be configured against any of these custom metrics. CopperEgg supports notifying different sets of users with differing notification mechanisms for each alert type.

Notification mechanisms include:

  • Email
  • SMS
  • PagerDuty
  • Twitter
  • HipChat
  • Campfire
  • Slack
  • OpsGenie

Webhooks are also exposed for clients to configure custom notification mechanisms.  More about setting up website probe alerts can be found here.