Add metric for outdated libraries (#957)

Add metrics that count how many running processes are linking to deleted
libraries on each machine. Deleted libraries are usually outdated
libraries, and outdated libraries may have known security
vulnerabilities.

The rationale behind storing these as metrics is allow the rollout of
security fixes to be tracked across a fleet of machines, ensuring that
all affected processes are restarted (e.g. via a reboot).

I'm parsing the output from `/proc/*/maps` because it's using `lsof -d
DEL` can be too slow, particularly if you have sockets that bind to
thousands of IP addresses.

The metric labels include the library path and the base filename, which
allows us to pinpoint the exact path of the deleted library but also
allows us to aggregate on the library name (or approximations of it)
even if library locations differ between operating system versions.

The metrics output and the CPU time consumed is as follows:

    user@host:~$ time sudo python processes.py
    # HELP node_processes_linking_deleted_libraries Count of running processes that link a deleted library
    # TYPE node_processes_linking_deleted_libraries gauge
    node_processes_linking_deleted_libraries{library_path="locale-archive", library_name="/usr/lib/locale"} 3
    node_processes_linking_deleted_libraries{library_path="libevent-2.0.so.5.1.9", library_name="/usr/lib/x86_64-linux-gnu"} 4

    real        0m0.071s
    user        0m0.030s
    sys 0m0.041s

Including the library filename and path will result in reasonably high
metrics cardinality, however I think the benefits when an urgent
security patch is being deployed outweigh concerns around cardinality.

This script assumes that library files do not contain spaces in their
path.

Signed-off-by: Matt Bostock <mbostock@cloudflare.com>

1 files changed, 70 insertions, 0 deletions

diff --git a/text_collector_examples/deleted_libraries.py b/text_collector_examples/deleted_libraries.py
new file mode 100755
index 0000000..b6d5809
--- /dev/null
+++ b/text_collector_examples/deleted_libraries.py
@@ -0,0 +1,70 @@
+#!/usr/bin/env python
+"""
+Script to count the number of deleted libraries that are linked by running
+processes and expose a summary as Prometheus metrics.
+
+The aim is to discover processes that are still using libraries that have since
+been updated, perhaps due security vulnerabilities.
+"""
+
+import errno
+import glob
+import os
+import sys
+
+
+def main():
+    processes_linking_deleted_libraries = {}
+
+    for path in glob.glob('/proc/*/maps'):
+        try:
+            with open(path, 'rb') as file:
+                for line in file:
+                    part = line.strip().split()
+
+                    if len(part) == 7:
+                        library = part[5]
+                        comment = part[6]
+
+                        if '/lib/' in library and '(deleted)' in comment:
+                            if path not in processes_linking_deleted_libraries:
+                                processes_linking_deleted_libraries[path] = {}
+
+                                if library in processes_linking_deleted_libraries[path]:
+                                    processes_linking_deleted_libraries[path][library] += 1
+                                else:
+                                    processes_linking_deleted_libraries[path][library] = 1
+        except EnvironmentError as e:
+            # Ignore non-existent files, since the files may have changed since
+            # we globbed.
+            if e.errno != errno.ENOENT:
+                sys.exit('Failed to open file: {0}'.format(path))
+
+    num_processes_per_library = {}
+
+    for process, library_count in processes_linking_deleted_libraries.iteritems():
+        libraries_seen = set()
+        for library, count in library_count.iteritems():
+            if library in libraries_seen:
+                continue
+
+            libraries_seen.add(library)
+            if library in num_processes_per_library:
+                num_processes_per_library[library] += 1
+            else:
+                num_processes_per_library[library] = 1
+
+    metric_name = 'node_processes_linking_deleted_libraries'
+    description = 'Count of running processes that link a deleted library'
+    print('# HELP {0} {1}'.format(metric_name, description))
+    print('# TYPE {0} gauge'.format(metric_name))
+
+    for library, count in num_processes_per_library.iteritems():
+        dir_path, basename = os.path.split(library)
+        basename = basename.replace('"', '\\"')
+        dir_path = dir_path.replace('"', '\\"')
+        print('{0}{{library_path="{1}", library_name="{2}"}} {3}'.format(metric_name, dir_path, basename, count))
+
+
+if __name__ == "__main__":
+    main()