pcp

Performance Co-Pilot

View the Project on GitHub

GSoC 17 - Final Report

This page concludes the last three months of the Google Summer of Code experience with open source organization Performance Co-Pilot(PCP). The experience was amazing as it involved an incredible amount of learning and coding (obviously :P). There was a lot of emphases given to scalable design, QA, code sanity, etc. I would like to thank Nathan Scott and Lukas Berk for for being such great mentors, also to Frank from the PCP community.

Prometheus Metrics importer and exporter

Performance co-pilot is a system performance and analysis framework which mainly involves collecting the metrics and analyzing the metrics. I was working on the collection part of the PCP and in simple terms, my work involved implementing a bridge between PCP and Prometheus for inter-usability of the metrics.

Overall project can be divided into three phases/parts:

PMDA Prometheus

I started with the PMDA as a proof of concept very early (April IIRC) with very basic features and it gave a very good start. By the end of the community bonding period, I was able to get the first version of PMDA.

PR 285: Merged

PR includes the first version of PMDA, config generator and QA. The initial design of the PMDA involved synchronous HTTP fetches from the Prometheus endpoints.

PR 303 and PR 305: Merged

PR includes QA for Prometheus Docker container and injection against PMDA.

PR 323: Merged

Pr includes completely different scalable design for Parallel Http fetches and auto config generation during the runtime of the PMDA. This design was finalized after undergoing multiple design proposal (PR 323: Closed and PR 320: Closed)

Prometheus Exporter

According to the proposal, the exporter was supposed to be a separate tool with the only purpose of exporting the PCP metrics in the Prometheus format. After a good discussion with mentors, it was decided to extend the existing PCP tool called pmwebd (provides restful APIs to access PCP metrics) as it avoids blocking another and redundant effort for implementing similar APIs. I would like to thank Frank Ch. Eigler for helping me out with pmwebd.

PR 323: Merged

PR includes pmwebd with Prometheus exporter, caching the API results in the session and batch fetches.

PR 335: Merged

PR includes QA for pmwebd Prometheus exporter.

PR 336: Merged (PCP-4)

PR included the addition of the labels to the existing pmwebd APIs has been merged into PCP-4.

PCP Python APIs

PCP Python library uses ctypes for binding with core PCP APIs written in C. It was my first ever experience on working with Python and C bindings. Most of the work was extending the Python API to support Labels.

PR 299 and PR 306: Merged

PR included minor changes to the Python APIs to handle duplicates

PR 318: Merged

PR included the addition of Python APIs to support the Labels.

Current Work (Extension)

After getting most of the proposed work done, Nathan give me an opportunity to work on the extension of Labels API. (Thanks a lot, Nathan :-))

Currently Labels are not supported by the pcp-archive. So I will be working on getting the Labels support for archives which involve working with the core libpcp and few other tools. I am hoping to complete the same by the end of the GSoC (end of August).


Looking forward to working with PCP on many upcoming challenges.

Again, I would like to express my gratitude to the Performance Co-Pilot Organization for mentoring me and Google for giving me this opportunity.

Cheers!