push

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/CHANGELOG.md

@@ -0,0 +1,11 @@
+## v1.2.0
+- Adding new pacakge-level functions if consumers prefer to use a singleton Reporter. See `DefaultReporter`, `EnableStatsDReporter`, `IncCounter`, `AddToCounter`, `UpdateGauge`, `RecordHistogram`, `RecordTiming`, `NewTimer`, `Flush`.
+
+## v1.1.0
+- Adding new `RecordHistogram` API for recording non-timing scalars.
+
+## v1.0.1
+- Fixing bug with `NewTimer` that would always record a timing of 1 second.
+
+## v1.0.0
+Initial Release

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/README.md

@@ -0,0 +1,88 @@
+author: alexanderh@qualtrics.com
+
+Sending Metrics with `golang/metrics`
+===
+
+This guide will walk you through how to configure a Go client to send metrics to the [metrics service](https://odo.corp.qualtrics.com/wiki/index.php/Metrics_As_A_Service).
+
+If you are looking for how to record runtime metrics about your Go application check out the [_____ guide found _____](TODO insert link).
+
+Click here for the [full GoDoc for the golang/metrics package](http://godoc-app.eng.qops.net/gitlab-app.eng.qops.net/golang/metrics).
+
+## Configuring hiera
+
+Each production host has a telegraf agent that accepts statsD UDP packets to be forwarded on to the metrics service. In the hiera confiuration for your app you need to add telegraf to the container's host file and inject the local telegraf address as an environment variable. 
+
+```yaml
+docker::apps:
+  app:
+      ...
+      docker_args: >
+        --add-host="telegraf:%{::ipaddress}"
+        -e STATS_REPORTER_ADDRESS="telegraf:8125"
+```
+
+## Using `golang/metrics`
+
+You can use the the package-level `DefaultReporter` or create an instance of a `Reporter` to be passed to you application's types.
+
+### Package-level Reporter
+
+To use the `DefaultReporter` you need to enable it. In your main func, enable the reporter before your app starts. Note that you will use the environment variable that you configured in hiera.
+
+```go
+func main() {
+	if err := metrics.EnableStatsDReporter(
+	    "my_cannonical_service_id",
+	    metrics.AddressFromEnv("STATS_REPORTER_ADDRESS"),
+	); err != nil {
+		fmt.Printf("unable to connect to statsd: %v", err)
+	}
+
+	// Ensure that metrics are flushed on shutdown
+	defer metrics.DefaultReporter.Close()
+}
+
+// Somewhere in the application
+metrics.IncCounter("my_request", metrics.Tag("foo", "bar"))
+```
+
+!!! warning
+    Metrics are buffered before being sent to StatsD. If your application does not close the DefaultReporter before shutdown, some collected metrics may be lost. (In the future, this package's API may be updated to better reflect this.)
+
+Then within your application you can call the other package-level funcs to send metrics.
+
+```go
+metrics.IncCounter("my_request", metrics.Tag("foo", "bar"))
+// or
+defer metrics.NewTimer("http_request", Tag("protocol", "http2")).Record()
+
+```
+
+!!! info
+    Note that when `metrics.EnableStatsDReporter` is used to enable the `DefaultReporter` the `golang/http/accesslog` package will use that `DefaultReporter` to send timing metrics of each request handled. The sent `http_request_time` metric will be tagged with `status_code` and `name` (which is the id/name provided by the consumer with `accesslog.SetRequired(ctx, accesslog.Name, ...)`)
+
+### Reporter Instance
+
+To create an instance of a `Reporter` use the constructor.
+
+```go
+reporter, err := metrics.NewReporter(
+	"my_cannonical_service_id", 
+	metrics.AddressFromEnv("STATS_REPORTER_ADDRESS"),
+)
+if err != nil {
+	fmt.Printf("unable to connect to statsd: %v", err)
+}
+```
+
+Then within your application pass the `reporter` variable to your types and call the methods rather than the package-level funcs.
+
+```go
+reporter.IncCounter("my_request", metrics.Tag("foo", "bar"))
+// or
+defer reporter.NewTimer("http_request", Tag("protocol", "http2")).Record()
+
+```
+
+Now your metrics can be queried in grafana or kairos!

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/client.go

@@ -0,0 +1,14 @@
+package metrics
+
+import "github.com/alexcesaro/statsd"
+
+type client interface {
+	Clone(...statsd.Option) *statsd.Client
+	Count(string, interface{})
+	Increment(string)
+	Gauge(string, interface{})
+	Histogram(string, interface{})
+	Timing(string, interface{})
+	Flush()
+	Close()
+}

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/design.md

@@ -0,0 +1,10 @@
+# Goals
+
+The goals of the metrics package are the following:
+
+1. Provide a `Reporter` interface so we standardize what our shared packages accept to make it easier to use Qualtrics packages. It will also make the transition easier when we extract a pkg from an existing service to make it a shared package.
+2. Provide a client with good defaults to store items in DevOps’ metrics service.
+3. Provide a `DiscardReporter` similar to [`ioutil.Discard`](https://godoc.org/io/ioutil#pkg-variables) for testing and dev environments.
+
+# Implementation
+The current implementation of `StatsDReporter` wraps an existing statsd client. In the future we can remove this dependency by writing our own while keeping the same `Reporter` interface. We also hope to change the `NewReporter` behavior to still return a "retrying" reporter if there is an error establishing a connection during creation.

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/metrics.go

@@ -0,0 +1,109 @@
+package metrics
+
+import (
+	"io"
+	"time"
+)
+
+// DiscardReporter is a Reporter on which all calls succeed without doing anything. This is helpful for tests for dev environments.
+var DiscardReporter = &discardReporter{}
+
+// DefaultReporter is a package level reporter that consumers can use with package level functions.
+// It will default to a DiscardReporter unless the consumer overrides or calls EnableStatsDReporter.
+var DefaultReporter Reporter
+
+func init() {
+	DefaultReporter = DiscardReporter
+}
+
+// The Reporter interface is meant to be an interface that can be used in applications
+// and shared packages so there is a common/consistent interface to facilitate using Qualtrics shared packages
+type Reporter interface {
+	// IncCounter increments the value of a counter metric.
+	IncCounter(name string, tags ...Meta)
+	// AddToCounter increments the value of a counter metric by the specified value.
+	AddToCounter(name string, value int64, tags ...Meta)
+	// UpdateGauge resets the state of a gauge to a new value.
+	UpdateGauge(name string, value int64, tags ...Meta)
+	// RecordHistogram records a histogram metric.
+	RecordHistogram(name string, v float64, tags ...Meta)
+	// RecordTiming records a timing metric.
+	RecordTiming(name string, d time.Duration, tags ...Meta)
+	// NewTimer provides a Timer with a Record method to record a timing metric.
+	NewTimer(name string, tags ...Meta) Timer
+	// Flush should process all metrics stored in an internal buffer.
+	Flush()
+	io.Closer
+}
+
+// Timer is an interface for recording timings.
+type Timer interface {
+	Record()
+}
+
+// Meta is a k/v pair of string data that can be store with a metric.
+type Meta struct {
+	Key   string
+	Value string
+}
+
+// Tag is a helper func when reporting metric data.
+func Tag(key string, value string) Meta {
+	return Meta{
+		Key:   key,
+		Value: value,
+	}
+}
+
+// IncCounter will call IncCounter on the DefaultReporter.
+func IncCounter(name string, tags ...Meta) {
+	DefaultReporter.IncCounter(name, tags...)
+}
+
+// AddToCounter will call AddTzoCounter on the DefaultReporter.
+func AddToCounter(name string, value int64, tags ...Meta) {
+	DefaultReporter.AddToCounter(name, value, tags...)
+}
+
+// UpdateGauge will call UpdateGauge on the DefaultReporter.
+func UpdateGauge(name string, value int64, tags ...Meta) {
+	DefaultReporter.UpdateGauge(name, value, tags...)
+}
+
+// RecordHistogram will call RecordHistogram on the DefaultReporter.
+func RecordHistogram(name string, v float64, tags ...Meta) {
+	DefaultReporter.RecordHistogram(name, v, tags...)
+}
+
+// RecordTiming will call RecordTiming on the DefaultReporter.
+func RecordTiming(name string, d time.Duration, tags ...Meta) {
+	DefaultReporter.RecordTiming(name, d, tags...)
+}
+
+// NewTimer will call NewTimer on the DefaultReporter.
+func NewTimer(name string, tags ...Meta) Timer {
+	return DefaultReporter.NewTimer(name, tags...)
+}
+
+// Flush will call Flush on the DefaultReporter.
+func Flush() {
+	DefaultReporter.Flush()
+}
+
+type discardTimer struct{}
+
+func (t *discardTimer) Record() {}
+
+// A DiscardReporter implements the Reporter interface by providing a null sink.
+type discardReporter struct{}
+
+func (dr discardReporter) IncCounter(name string, tags ...Meta)                    {}
+func (dr discardReporter) AddToCounter(name string, value int64, tags ...Meta)     {}
+func (dr discardReporter) UpdateGauge(name string, value int64, tags ...Meta)      {}
+func (dr discardReporter) RecordHistogram(name string, v float64, tags ...Meta)    {}
+func (dr discardReporter) RecordTiming(name string, d time.Duration, tags ...Meta) {}
+func (dr discardReporter) NewTimer(name string, tags ...Meta) Timer {
+	return &discardTimer{}
+}
+func (dr discardReporter) Flush()       {}
+func (dr discardReporter) Close() error { return nil }

golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/metrics/reporter.go

@@ -0,0 +1,209 @@
+package metrics
+
+import (
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/alexcesaro/statsd"
+	"github.com/pkg/errors"
+)
+
+type config struct {
+	prefix       string
+	addr         string
+	errorHandler func(error)
+}
+
+// Option is a type for configuring a StatsDReporter
+type Option func(*config) error
+
+// Address sets the address where we send metrics
+// The default is "telegraf:8125". This is so "telegraf"
+// can be mounted into docker containers as a network interface
+func Address(addr string) Option {
+	return Option(func(c *config) error {
+		c.addr = addr
+		return nil
+	})
+}
+
+// AddressFromEnv reads the address from the specified env variable.
+// If the specified env variable is not set it will return an error.
+func AddressFromEnv(varName string) Option {
+	value, isSet := os.LookupEnv(varName)
+	if !isSet {
+		return Option(func(c *config) error {
+			return fmt.Errorf("missing variable (%s) for metrics.AddressFromEnv", varName)
+		})
+	}
+	return Option(func(c *config) error {
+		c.addr = value
+		return nil
+	})
+}
+
+// ErrorHandler sets the function called when an error
+// happens when sending metrics to the remote metrics store
+func ErrorHandler(h func(error)) Option {
+	return Option(func(c *config) error {
+		c.errorHandler = h
+		return nil
+	})
+}
+
+// StatsDReporter is a concrete implementation of Reporter that sends metrics to a statsd service.
+// It will buffer and periodically flush metrics.
+type StatsDReporter struct {
+	client       client
+	errorHandler func(error)
+}
+
+// MetricTimer is a concrete implementation of Timer that records a timing.
+type MetricTimer struct {
+	Reporter *StatsDReporter
+	Start    time.Time
+	Name     string
+	Tags     []Meta
+}
+
+// Record internally calls RecordTiming to submit the MetricsTimer.
+func (t *MetricTimer) Record() {
+	t.Reporter.RecordTiming(t.Name, time.Since(t.Start), t.Tags...)
+}
+
+// EnableStatsDReporter will create a new StatsDReporter as the DefaultReporter or return an error.
+func EnableStatsDReporter(prefix string, opts ...Option) error {
+	r, err := NewReporter(prefix, opts...)
+	if err != nil {
+		return err
+	}
+	DefaultReporter = r
+	return nil
+}
+
+// NewReporter creates a new StatsDReporter.
+// If a connection cannot be estabilished with the remote metrics service
+// a DiscardReporter and and error will be returned.
+func NewReporter(prefix string, opts ...Option) (Reporter, error) {
+	cfg := &config{
+		prefix:       prefix,
+		addr:         "telegraf:8125",
+		errorHandler: func(error) {},
+	}
+
+	for _, o := range opts {
+		if err := o(cfg); err != nil {
+			return DiscardReporter, err
+		}
+	}
+
+	client, err := statsd.New(
+		statsd.Prefix(cfg.prefix),
+		statsd.Address(cfg.addr),
+		statsd.TagsFormat(statsd.InfluxDB),
+		statsd.ErrorHandler(cfg.errorHandler),
+	)
+
+	if err != nil {
+		return DiscardReporter, errors.Wrap(err, "error creating client/connection for StatsDReporter")
+	}
+
+	reporter := &StatsDReporter{
+		client:       client,
+		errorHandler: cfg.errorHandler,
+	}
+
+	return reporter, nil
+}
+
+func mapMetasToSliceMetas(tags []Meta) []string {
+
+	var kvs []string
+	for _, t := range tags {
+		kvs = append(kvs, t.Key, t.Value)
+	}
+	return kvs
+}
+
+func (mr *StatsDReporter) getClientWithMetas(tags ...Meta) client {
+	if tagsInSlice := mapMetasToSliceMetas(tags); len(tagsInSlice) > 0 {
+		return mr.client.Clone(
+			statsd.Tags(tagsInSlice...),
+		)
+	}
+	return mr.client
+}
+
+// IncCounter increments a metric count.
+func (mr *StatsDReporter) IncCounter(name string, tags ...Meta) {
+	if name == "" {
+		mr.errorHandler(errors.New("missing name for IncCounter func"))
+		return
+	}
+	mr.getClientWithMetas(tags...).Increment(name)
+}
+
+// AddToCounter adds a given value to a metric count.
+func (mr *StatsDReporter) AddToCounter(name string, value int64, tags ...Meta) {
+	if name == "" {
+		mr.errorHandler(errors.New("missing name for AddToCounter func"))
+		return
+	}
+	mr.getClientWithMetas(tags...).Count(name, value)
+}
+
+// UpdateGauge sets a new value for a metric.
+func (mr *StatsDReporter) UpdateGauge(name string, value int64, tags ...Meta) {
+	if name == "" {
+		mr.errorHandler(errors.New("missing name for UpdateGauge func"))
+		return
+	}
+	mr.getClientWithMetas(tags...).Gauge(name, value)
+}
+
+// RecordHistogram submits a new timing metric.
+func (mr *StatsDReporter) RecordHistogram(name string, v float64, tags ...Meta) {
+	if name == "" {
+		mr.errorHandler(errors.New("missing name for RecordHistogram func"))
+		return
+	}
+
+	mr.getClientWithMetas(tags...).Histogram(name, v)
+}
+
+// RecordTiming submits a new timing metric.
+func (mr *StatsDReporter) RecordTiming(name string, d time.Duration, tags ...Meta) {
+	if name == "" {
+		mr.errorHandler(errors.New("missing name for RecordTiming func"))
+		return
+	}
+
+	mr.getClientWithMetas(tags...).Timing(name, d.Nanoseconds()/1000000)
+}
+
+// NewTimer creates a MetricsTimer that can be recorded later.
+// A common pattern is to create and immediately defer the closure of a Timer:
+//
+//     defer reporter.NewTimer("http_request", Tag("protocol", "http2")).Record()
+func (mr *StatsDReporter) NewTimer(name string, tags ...Meta) Timer {
+	return &MetricTimer{
+		Reporter: mr,
+		Start:    time.Now(),
+		Name:     name,
+		Tags:     tags,
+	}
+}
+
+// Flush flushes all metrics that are currently buffered.
+func (mr *StatsDReporter) Flush() {
+	mr.client.Flush()
+}
+
+// Close will close the StatsDReporter and all open statsd connections.
+func (mr *StatsDReporter) Close() error {
+	mr.client.Close()
+	return nil
+}
+
+var _ Reporter = &StatsDReporter{}