qmp-testing-suite/golang-producer-consumer/vendor/gitlab-app.eng.qops.net/golang/qmp/README.md

author: breel@qualtrics.com

Qualtrics Messaging Platform Go Client

Installation

go get -u gitlab.app-eng.qops.net/golang/qmp/...

---

Limitations

The Golang QMP library is pure Go (hooray!). Unfortunately, avoiding Kafka's native libraries comes with some limitations.

Vendoring

• The github.com/satori/go.uuid in the library assumes no later than v1.1.0.

QSL

• The Go QSL library does not support fast filtering, partial decoding, or aliases. It does support full decoding and re-encoding into a smaller, compatible schema.
• The Go QSL library produces asynchronously. QSL Messages are not safe to modify until a Producer reaches its callback.
• The schema spec compatibility checking does not play nicely with complex schema declarations. It cannot interpret the compatibility of a union with any schema spec.
• The Qualtrics Schema Registry (maintained by the core QMP team) injects namespaces into its schemas. If your schema isn't processing with the Go library, try fetching the registry version and looking for injected namespaces. Check by visiting a variety of http://qmp-schema-registry.service.consul:8081/subjects/qmp.configuration-value/versions/1.
• You must explicitly list the type of any Avro union. For example, if your schema has {"type": ["null", "custom"]}, then you must set its value with qsl.Union("namespace.custom", value).

QPCL

• You must .Close() all Producers and .Stop() all Consumers or they will leak memory and you're gonna have a bad time.
• All callback executions are synchronous to a Consumer or Producer's message processing. Only one message is processed at a time. If your callback takes time, consider creating a goroutine in the callback.
• Editing a QMessage before a Producer reaches its callback will change the callback. It will not change the published message.

---

Client Usage

Initialize

The client library first needs to be 'registered'. This just tells qmp what version of the library you are using which helps track usage and manage deprecation.

The provided client-name should just be the name of your service.

import qpcl "gitlab-app.eng.qops.net/golang/qmp"
import qsl "gitlab-app.eng.qops.net/golang/qmp/qsl"

// Register the serialization library
err := qsl.RegisterApplication("my-client-name")
if err != nil { /* handle */ }

// Register the producer/consumer library
err := qpcl.RegisterApplication("my-client-name")
if err != nil { /* handle */ }

Consume Messages

qc, err := qpcl.NewConsumer(qpcl.CBulk, "topic-name", "group-id", qReader, numberThreads)
if err != nil { /* handle */ }

qc.Start(func(qMessage qsl.Message){
   /* handler for received messages */
   fmt.Println("Message received with Trace ID", qMessage.GetTraceID())
})

// ...

err = qc.Stop()
if err != nil { /* handle */ }

Produce Messages

qp, err := qpcl.NewProducer(qpcl.PBulk, "topic-name")
if err != nil { /* handle */ }

qp.Put(qMessage, func(err error, qMessage qsl.Message) {
   /* handler for sent messages */
   fmt.Println("Message sent with Trace ID", qMessage.GetTraceID(), "and error", err)
})

// ...

err = qp.Close()
if err != nil { /* handle */ }

---

Message Usage

Initialize

import qsl "gitlab-app.eng.qops.net/golang/qmp/qsl"

// ...

err := qsl.RegisterApplication("my-client-name")
if err != nil { /* handle */ }

Read Messages

qr, err := qsl.NewReader("schema.name.v1")
if err != nil { /* handle */ }

// ...

qm, err := qr.NewMessage(encodedAvroObject)
if err != nil { /* handle */ }

Write Messages

qw, err := qsl.NewWriter("schema.name.v1")
if err != nil { /* handle */ }

// Blank QMessage from writer's schema
msg, err := qw.NewMessage()
if err != nil { /* handle */ }
msg.SetTraceID("Look at me, I'm Mr. Meeseeks")

// QMessage from a QMessage compatible with writer's schema
msg, err = qw.NewFromQMessage(msg)
if err != nil { /* handle */ }

// QMessage from a map of the writer schema's fields and their values
msg, err = qw.NewFromObject(map[string]interface{}{"schema_id":"my.schema.v1"})
if err != nil { /* handle */ }

Message Quick Reference

Find the full specification for QMessages here.

mapped := msg.GetAvroObject()                                  // map[string]interface{} representation of the message
bytes, err := msg.GetSerializedBytes()                         // []byte raw avro value
writerSchemaSpec := msg.GetWriterSchema()                      // string writer schema from schema registry
messageID, err := msg.GetMessageID()                           // string uuid for the message
timestamp, err := msg.GetTimestamp()                           // int64 timestamp of message creation
writerHost, err := msg.GetWriterHost()                         // string often unhelpful
writerApp, err := msg.GetWriterApp()                           // string of the client's string passed to RegisterApplication()
traceID, err := msg.GetTraceID()                               // string uuid of a series of transactions
orgID, err := msg.GetOrganizationID()                          // string of qualtrics or a team

err := SetWriterSchema("schema.id.v1", "{type: something}")    // change the message's serialization schema
err := SetOrganizationID("hello from sp1")                     // change the message's org ID
err := SetWriterHost("az1")                                    // change the message's writer host
err := SetTraceID("follow me down the rabbit hole")            // replace the transaction ID with another string

value, err := msg.Get("key")                                   // interface{} of the payload value named "key"
err := msg.Set("key", qsl.Union("namespace.name", value))      // set the value of the payload at "key"

mapped, err := msg.GetPayload()                                // map[string]interface{} of the payload
err := msg.SetPayload(mapped)                                  // set the payload yourself. Better match your schema!

bytes, err := msg.GetOriginalBytes()                           // []byte if the message was consumed/read in
readerSchemaSpec, err := msg.GetReaderSchema()                 // string of the schema if the message was consumed/read in
err := UseOriginalBytes(true)                                  // Produce the raw avro value if the message was consumed/read in. Use to forward a QMessage.
yesOrNo := msg.UsingOriginalBytes()                            // bool whether the message is forwarding or not if the message was consumed/read in

---

Resources

• GoDoc
• QSL Specification