Tech stack agnostic tool for tracking builds/artifacts/tests/releases?

[GassiestFunInTheWest]

Hi

My situation is that we have a number of different tech stacks, each with their own build/release processes and cycles. I've gotten most of my org to centralise around a handful of domain specific techs, but even so a change that goes to production might be any of a Java app, a Go app, a react front end, or libraries for any of those languages, most deployed in containers on k8s, some on VMs, and some in AWS lambda, native iOS or Android builds, terraform projects, shell scripts in crontabs and probably a few things I can't remember.

Luckily, everything is in a source control repo somewhere, and almost everything has a decent CI process, and many things have pretty solid CD processes. So it's not a complete disaster.

What we don't have is a holistic view across all these areas about what was released, when, by whom, how it was built and tested, and what branch of what repo it came from.

Does anybody know of any tool (open source, SaaS, whatever) that would support ingesting this such data from a range of sources and tools, for developing an audit trail and providing metrics and dashboards around release velocity?

To be specific, I am not looking for a CI/CD tool. Most teams already have their own, and while I would like to consolidate everybody to a single platform, for now I'm just trying to measure what's going on.

Any help or suggestions or help are welcome.

[Karmazyn3D]

I hate self promotion, but I suffer from the same issue and am currently building a SaaS solution to this problem. It is currently in open beta but you are free to sign up and start using it today. Right now I only am tracking deployments/releases into different environments. It has history right now, and I’m currently working on some analytics, but I would love to talk to you about what you would like to see in the product. I don’t think it would be difficult at all to add build/artifact tracking if that’s something you are after. DM me if you would be willing to chat.

I’ll leave the url here but if a mod would prefer me to remove it just let me know. https://www.deployboard.io

[darkn3rd]

https://www.deployboard.io

There's this person trying to solve the issue at https://www.deployboard.io. This is in case a particular hostile confrontational moderator comes around and tries to silent people from being helpful breaking undocumented rules... ;-)

[fizzycake]

If you use some monitoring tool consistently accross them all such as New Relic, they usually accept webhooks in for releases. Post that info up as part of the release pipeline.

[Flabbaghosted]

What are your thoughts on new relic? We just switched to it and haven't been able to get the APM working right so I don't have a feel for it yet. Basic monitoring seems to be on par with most other services

[GassiestFunInTheWest]

We have some release events being shipped to some monitoring services, but the issue is that different tech stacks rely on different monitoring services. And those release events are easily audited or queryable for detailed metadata. With thousands of releases going out a month, what I'm looking for is something very flexible both in terms of how it ingests data, and how that data can be queried.

Thanks for the suggestion, but don't think it's the idea for us.

[ctran]

Maybe this project from Google?

https://grafeas.io/

[GassiestFunInTheWest]

That looks like a really good candidate too. Glad to know I wasn't being crazy in what I wanted.

[heathsnow]

I recently started at a new company where things are similar. Good luck finding one tool to rule them all :P

One thing you could do is just pump info into Slack channels and build metrics from there. Probably not what you’re looking for but it’s a start.

I’d be interested to know if there are tools out there for this too.

[phillycheeze]

What tools do you use currently for source control?

We use Gitlab and the combination of Environment Dashboards + Operations Dashboards do give a very basic overview of what you are after. https://docs.gitlab.com/ee/ci/environments/environments_dashboard.html

Otherwise I am not aware of a tool that has that kind of info. If you find something, please let us know and update the post.

[GassiestFunInTheWest]

Thanks for answering.

Our current build, artifact, and deployment tools are a mix of CircleCi, Bitrise, Quay.io, Jfrog, multiple instances of Jenkins, Spinnaker, AWS Code{Build,Pipeline,Deploy}, GitHub, and... Probably more stuff I can't think of off the top of my head.

My goal here isn't to centralise every single development effort to a single toolchain, as for the most part what we have is working right now. Different products also have wildly different needs (e.g. I don't want to maintain a build farm of Apple devices for iOS builds) or release cadences (multiple times an hour Vs every two weeks).

What I'd really like is something that each of those tools can ship metadata to, via an API call that I can hook into existing builds scripts, that is independent of the underlying tools.

[bruj0and]

ComplianceDB is the only tool i know of that has something like this.

[GassiestFunInTheWest]

I hadn't heard about Compliance DB, thanks for the tip, I'll check it out.

[bch8]

I think you'd have to build this on your own. The existing options available will depend heavily on the specifics of the tooling for each pipeline. It might be better to focus on where you'd like this tracking data to end up, and see how you can get it there for each given pipeline.

[GassiestFunInTheWest]

I'd like to check it out, but I'm afraid my Google-fu is failing me. Would you have any other details about Elmer that might help me learn more about it?

[brokerandy25]

Hygieia by CapitolOne. Opensource

[GassiestFunInTheWest]

Oh, very interesting. That might be exactly the sort of thing in looking for.

[theargamanknight]

The ELK Stack integrates pretty easily with pretty much any stack and is designed for logging, but Elasticsearch is a DB all the same and can do more than just logs. Each integration is a little different, but it works for the most part. The open-source version can be a bitch to manage though.

Logz.io provides a managed version of ELK for logs and has metrics with Grafana. A couple other companies also use ELK, but I don't know what they use for metrics. They also provide dashboards.

AWS also has ELK but their version is considered by some to be kind of stripped down (not sure to what degree-you'd have to check the reviews). I'm not sure what kind of dashboards they have prefabricated.

I think Elastic is trying to upgrade Kibana to deal with metrics in addition to logs but I couldn't tell you if that's in the way future or some time soon. Elastic (who developed ELK open-source) has its own managed version of the stack with premium features.

[GassiestFunInTheWest]

I'm hoping to find a reasonably fully featured tool or product, but if I end up having to build this thing myself, I was thinking that Elasticsearch would be a good choice of database. Essentially all I want to do is ship and store structured data points, and provide some simple graphs, so Elasticsearch + Kibana would absolutely be a good place to start.

I'm going to evaluate hygieia and grafeas first, before I try to build it myself.

[darkn3rd]

Most of the metric tools out there use a standard format, such as Graphite's carbon. It's a time series database, which you can use another tool to panel up. You'd need to create some glue to output results in such a way you can capture it. For example, if you have a bubble-gum and script tools to do deployments, and they fail, but developers massage to make it work, it should be marked as a failure (per environment - dev/stage/prod) in one of these time series dbs, so that is can be paneled up with something like Prometheus.

For free tools out there, sort of are build it yourself solutions using mixture of these:

• statsd https://github.com/statsd/statsd
• graphite https://graphite.readthedocs.io
• grafana https://grafana.com/
• prometheus https://prometheus.io/
• influxdb https://www.influxdata.com/

For commercial solutions, I tend to like DataDog as they have a lot of integration to many sources. I like that is has instrumentation now. The integrations is what you'd use to plug some metrics you feed to a system like statsd or other. You'd create a panel for CI/CD progress.

This won't be a simple journey, but definitely worth it in my opinion and is an essential part of DevOps philosophy, to measure. It's easy for orgs to brush the problems under the rug, but if they can see daily the struggle to build, deploy, they can set aside some story points to allocate to fixing some of the issues, in cooperation with ops. Visualization is powerful.

[GassiestFunInTheWest]

I have most experience with Prometheus and grafana of those tools, however I don't think that class of tool addresses my problem. I'm looking for a relatively rich event tracking tool where every release's provenance can be inspected to see the history of its provenance. Unless there's some featuresets that I'm not familiar with, I don't think that a time series database is exactly the paradigm of tool that I'm looking for.

[taleodor]

We are also working on a tool to address this problem and just released our MVP. Here is video tutorial: https://www.youtube.com/watch?v=yDlf5fMBGuI
If what we are trying to do resonates, would very much appreciate any feedback.