Skip to Main Content
Back to Resources

Visualizing TeamCity Dependencies with Python

Published: March 23, 2017 (Updated: September 12, 2019)

James Waugh, Accusoft Software Engineer

In many cases at Accusoft, we are tasked with commanding large TeamCity build chains, sometimes interacting with dozens of build configurations: build tests, run them, and build all their dependencies! Being able to visualize these relationships is a benefit to my team that allows them to discuss these relationships and discover potential issues, so I created a Python 2.7 script to do so.

The script is invoked by passing the TeamCity ID whose graph is requested. This ID can be found in the settings of the TeamCity configuration in question.

./tcdependencygraph.py My_Project_ID

Here’s what the output looks like:
TeamCity Dependencies

The gist to accomplishing this is:

  • Get a project’s dependencies as JSON
  • For each of that project’s dependencies, recursively traverse them and construct a graph
  • Render the graph utilizing the Dot language and Graphviz.

The required libraries are requests, pydotplus, and networkx:

pip install requests pydotplus networkx

For all network requests, such as that to the TeamCity API and Google Charts, the requests module is used. First, we start with the TeamCity API. We are only concerned with one endpoint here:

/guestAuth/app/rest/buildTypes/id:/

This will give us the properties of a project. /guestAuth/ is used to simplify authentication. In code, it is utilized like so:

def getProjectJson(projectId):
    url = "{0}/guestAuth/app/rest/buildTypes/id:{1}/".format(server, projectId)
    headers = {"Accept": "application/json"}
    response = requests.get(url, headers=headers)
    status = response.status_code
    if status is not 200:
        raise Exception("Project ID '{0}' cannot be read (status: {1})".format(projectId, status))
    return response.json()

For this script, I preferred to return that data in JSON instead of XML. This is done by setting the “accept” header to “application/json”. In the returned JSON, The important bits are the snapshot-dependencies and artifact-dependencies properties:

  "snapshot-dependencies": {
    "count": 0
  },
  "artifact-dependencies": {
    "count": 4,
    "artifact-dependency": [
        ...
        "source-buildType": {
           ... 
        }
    ],
  }

Each of these array entries represents a dependency in the project. Inside each element, the “source-BuildType” is given.

"source-buildType": {
    "id": "ProjectIDHere",
    "name": "..",
    "projectName": "..",
    "projectId": "..",
    "href": "..",
    "webUrl": ".."
}

This is the holy grail: The link to the next project. This is modeled as a node in the graph, and edges are added corresponding to their relationships though the source-buildType. We can again get the dependency’s name and JSON through our getProjectJson function, and recursively repeat this process until a project has no dependencies.

The script utilizes Python’s NetworkX (NX) library to internally create a graph. Nodes and edges are added to the graph based on the type of dependency encountered: black edges for artifact dependencies, and red for snapshot. Then, PyDotPlus is used to obtain the Dot source code. The NX graph is first converted to a PyDot graph by integration that NX provides, and then to a string.

dotSource = nx.drawing.nx_pydot.to_pydot(graph).to_string()

For large graphs, sometimes changing the rank to LR results in a better image:

    dotGraph = nx.drawing.nx_pydot.to_pydot(graph)
    dotGraph.set('rankdir', 'LR')
    dotSource = dotGraph.to_string()

The Google Charts API is used to render the graph’s image from this source string. In this request, the Dot source code is URL-escaped: most notably changing spaces to ‘+’. When generating large graphs, sometimes you may get a 413 (Request Too large) error. If this happens, the provided Dot source can be rendered using your favorite Graphviz interface.

def renderGraph(chartName, dotSource):
    baseurl = "https://chart.googleapis.com/chart?chl={0}&cht=gv"
    r = requests.get(baseurl.format(dotSource), stream=True)
    status = r.status_code
    if status is not 200:
        raise Exception("Could not generate graph (status: {0})".format(status))
    return r.raw

A final note is that, since the names of projects and their dependencies can be nearly identical except for the last configuration name, they should be cut down to fit better on the nodes of the graph. The method here eliminates all common portions between the “::” of the TeamCity full project name. For this, Python’s difflib function get_matching_blocks is used to compare it to the starting project’s name.

def trimProjectName(topProjectName, destProjectName):
    # Split on "::". Return the number of starting array elements that match in both.
    topSplit = topProjectName.split(' :: ')
    destSplit = destProjectName.split(' :: ')
    sequenceMatcher = SequenceMatcher(None, topSplit, destSplit)
    match = sequenceMatcher.get_matching_blocks()[0]

    # If there are no common elements, return the full name of the project.
    # Otherwise the beginning strings are cut off.
    if match.size is 0:
        return destProjectName

    return " :: ".join(destSplit[ match.b + match.size : ]).encode('ascii')

The common elements are removed, and the result is combined with the configuration’s name.

The final script is as follows:

#!/usr/bin/env python
# Script to create a graph of a TeamCity project's dependencies
import json, sys, re, urllib, shutil
from difflib import SequenceMatcher
import requests
import networkx as nx

#Configuration
server = "http://your.teamcity.server.com"

def renderGraph(chartName, dotSource):
    baseurl = "https://chart.googleapis.com/chart?chl={0}&cht=gv"
    r = requests.get(baseurl.format(dotSource), stream=True)
    status = r.status_code
    if status is not 200:
        raise Exception("Could not generate graph (status: {0})".format(status))
    return r.raw

def generateGraph(projectId):
    G = nx.MultiDiGraph(name=projectId)
    topNode = getProjectJson(projectId)
    generateGraph_impl(G, topNode, topNode)
    return G

def getProjectJson(projectId):
    url = "{0}/guestAuth/app/rest/buildTypes/id:{1}/".format(server, projectId)
    headers = {"Accept": "application/json"}
    response = requests.get(url, headers=headers)
    status = response.status_code
    if status is not 200:
        raise Exception("Project ID '{0}' cannot be read (status: {1})".format(projectId, status))
    return response.json()
    
def generateGraph_impl(outputGraph, topNode, node):
    addDependenciesToGraph(outputGraph, topNode, node, 'artifact-dependencies')
    addDependenciesToGraph(outputGraph, topNode, node, 'snapshot-dependencies') 

def addDependenciesToGraph(outputGraph, topNode, node, dependencyName):
    for dep in getDependencies(node, dependencyName):
        addDependencyToGraph(outputGraph, topNode, node, dep, dependencyName)
        generateGraph_impl(outputGraph, topNode, getProjectJson(dep['id']))     

def getDependencies(nodeJson, dependencyName):
    arrayNameMap = {
        'snapshot-dependencies' : 'snapshot-dependency',
        'artifact-dependencies' : 'artifact-dependency'
    }
    dependencyDict = nodeJson[dependencyName]
    result = [ ]
    if int(dependencyDict['count']) != 0:
        for dependency in dependencyDict[arrayNameMap[dependencyName]]:
            result.append(dependency['source-buildType'])
    return result

def addDependencyToGraph(outputGraph, topNode, source, dependency, dependencyName):
    edgeColors = {
        'artifact-dependencies': 'black',
        'snapshot-dependencies': 'red'
    }
    nodeName, depNodeName = getNodeNames(topNode, source, dependency)

    # Check if this edge has the same color. e.g, snapshot or artifact dependency.
    # We will add another edge if a different dependency exists, or if there is no edge
    edgeColor = edgeColors[dependencyName]
    if not outputGraph.has_edge(nodeName, depNodeName):
        outputGraph.add_node(nodeName)
        outputGraph.add_edge(nodeName, depNodeName, color=edgeColor)
    elif all(attrs['color'] != edgeColor for attrs in outputGraph[nodeName][depNodeName].values()):
        outputGraph.add_edge(nodeName, depNodeName, color=edgeColor)
        
def getNodeNames(topNode, srcNode, depNode):
    topProjectName = topNode['projectName']
    srcProjectName = srcNode['projectName']
    depProjectName = depNode['projectName']
    srcName = srcNode['name']
    depName = depNode['name']

    # If the project names start the same, cut off the common substring to be 
    # easier to read on the graph. This is combined with the name in brackets.
    srcResult  = trimProjectName(topProjectName, srcProjectName).replace(" :: ", ".")
    depResult  = trimProjectName(topProjectName, depProjectName).replace(" :: ", ".")
    srcResult += ' [' + srcName + ']'
    depResult += ' [' + depName + ']'

    return srcResult.strip(), depResult.strip()

def trimProjectName(topProjectName, destProjectName):
    # Split on "::". Return the number of starting array elements that match in both.
    topSplit = topProjectName.split(' :: ')
    destSplit = destProjectName.split(' :: ')
    sequenceMatcher = SequenceMatcher(None, topSplit, destSplit)
    match = sequenceMatcher.get_matching_blocks()[0]

    # If there are no common elements, return the full name of the project.
    # Otherwise the beginning strings are cut off.
    if match.size is 0:
        return destProjectName

    return " :: ".join(destSplit[ match.b + match.size : ]).encode('ascii')

if __name__ == '__main__':
    if(len(sys.argv) < 2):
        print "Usage: tcdependencygraph.py "
        sys.exit(1)
    id = sys.argv[1]
    
    # Generate Dot source
    graph = generateGraph(id)
    dotSource = nx.drawing.nx_pydot.to_pydot(graph).to_string()

    # Write Dot source
    with open(id + ".dot", 'wb') as f:
        f.write(dotSource)

    # Render and write PNG
    with open(id + ".png", 'wb') as f:
        pngImage = renderGraph(id, dotSource)
        shutil.copyfileobj(pngImage, f)

James Waugh is a Software Engineer on the SDK division at Accusoft. He joined the company in 2016 as a Software Engineer in Support. James now contributes to the native products team, and has a BS in Computer Engineering.