Diagnostic statistics with dominostats.json

Understanding the performance of your experiments can involve analyzing many outputs and results. It’s often useful to see key metrics at a glance across all your runs, to allow you to quickly identify which experiments are worth investigating further.

Domino’s diagnostic statistics functionality allows you to do just that.

To use this feature, write a file named dominostats.json to the root of your project directory. Use keys in this JSON file to identify the outputs you're interested in, and then add the corresponding values.

Here is some example R code that writes three key/value pairs to dominostats.json:

diagnostics = list("R^2" = 0.99, "p-value" = 0.05, "sse" = 10.49)
library(jsonlite)
fileConn<-file("dominostats.json")
writeLines(toJSON(diagnostics), fileConn)
close(fileConn)

Here is the same data being written to dominostats.json by Python code:

import json
with open('dominostats.json', 'w') as f:
    f.write(json.dumps({"R^2": 0.99, "p-value": 0.05, "sse": 10.49}))

The resulting dominostats.json file from these code examples looks like this:

{
"sse": 10.49,
"R^2": 0.99,
"p-value": 0.05
}

The dominostats.json file is deleted before each run automatically by Domino. Therefore, past dominostats.json files will not pollute new Jobs on your Jobs dashboard. If Domino detects that this file has been written to the project root by a Job, it will parse the values out and show them as columns on the Jobs dashboard. You can see the keys represented as available columns in the dashboard, and each row contains the corresponding value from a Job.

Screen_Shot_2019-07-17_at_11.08.40_AM.png

You can also click Jobs Timeline at the top of the dashboard to expand a line chart of dominostats.json values over time. This chart shows all Jobs displayed in the current dashboard. To filter to a specific set of related Jobs, use tagging to create a separate dashboard view.

Screen_Shot_2019-07-17_at_11.13.31_AM.png

The x-axis ticks on the timeline represent individual Jobs, and the y-axis represents the values for the statistics in those jobs. Hover along the chart to see individual data points as tooltips, and click at a point to open the details for the Job that produced that value. You can also click and drag on the chart to zoom in, and you can click on each stat in the legend at upper right to toggle its line on and off.

Was this article helpful?
1 out of 1 found this helpful