Environment management best practices

Overview

This document covers best practices for compute environment management. As a Domino admin, you will have the power and responsibility to curate the environments used by your organization. A proactive approach to environment management can prevent sprawl, avoid repetition in environment creation, and equip users with the tools they need to succeed in Domino.

As an admin, your objective is to find a balance between giving users the freedom to be agile in development, while also maintaing enough control that you don’t end up with duplicate or unnecessary environments. Admins and users are able to create arbitrary number of environments in Domino. You’ll want to manage their creation so that you don’t end up with dozens of global environments and hundreds of user environments, which can make it hard for users to know which environments to use, and hard for you as an admin to maintain.

Don’t let your Domino look like this:

too-many-environments.png

Best practices

Domino recommends admins follow these best practices:

  1. Try to keep as few global environments as possible.

    In our experience, the benefits of focusing on a small number of global environments with broad applications, outweighs the benefit of creating many niche global environments. When a user requests an environment with new capabilities, consider whether you can add their requested features to an existing global environment instead of creating a new one.

  2. Use clear, descriptive names for your environments.

    A common cause of environment duplication is a user who cannot tell that an existing environment meets his or her needs. Clear, descriptive names on all environments will make the whole catalog comprehensible to new users or admins, and makes Domino easier to work with and maintain.

  3. Add comments to the Dockerfile for each environment.

    You can add comments to a Dockerfile like this:

    # This is a comment that could provide a helpful description of the code to follow
    RUN echo 'This is an executed Dockerfile instruction'
    
    # Here's a block that installs Ruby
    RUN \
      apt-get update && \
      apt-get install -y ruby
    

    Do yourself and future colleagues a favor by investing in a well-commented Dockerfile. Each section should a have clear heading and comments to explain its purpose and implementation.

  4. Share responsibility for environment management

    If you have multiple teams or departments doing separate work in Domino, they should be responsible for maintaining their own team-specific environments. Find an advanced user in each team and make him or her a deputy for environment management. This person should be responsible for planning and understanding the environments his or her team needs, and should work with you on implementation. This reduces the workload of the admins, and ensures that environments are designed by someone with context on what users need.

  5. Keep global images up-to-date and comprehensive

    You should strive to have global images that cover the majority of users’ needs. Users should only need to make minor additions to global environments when creating their own user environments, such as installing a specific version of a package. You don’t want a situation where users are re-installing Python or making other major changes, as this will result in a bloated and poorly performing environment.

  6. Avoid time-consuming image pulls by caching global environments on the executor machine image

    You should cache your global environments in your executor template machine image. This ensures that each new executor starts up with the base Docker image for any environment already cached. If users are setting up environments that have base images very different from what is cached on the machine image, it can lead to long pull times when launching executors. Contact Domino Support for help with modifying your machine image.

  7. Clean up old or poorly maintained environments

    Create a culture of tidiness around environment creation and management. Enforce a standard of quality in naming and Dockerfile commenting, and be assertive about pruning unnecessary environments. See the following section of this document for a walkthrough.

How to clean up your catalog of environments

Over time, it’s inevitable that the number of environments in your Domino will grow. It’s valuable to do an occasional review to weed out unused environments, update active ones, and consolidate where possible. Depending on the size of your organization and your use of Domino, this may be a yearly or quarterly task.

Look at current environment usage

As an admin, you can see all environments being used across your deployment in the Environments Overview. The table of environments on this screen can be sorted by the # of Projects column to get a quick understanding of which environments are in common use. You can also enter global in the search box to filter for global environments.

search-global-environments.png

Click the name of an environment to see Dockerfile details, as well as the list of projects and models using that environment. The list will also include the date of the last run for each project and model. Keep an eye out for user environments that make duplicate changes to global base environments, as well as unused or poorly-maintained environments.

Plan changes to global environments

Based on what you learned by reviewing existing environments, you should plan an updated set of global environments that include the tools and features frequently added by users. In same cases, it might be as simple as adding a few packages to an existing global environment. You can also create a new global environment when necessary, but we recommend erring on the side of larger, more consolidated environments. Doing so will make it easier for your users to choose an environment, and it will be easier for you to manage and maintain the collection of environments in your deployment.

Before executing your plan and changing the available global environments, it’s best to inform your users of the impending changes and solicit their feedback. Explain the changes to existing environments, announce the creation of new ones, and provide recommendations for which environment to use for various types of projects.

Sunset deprecated and unused environments

As an admin, you have the power to archive any environment. All old projects will still be able to use an archived environment, but new projects won’t be able to select it. Historical runs will still reference an archived environment, so archiving never breaks reproducibility. Use archiving to encourage adoption of new, up-to-date, and consolidated environments. Environments can be un-archived at any time. Here are a few Mongo queries that are useful during this process:

  • To work with environment data in Mongo, you need its ObjectID. To find the ObjectID of an environment fromits name, run the following query, replacing ` with the name of the target environment.

    db.environments_v2.find({"name" : "<environment-name>"})
    
  • If you want to deprecate an environment and migrate all projects using it to a new one, you can use the following command. You will need the ObjectID of each environment.

    db.projects.update({"overrideV2EnvironmentId":ObjectId("<old-environment-ID>")}, { $set: { "overrideV2EnvironmentId": ObjectId("<new-environment-ID>") } },{multi:true})`
    
  • You can unarchive an environment and make it available again by running the following command. You will need the ObjectID of the environment.

    db.environments_v2.update({ _id: ObjectId("<environment-ID>")},{$set: { "isArchived":false}})
Was this article helpful?
0 out of 0 found this helpful