Follow

Why is my run stuck?

If your run is stuck in same run state for more than 30 minutes and your console output has not updated, it's likely you've encountered an unreachable executor. Most likely, the run is being executed on an undersized machine and attempting to use more than the available RAM.

To confirm, first, compare the size of your project, including all imported files, associated with the run and the machine on which the run is executed.

How to check files size?

  1. Navigate to the Files tab
  2. Scroll to the bottom of the file list

Note: don't forget to add the size of your imported projects.

filesize.png

How to check machine size?

  1. Navigate to the Details page to the left of the runs list
  2. Compare the Hardware Tier to the total file size

sizeofmachine.png

For example, the machine you’re running on may have 3GB RAM, while your files are 10GB in size. In this case, you are using an undersized machine.

Using a smaller machine can cause heavy swap utilization*, which causes the machine to reach a crashed/unreachable state. It's unclear why but these machines are able to accept new runs, but then get stuck without sending a failure message to the end-user.

If this is the case, please reach us at support@dominodatalab.com and make sure to include the url of the stuck run.

*Swap utilization is when a hard disk, also called swap space, is used when the amount of virtual memory (RAM) is full. If the system needs more memory resources, it compensates with swap space. Compared to virtual memory, obtaining files from swap space can be very slow. So, if files in swap memory are frequently addressed, it can get very expensive.

 

Was this article helpful?
0 out of 0 found this helpful

Comments