Follow

How to Upload Files Larger than 550MB

Out of the box, Domino limits the size of a file that can be uploaded through the Web UI to 550MB.

Note: If you are using an on premises or VPC deployment of Domino, this size limit is configurable by your administrator thus, your limit may be different. 

There are two easy ways to upload files larger than 550MB. 

Using the CLI

If your files are on your computer, the easiest method is to use the command line (CLI) client. This will allow you to quickly sync any project that you have on your local machine with the project that you have on Domino. 

If you have not installed the CLI client yet, you can do so by following these instructions

Once the CLI client is installed and you have authenticated (per the installation instructions), navigate to the folder on your local machine that contains the file(s) you want to be uploaded to your Domino project. 

We'll associate your Domino project with this folder so that you can easily sync files to and from it. 

Type domino restore to identify which Domino project to associate with this folder. It will download any project files that you have in Domino for that project.

Finally, type domino sync, which will sync the files in your local directory to the project in Domino, thereby uploading any files over 550MB.

Note: If you are using our cloud environment, the CLI client has a max file size of 5GB. This is a limitation of AWS S3. If your file is larger than 5GB, contact support@dominodatalab.com

Using wget or an interactive session within Domino

If your file is on the web or accessible via a URL, you can have Domino download the file to your project. You can do this through a shell script or an interactive session.

We'll walk through using a shell script. 

Create a new .sh file in your Domino file tab and run it in Domino. See example, below. 

This will import the files downloaded through wget. They will appear in the Files tab. You only need to do this once to bring the files into your project.  

 

Note: if you are working with large data, e.g. 10s of GB, you may see longer "Preparing" times. During the "Preparing" stage of the run, your files are being copied to the executor. Larger amounts of data take longer. For large data projects, Domino natively integrates with Hadoop, Spark and other big data platforms. Please contact support@dominodatalab.com for more information.  

 

Was this article helpful?
0 out of 0 found this helpful

Comments