Upload Files to Server via Command Line
The Seven Bridges CLI Uploader is the recommended tool for uploading data to the 7 Bridges Platform. It is integrated into the Vii Bridges CLI.
The Control Line Uploader is recommended for big scale uploads. If you need to upload smaller scale uploads instead, we recommend using the Web uploader.
The maximum number of files y'all tin can submit for upload is 250,000.
Syntax:
sb [global-parameters] upload <upload-subcommand> [command-parameters] The supported global parameters are:
| Option | Brusk parameter | Description |
|---|---|---|
| | Configuration file to utilize instead of the default one. | |
| | -h | Display help. |
| | Configuration profile to employ from credentials file. Please note that this parameter is applicable but to "offset" subcommand, "resume" and "delete" subcommands volition utilise the same profile as the one used for starting the upload job, while for "status" subcommand this parameter is not applicable. (default "default") | |
| | Run the control with debug information in the output. |
| Subcommand | Description |
|---|---|
| | Start the upload job |
| | Check the upload status |
| | Resume the upload job |
| | Delete an upload job. Delight note that but jobs that are paused can be deleted. To pause an upload job, employ CTRL+C. |
Read below for detailed information and instructions for all of the commands.
This command initializes a new upload job. Each task is identified past its unique proper name which can be assigned manually (using the corresponding command parameter) or automatically. Job proper name can be used to rails the upload condition and resume its execution (in example information technology is paused).
The upload is performed in the foreground, in the current active command shell session. The command shell session needs to remain open during the upload command execution, otherwise the upload will be paused and can be resumed. During the upload providing any other input will not be possible.
You tin perform multiple uploads (i.due east. start multiple upload jobs) on a single automobile. The following are the statuses for an upload job:
- Initializing -preparing the list of items (files and folders) which will be uploaded
- Running- the items are being uploaded.
- Paused - the upload job tin can exist interrupted past terminating active control vanquish or issuing the CTRL+C command
- Completed- this status denotes that all items are processed and the upload job is washed; it doesn't necessarily mean all items have been uploaded successfully. The log file contains more detailed information nigh the upload job, including failed items. The status command provides an overview of the upload task, including the number of uploaded, failed, skipped, and remaining items.
sb [global-parameters] upload start [<input> | --manifest-file <manifest-file>] --destination <path> [command-parameters] | Name | Description |
|---|---|
| | Upload destination, which can be project root or a specific folder inside a Platform project. Destination tin can be specified either by name or ID (e.g. "sbguser/sbgproject" or "sbguser/sbgproject/directory1" or "5dc01c9ae4b01e2d090a700a"). |
| Proper noun | Description |
|---|---|
| | The Item (file or folder) or list of items to exist uploaded. Past default, the source folder structure is preserved on the Platform. The |
| | The manifest file which contains the list of items that volition exist uploaded as well equally the accompanying metadata (merely applies to files). By default, the source folder structure is preserved on the Platform after the upload. The |
| | The upload job name which must be unique. The immune characters are A-Za-z0-9_-. The maximum number of characters is 20. If omitted, the name will be automatically assigned using the post-obit format "DAY_DDMMMYYYY_HHMMSS", for example Thu_27Feb2020_231455). |
| | If a file with the aforementioned name already exists at the upload destination, an underscore followed by a serial number will be added as a prefix. In instance of a binder, the contents of both folders will be merged. In example a file and a folder acquit the aforementioned name, the upload volition exist skipped. The |
| | If a file with the same name already exists at the upload destination, it will be overwritten by the file that is uploaded. In case of a folder, the contents of both folders will be merged. In case a file and a folder bear the same name, the upload will be skipped. The |
| | Tag which volition be prepare for each file once its upload is complete. Please note that tagging is not applicable to folders on the Vii Bridges Platform. It is possible to fix maximum 32 tags, with each tag being maximum 64 characters long. |
| | Preferred size of the upload function in bytes. If omitted, the default value of 64MiB volition be used. If you have an unstable network connection, we recommend using '--chunk-size 8000000'. |
| | Maximum number of parallel file uploads. The allowed range for this value is one to viii. The default value is 8. If you have a low upload speed or low (filesystem) read speed (eg. magnetic discs) or if you desire to limit system resources used by the uploader, nosotros recommend using |
| | Maximum allowed network bandwidth for the process that executes this upload job. Should be specified in kbps (kilobits per 2d). If omitted, the maximum possible bandwidth will be used. |
The following table illustrates the unabridged naming conflict resolution machinery:
Apply the status command to bank check the upload condition. You can bank check the status for all upload jobs by omitting the name parameter.
sb [global-parameters] upload status [<name>] This volition return the following information almost each of the upload jobs:
-
Upload job name
-
Status
-
Processed (percentage completed, bytes uploaded / full bytes)
-
Average upload speed (total bytes uploaded / total time spent in "Running" status)
-
ETA (Estimated Fourth dimension of Inflow - expected remaining time for job completion)
To bank check a specific upload job, you should specify thenameof the upload job. This will render more detailed data about the upload job: -
Upload job name
-
Status
-
Log file path
-
Fourth dimension submitted
-
Upload command (notation: if wildcard is used in upload command, expanded command will be displayed)
-
Full number of submitted files
-
Number of uploaded files (successfully uploaded to the Platform)
-
Number of skipped files
-
Number of failed files
-
Number of remaining files (files in "Queued" and "Uploading" state)
-
Processed (percent completed, bytes uploaded / total bytes)
-
Average upload speed - total bytes uploaded / total time spent in "Running" status
-
ETA (Estimated Time of Arrival - expected remaining time for job completion)
The information about completed jobs volition be kept in job history for at least 1 month after information technology has been completed or paused, unless it'south been deleted (using thesb upload deletepick, encounter below). Subsequently this time, the completed and paused jobs volition exist removed from the list of upload jobs.
| Proper name | Description |
|---|---|
| | The name of the upload job. Utilize this parameter to obtain more detailed information about a specified job. |
Utilise this command to resume a previously paused upload job. The upload volition resume its execution from where it had been paused.
sb [global-parameters] upload resume [<name>] | Proper noun | Description |
|---|---|
| | Specify the name of the paused chore you wish to resume. This parameter is required unless there's only one paused chore. |
Utilize this command to delete a specified upload chore or all upload jobs. Simply jobs that have been completed or paused tin be deleted. To interruption the job, use CTRL+C.
sb [global-parameters] upload delete [<name> | --all ] Please go along in mind that an upload task is kept in the job history for (at least) 1 month after information technology has been completed or paused and is automatically deleted from the list after that time.
| Proper name | Description |
|---|---|
| | Upload job name, required if the -all parameter is not used. The |
| | Deletes all upload jobs which are COMPLETED or PAUSED. The |
This section will list several examples on how to upload files. The destination project that is used in all of the examples is: rfanklin/my-project.
Delight note that tags volition not exist shown in the examples.
The following snippet shows the example of a directory tree on a local calculator (from the local path where sb commands are executed):
├── dir1 ├── dir2 │ ├── dir2-i │ │ └── file4.bam │ └── dir2-two ├── file1.bam ├── file2.bam └── file3.bam Upload a binder and 2 files into the project root and tag the uploaded files.
Command
sb upload showtime dir1/ file1.bam file2.bam --destination rfranklin/my-project --tag upload1 Concluding output
upload chore proper noun: Thu_08Apr2021_182030 Consummate Successfully uploaded 2 of 2 files Result on the Platform
├── dir1 ├── file1.bam └── file2.bam Upload a folder (with underlying folder structure and i file included) and 3 files into the existing folder within the project.
- assigning proper noun to the upload job (for easier status tracking)
- tagging uploaded files
- and specifying a custom setting for the number of parallel file uploads (covering the
low upload speed example).
Command
sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel two Terminal output
upload job name: upload2 COMPLETE Successfully uploaded 4 of 4 files Effect on the Platform
The following volition be the result on the Platform, assuming that the Example 1 (see above) is already executed and that the result remains intact:
├── dir1 │ ├── dir2 │ │ ├── dir2-1 │ │ │ └── file4.bam │ │ └── dir2-ii │ ├── file1.bam │ ├── file2.bam │ └── file3.bam ├── file1.bam └── file2.bam Upload ii folders (with underlying folder structure and 1 file included) and 1 file into the project root, while:
- choosing car-rename equally a method for resolving proper name conflict
- tagging uploaded files
- and specifying a custom setting for the chunk size (roofing the unstable network connectedness case).
Command
sb upload kickoff dir1/ dir2/ file1.bam --destination rfranklin/my-projection --autorename --tag upload3 --clamper-size 8000000 Last output
upload job proper name: Thu_08Apr2021_182550 Complete Successfully uploaded two of 2 files Consequence on the Platform
The post-obit will exist the issue on the Platform, assuming that the 2 previous examples are already executed:
├── _1_file1.bam ├── dir1 │ ├── dir2 │ │ ├── dir2-1 │ │ │ └── file4.bam │ │ └── dir2-2 │ ├── file1.bam │ ├── file2.bam │ └── file3.bam ├── dir2 │ ├── dir2-1 │ │ └── file4.bam │ └── dir2-ii ├── file1.bam └── file2.bam Bank check the status for all upload jobs.
Control
Terminal output
Upload job proper name Status Processed Average speed Estimated time Thu_08Apr2021_182030 COMPLETED 100% (85.00/85.00B) 85.00 Bps Northward/A upload2 COMPLETED 100% (114.00/114.00B) 38.00 Bps Due north/A Thu_08Apr2021_182550 COMPLETED 100% (68.00/68.00B) 68.00 Bps N/A Become the detailed status for an upload task.
Command
Terminal output
Upload job name: upload2 Status: COMPLETED Log file path: /home/nikola/.sevenbridges/sb/logs/sb.log Time submitted: 08.04.2021 18:23 Command: sb upload start dir2 file1.bam file2.bam file3.bam --destination rfranklin/my-project/dir1 --name upload2 --tag upload2 --parallel 2 Full files: 4 Total size: 114.00 B # uploaded: 4 # skipped: 0 # failed: 0 # remaining: 0 % processed: 100% Average upload speed: 38.00 Bps ETA N/A Delete single upload job and cheque the status.
Commands
sb upload delete upload2 sb upload status Terminal output
Upload job name Condition Processed Average speed Estimated time Thu_08Apr2021_182030 COMPLETED 100% (85.00/85.00B) 85.00 Bps Due north/A Thu_08Apr2021_182550 COMPLETED 100% (68.00/68.00B) 68.00 Bps Due north/A Delete all upload jobs and bank check the status.
Commands
sb upload delete --all sb upload status Last output
Upload chore name Condition Processed Average speed Estimated time Updated 11 months ago
Source: https://docs.sevenbridges.com/docs/upload-via-the-command-line
0 Response to "Upload Files to Server via Command Line"
Post a Comment