Analyze Data in Your Own Bucket

If your lab's data lives in its own Amazon S3 bucket, you can analyze it there without uploading it. Connect and mount the bucket once (mounting lets your chats read it), then point any chat at the whole bucket or just a few folders. Nothing is copied into Drylab.

When to use it

• Your sequencing core delivers FASTQ or count files straight into your lab's S3 bucket.
• The dataset is large (many runs, imaging, raw reads), so uploading it would take hours and use your Vault storage.
• Your institution requires the data to stay in your own cloud account.
• You want to run the same analysis on one run first, then on the next run or on the whole bucket.

Prompt examples

Type @y to open the Your Data tab, pick your bucket, then click its chip to choose folders or files. The examples use a bucket called my-lab-bucket.

Start with a few folders:

@y my-lab-bucket Run FastQC on the FASTQ files in the folders I picked and give me one summary table per run, with any samples that fail flagged.

Count matrices from a sequencing run:

@y my-lab-bucket Load the count matrices in the selected folder, check sample-level QC, and run a PCA colored by condition.

Inventory first, then decide:

@y my-lab-bucket List what's in this bucket: file types, total size per top-level folder, and which folders look like raw reads versus processed results. Don't analyze anything yet.

Rerun on another part of the bucket (after clicking the chip in your earlier message and choosing a different folder):

Run the same QC as before on these items and add the results to the same summary table.

Steps

Connect and mount the bucket (once)

1. Open Vault > Data Connector, click New connection and choose Amazon S3.
2. Follow the setup (One-click connect (Recommended) or Manual), enter the bucket name, and choose Read-only access (recommended). See Mount Cloud Storage for each step.
3. Click Mount (or Mount Now right after connecting). The bucket now shows Mounted.

Mention it in a chat

1. In the chatbox, type @y (the letter y and a space). The Your Data tab lists your connected buckets and shows whether each one is Mounted or Not mounted.
2. Pick your mounted bucket. A chip that reads All appears in your message: Drylab can read the entire bucket.
3. To narrow it down, click the chip. The picker "Choose files in …" opens on the bucket. Open folders and tick the folders or files you need; your picks stay ticked as you browse and are listed under Selected.
4. Click Use 2 items (or Use entire bucket). The chip now reads 2 items.
5. Write your request and send it. Drylab reads the files where they are, in your bucket.

Rerun on another part of the bucket

1. Click the bucket chip in the message you already sent. The picker opens with that message's selection.
2. Change the selection, then click Analyze 3 items (the number you picked) or Analyze entire bucket. Drylab adds a new bucket chip with that selection to your chatbox.
3. Type your request and send it. The earlier message stays as it was.

Set up the connection in the web app: Data Connector is in the web app's Vault. In the Desktop App, your project's data is already on your computer; see Choose the Right Files.

Tips

• Start small. Try the analysis on one run folder first, check the results, then rerun on more folders or on the entire bucket.
• Folders are fine. Ticking a folder points Drylab at everything inside it, so you don't have to tick files one by one.
• Rerun soon after sending. The sent chip remembers your selection only until you reload the page. After a reload it shows All and the picker opens with nothing ticked.
• Keep results in Drylab. Drylab reads your bucket in place and saves its results in the chat's folder in your project, not in your bucket. See Save Results to the Same Workspace.

Related

• Mount Cloud Storage
• Data Connector
• Choose files from a dataset or bucket
• Compare with Public Data