Traditionally, after a list of run accessions has been filtered on the NCBI website, the accessions are used to download and extract fastq using the SRA toolkit to enter into the next steps of the workflow. A newer compressed data type, generated from raw submitted data containing SARS-CoV-2 sequence, is also accessible to Galaxy users from SRA in the Cloud.

SRA Aligned Read Format (SARF) provides further output options beyond basic fastq format, for example:

1. contigs created from the raw reads in the run
2. reads aligned back to the contigs
3. reads with placeholder quality scores
4. VCF files can also be downloaded for these records relative to the SARS-CoV-2 RefSeq record

• These formats can speed up workflows such as assembly and variant calling.
• This data format is still referenced by the Run accession and accessed using the SRA toolkit.
• This workshop describes the SARF data objects along with associated searchable metadata, and demonstrates a few ways to enter them into traditional workflows.

Agenda

In this tutorial, we will cover:

1. Introduction
   1. SRA Aligned Read Format
2. Workflow Diagram
3. Finding SRA SARS-CoV-2 Runs of Interest
4. Query SARF Metadata
5. Importing SARFs of Interest
6. Importing VCFs for SARS-Cov-2 Runs
7. Feedback for NCBI
8. Other NCBI Resources
9. Acknowledgements
10. Affiliations

Introduction

The aim of this tutorial is to introduce you to some of SRA’s new SARS-CoV-2 cloud resources and data formats, then show you how to filter for Runs of interest to you and access that data in your format of choice in Galaxy to use in your analysis pipeline.

SRA Aligned Read Format

All data submitted to SRA is scanned with our SARS-CoV-2 Detection Tool which uses a Kmer-based approach to identify Runs with Coronaviridae content. The initial scope of the project is limited to those runs deposited in SRA with at least 100 hits for SARS-CoV-2 via the SARS-CoV-2 Detection Tool, a read length of at least 75, and generated using the Illumina platform.

1. For these Runs, Saute was used to assemble contigs via guided assembly, with the SARS-CoV-2 refseq genomic sequence (NC_045512.2) used as the guide.

2. If contigs were successfully assembled, reads were mapped back to the contigs and coverage calculated. These contigs with the reads mapped back and with quality scores removed (to keep the object size small) are the aligned read format files.

3. The SRA toolkit can be used to dump just the contigs in fasta format, the reads aligned to the contigs in sam format or the raw reads in fastq format with placeholder quality scores.

4. The contigs were also assessed via megablast against the nucleotide blast database and the results made available for search.

5. In addition, to support investigation of viral evolution during the pandemic and after the introduction of vaccines, variants are identified relative to the SARS-CoV-2 RefSeq record for each processed run using BCFTools.

The SRA aligned reads, the VCF files, the results of these analyses (such as BLAST and VIGOR3 annotation), and the associated BioSample and sequencing library metadata are available for free access from cloud providers.

Comment

These data can be dumped in sam format using the sam-dump tool in the SRA Toolkit, but this function doesn’t work within Galaxy yet. We hope to include that functionality in a future update.

Workflow Diagram

Open image in new tab

Finding SRA SARS-CoV-2 Runs of Interest

Metadata for SARS-CoV-2 submissions to the SRA includes submitted sample and library information, BLAST results, descriptive contig statistics, and variation and annotation information. These metadata are updated daily and made available to query in the cloud using Google’s BigQuery or Amazon’s Athena services. However, the raw underlying information is also provided as a group of json files that can be downloaded for free from the Open Data Platform without logging in to the cloud. These json files can be imported to Galaxy and queried there to find Runs of interest.

Comment

Some of these tables include complex data fields (array of values) that don’t have a clean analogue in a classic SQL database or table and these can’t be easily queried in Galaxy currently. If you require access to cloud tables or fields not available in Galaxy we recommend accessing those natively in BigQuery or Athena.

We will import the JSON files into Galaxy to query them directory, however the files are split up for efficient querying in the cloud and updated daily, so we first need to get the most up-to-date list of files so we can import those to Galaxy. We’ll just be using a couple of tables in this training, but the other tables can be imported in the same way, using the index files below.

Comment

These metadata files are updated daily around 5:30pm EST. If you try to access the data around this time but encounter an error, trying again a short while later should resolve the issue. Time Zone Converter

Hands-on: Loading SRA Aligned Read Format (SARF) Object Metadata URLs into Galaxy

This step needs to be repeated at the beginning of an analysis to refresh the metadata to the latest daily version.

1. Go to your Galaxy instance of choice such as one of the UseGalaxy.org.au, UseGalaxy.eu, UseGalaxy.fr, UseGalaxy.org or any other.

2. Create a new history

   Tip: Creating a new history

   To create a new history simply click the new-history icon at the top of the history panel:

   

3. Rename your history, e.g. “NCBI SARF”

   Tip: Renaming a history

   1. Click on galaxy-pencil (Edit) next to the history name (which by default is “Unnamed history”)
   2. Type the new name
   3. Click on Save
   4. To cancel renaming, click the galaxy-undo “Cancel” button

   If you do not have the galaxy-pencil (Edit) next to the history name (which can be the case if you are using an older version of Galaxy) do the following:

   1. Click on Unnamed history (or the current name of the history) (Click to rename history) at the top of your history panel
   2. Type the new name
   3. Press Enter

4. Click the upload icon toward the top left corner.

   By default the familiar simple upload dialog should appear. This dialog has more advanced options as different tabs across the top of this dialog though.

5. Click Rule-based as shown below.

   Open image in new tab
   
   

   1. Copy/paste the URLs into the provided box:

      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/contigs.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/annotated_variations.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/blastn.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/metadata.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/peptides.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/tax_analysis.filelist
      https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/variations.filelist
      

   2. Click Build

   3. From Column menu select Basename of Path of URL
      • “From Column?”: A
   4. From Rules menu select Add / Modify Column Definitions
      • Click Add Definition button and select URL
        • “URL”: A
      • Click Add Definition button and select Name (not name tag!)
        • “Name”: B

   5. Click Apply. You should see a table with two columns, the left being the URL column, and the right being the Name column with just the filename.

      You are now ready to start the upload.

   6. Click the Upload button

With that you should have 7 different files in your history. If you examine the files you’ll see that they each have a list of filenames like the following table, with either today or yesterday’s date in the filename, depending on your timezone offset from NCBI’s offices. [Time zone converter](

2021-05-27.000000000000.json.gz
2021-05-27.000000000001.json.gz
2021-05-27.000000000002.json.gz
2021-05-27.000000000003.json.gz
2021-05-27.000000000004.json.gz
2021-05-27.000000000005.json.gz
2021-05-27.000000000006.json.gz
2021-05-27.000000000007.json.gz

Hands-on: Loading SRA Aligned Read Format (SARF) Contig Metadata into Galaxy

Next we will convert this list of filenames to the HTTP URLs for easy import into Galaxy.

1. Open the Rule-based upload tab again, but this time:
   1. “Upload data as”: Collection(s)
   2. “Load tabular data from”: History Dataset

   3. “Select dataset to load”: contigs.filelist

   4. Click Build to bring up the rule builder.

   5. Make the following changes in the Rule Builder

      • From Column menu select Fixed Value
        • “Value”: https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/contigs/
        • Apply
      • From Column menu select Concatenate Columns
        • “From Column”: B
        • “From Column”: A
        • Apply
      • From Rules menu, select Add / Modify Column Definitions
        • Add Definition, URL, Select Column C
        • Add Definition, List Identifier(s), Column A
        • Apply

   6. Name the collection contigs.json

   7. Click the Upload button.

   Once those download jobs have all turned green in the history list, we’ll concatenate these into a single file.

2. Concatenate datasets ( Galaxy version 0.1.0) tail to head (cat) files with the following parameters:

   • param-collection “Datasets to Concatenate”: contigs.json collection
   • Click Run Tool

3. Rename this history item to contigs.single.json

   Tip: Renaming a dataset

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, change the Name field to contigs.single.json
   • Click the Save button

4. JQ ( Galaxy version 1.0) process JSON files with the following parameters:

   • param-file “JSON Input”: contigs.single.json
   • “jq filter”: [.[]]
   • “Convert output to tabular”: yes
   • Click Run Tool

5. Rename this file contigs.tsv

   Tip: Renaming a dataset

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, change the Name field to contigs.tsv
   • Click the Save button

Comment

The conversion of the json files to tabular format takes some time, this is a good time to go make some tea.

Query SARF Metadata

Now that a table has been generated, we will query the table to find the runs of interest. It is a good idea to save the table for future queries on the same dataset. Rerunning the import steps above without filtration will provide a different set of metadata each day.

Hands-on: Query the SRA Metadata Table using SQLite

Next we’ll query this metadata using the Query tabular tool to get a list of all Runs containing contigs of greater than 20,000 nucleotides and average coverage of at least 100X.

1. Run Query Tabular using sqlite sql ( Galaxy version 3.0.0) with the following parameters:

   Tip: Can't find it?

   If you’re not using the GTN-in-Galaxy view, you can search for ‘sql’ to find it.

   • In “Database Table”:
     • param-repeat “Insert Database Table”
       • param-file “Tabular Dataset for Table”: contigs.tsv
       • In “Table Options”
         • In “Table Options”
           • “specify name for table”: SARS_contigs
           • “Specify Column Names”: name,run,coverage,tax_id,hits,length,md5

   • “SQL Query to generate tabular output”:

      SELECT DISTINCT run
      FROM SARS_contigs
      WHERE length > 20000 AND coverage > 100 AND run like '%SRR%'
      ORDER BY run ASC
      LIMIT 10
     

   • “include query result column headers”: no

   Tip: Save SQlite Database for future queries

   If you plan to do multiple queries on the same SQL database or want to skip preprocessing the metadata for future work, it may be useful to set

   • param-repeat “Save the sqlite database in your history” to Yes

2. Click Run Tool and rename the output file to Run_list

   Tip: Renaming a dataset

   • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
   • In the central panel, change the Name field to Run_list
   • Click the Save button

   Tip: Need fastq?

   Use Run_list to bring in Fastq files with Submitted Quality Scores (+BQS format)

   Tip: Column Headers for the Other Metadata Tables

   We are not going to bring in the other metadata tables in this tutorial. Here is a list of column headers for contigs and the other tables. You can find full definitions for these columns here:

   https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud-based-examples/

   https://www.ncbi.nlm.nih.gov/sra/docs/aligned-metadata-tables/

   contigs

   name,run,coverage,tax_id,hits,length,md5
   

   annotated_variations

   run,chrom,pos,id,ref,alt,qual,filter,info,format,sample_a,ac,an,bqb,dp,dp4,dp4_1,dp4_2,dp4_3,dp4_4,idv,imf,mq,mq0f,mqb,mqsb,rpb,sgb,vdb,g_gt,g_pl,g_pl_1,g_pl_2,g_dp,g_ad,g_ad_1,g_ad_2,protein_position,ref_codon,alt_codon,ref_aa,alt_aa,protein_name,protein_length,variation
   

   blastn

   acc,qacc,staxid,sacc,slen,length,bitscore,score,pident,sskingdom,evalue,ssciname
   

   metadata

   acc,assay_type,center_name,consent,experiment,sample_name,instrument,librarylayout,libraryselection,librarysource,platform,sample_acc,biosample,organism,sra_study,releasedate,bioproject,mbytes,loaddate,avgspotlen,mbases,insertsize,library_name,biosamplemodel_sam,collection_date_sam,geo_loc_name_country_calc,geo_loc_name_country_continent_calc,ena_first_public_run,ena_last_update_run,sample_name_sam,datastore_filetype,datastore_provider,datastore_region,attributes,jattr
   

   peptides

   name,contig,mat_peptide,run,location,gene,product,ref_db,ref_id,sequence
   

   tax_analysis

   run,contig,tax_id,rank,name,total_count,self_count,ilevel,ileft,iright
   

   variations

   run,chrom,pos,id,ref,alt,qual,filter,info
   

Tip: Download Fastq with Quality Scores

If you would like to dump the raw, underlying data in fastq format with the original quality scores, you can stop here and use the Faster Download and Extract Reads in FASTQ ( Galaxy version 2.11.0+galaxy0) tool with the following parameters:

• “input type”: list of SRA accessions, one per line
• param-file “sra accession list”: Run_list

Tip: Importing a list of SRR from Athena or BigQuery

If you opted to conduct your metadata search in the cloud using AWS Athena or GCP BigQuery instead of importing the json file to Galaxy, you can save a list of your Run accessions from that search result and import that file as the Run_list to proceed with the rest of this tutorial.

Importing SARFs of Interest

Now that we have assembled a list of Runs that have contigs we are interested in, we’ll construct the path to the SARFS in the cloud and import those to Galaxy so we can work with them.

Hands-on: Importing SARFs of Interest

1. Upload Data

   1. Open the Rule-based upload tab again, but this time:
      • “Upload data as”: Collection(s)
      • “Load tabular data from”: History Dataset
      • “Select dataset to load”: Run_list

   2. Click Build to bring up the rule builder.

   3. Make the following changes in the Rule Builder

      • From Column menu select Using a Regular Expression
        • Check “Create column from expression replacement”
        • “Regular Expression”: (.*)
        • “Replacement Expression”: https://sra-pub-sars-cov2.s3.amazonaws.com/RAO/\1/\1.realign
        • Apply
      • From Rules menu select Add / Modify Column Definitions
        • Click Add Definition button and select URL
          • “URL”: B
        • Click Add Definition button and select List Identifier
          • “List Identifier”: A
          • Apply

   4. Name the output collection sarf_path before clicking Upload

      Comment

      Please note that there can be some lag in availability of SARF/VCF files in the cloud (particularly for newly submitted data). So it’s possible to get a download error for a file that isn’t yet present in the cloud. In these cases waiting ~24 hours will generally resolve the issue and allow you to access the file.

2. Now we will use the SRA toolkit to retrieve the contigs in fasta format

   Run Download and Extract Reads in FASTA/Q format from NCBI SRA ( Galaxy version 2.11.0+galaxy0) with the following parameters:

   • “Select input type”: SRA Archive in current history
   • param-collection“sra archive”: sarf_path
   • In “Advanced Options”:
     • “Table name within cSRA object”: REFERENCE
   • Click Run Tool

   The resulting dataset includes the contigs generated from these runs with placeholder ? for quality scores

   • Rename this collection to sarf_contigs

   Run Fastq to Fasta converter ( Galaxy version 1.1.5)

   • param-collection“FASTQ file to convert”: sarf_contigs (note: in the video this did not get renamed)
   • Click Run Tool

   The resulting dataset includes the contigs generated from these Runs in fasta format

   Tip: Fastq format option

   If you prefer to dump the raw reads in fastq format with placeholder quality scores, leave the Table name within cSRA object field blank.

Importing VCFs for SARS-Cov-2 Runs

This example starts with the same Run_list generated for importing SARFs.

A Run_list could also be imported after querying metadata in the cloud using Google’s BigQuery or Amazon’s Athena services. Metadata about these runs includes submitted sample and library information, BLAST results, descriptive contig statistics, and variation and annotations. See the tutorial video for a short demo on how to search and download Run_list from the cloud.

Hands-on: Importing VCFs of Interest

1. Upload Data

   1. Open the Rule-based upload tab again, but this time:
      • “Upload data as”: Collection(s)
      • “Load tabular data from”: History Dataset
      • “Select dataset to load”: Run_list

   2. Click Build to bring up the rule builder.

   3. Make the following changes in the Rule Builder

      • From Column menu select Using a Regular Expression
        • Check “Create column from expression replacement”
        • “Regular Expression”: (.*)
        • “Replacement Expression”: https://sra-pub-sars-cov2.s3.amazonaws.com/VCF/\1/\1.vcf
        • Apply
      • From Rules menu select Add / Modify Column Definitions
        • “URL”: B
        • “List Identifier”: A
        • Apply

   4. Name the output collection VCFs before clicking Upload

      Comment

      Please note that there can be some lag in availability of SARF/VCF files in the cloud (particularly for newly submitted data). So it’s possible to get a download error for a file that isn’t yet present in the cloud. In these cases waiting ~24 hours will generally resolve the issue and allow you to access the file.

2. Use VCFs in Another Galaxy Tool. Once you have imported the VCF files, you can use them in your standard pipeline- here we will annotate them with SnpEff.

   Run SnpEff eff: annotate variants for SARS-CoV-2 ( Galaxy version 4.5covid19) with the following parameters:

   • param-collection “Sequence changes (SNPs, MNPs, InDels)”: the VCFs collection we just created

   Comment

   Please note that there are 2 Snepff tools, please choose the one for SARS-CoV-2

Feedback for NCBI

If you enjoyed this tutorial, please consider filling out this feedback link for NCBI: https://nlmenterprise.co1.qualtrics.com/jfe/form/SV_0jQct4IQOgfYYaq

Note: the survey will stay open until July 31, 2021 (it may say it expires June 30, but we will extend the deadline at the end of the month)

(There is another survey link below for Galaxy).

Other NCBI Resources

• Getting started with SRA in the Cloud
• NCBI SARS-CoV-2 Resources
• STAT tool
• SRA Aligned Read Format
• VCF generation

If you have questions or feedback you can email the SRA helpdesk: sra@ncbi.nlm.nih.gov

Acknowledgements

The authors would like to acknowledge the SRA Product Team at NCBI and the Community Outreach staff at Galaxy for their assistance on this tutorial.

Affiliations

Adelaide Rhodes & Jon Trow - Computercraft assigned to NCBI/NLM/NIH

You've Finished the Tutorial

Please also consider filling out the Feedback Form as well!

Key points

• NCBI Publishes datasets in the cloud that you can easily process with Galaxy

• The Rule Based Uploader simplifies processing and downloading large numbers of files

Frequently Asked Questions

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.
Did you use this material as a learner or student? Click the form below to leave feedback.

Citing this Tutorial

1. Jon Trow, Adelaide Rhodes, SRA Aligned Read Format to Speed Up SARS-CoV-2 data Analysis (Galaxy Training Materials). https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.html Online; accessed TODAY
2. Hiltemann, Saskia, Rasche, Helena et al., 2023 Galaxy Training: A Powerful Framework for Teaching! PLOS Computational Biology 10.1371/journal.pcbi.1010752
3. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012

BibTeX

@misc{galaxy-interface-ncbi-sarf,
author = "Jon Trow and Adelaide Rhodes",
	title = "SRA Aligned Read Format to Speed Up SARS-CoV-2 data Analysis (Galaxy Training Materials)",
	year = "",
	month = "",
	day = "",
	url = "\url{https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.html}",
	note = "[Online; accessed TODAY]"
}
@article{Hiltemann_2023,
	doi = {10.1371/journal.pcbi.1010752},
	url = {https://doi.org/10.1371%2Fjournal.pcbi.1010752},
	year = 2023,
	month = {jan},
	publisher = {Public Library of Science ({PLoS})},
	volume = {19},
	number = {1},
	pages = {e1010752},
	author = {Saskia Hiltemann and Helena Rasche and Simon Gladman and Hans-Rudolf Hotz and Delphine Larivi{\`{e}}re and Daniel Blankenberg and Pratik D. Jagtap and Thomas Wollmann and Anthony Bretaudeau and Nadia Gou{\'{e}} and Timothy J. Griffin and Coline Royaux and Yvan Le Bras and Subina Mehta and Anna Syme and Frederik Coppens and Bert Droesbeke and Nicola Soranzo and Wendi Bacon and Fotis Psomopoulos and Crist{\'{o}}bal Gallardo-Alba and John Davis and Melanie Christine Föll and Matthias Fahrner and Maria A. Doyle and Beatriz Serrano-Solano and Anne Claire Fouilloux and Peter van Heusden and Wolfgang Maier and Dave Clements and Florian Heyl and Björn Grüning and B{\'{e}}r{\'{e}}nice Batut and},
	editor = {Francis Ouellette},
	title = {Galaxy Training: A powerful framework for teaching!},
	journal = {PLoS Comput Biol}
}

                   

Galaxy Administrators: Install the missing tools

You can use Ephemeris's shed-tools install command to install the tools used in this tutorial.

shed-tools install [-g GALAXY] [-a API_KEY] -t <(curl https://training.galaxyproject.org/training-material/api/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.json | jq .admin_install_yaml -r)

Alternatively you can copy and paste the following YAML

---
install_tool_dependencies: true
install_repository_dependencies: true
install_resolver_dependencies: true
tools:
- name: text_processing
  owner: bgruening
  revisions: ddf54b12c295
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: fastqtofasta
  owner: devteam
  revisions: 297962e79f39
  tool_panel_section_label: FASTA/FASTQ
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: jq
  owner: iuc
  revisions: 5ff75eb1a893
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: query_tabular
  owner: iuc
  revisions: 33d61c89fb8d
  tool_panel_section_label: Text Manipulation
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: snpeff_sars_cov_2
  owner: iuc
  revisions: 2a3a00c1fa0a
  tool_panel_section_label: Variant Calling
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: sra_tools
  owner: iuc
  revisions: 69ebb7f46e45
  tool_panel_section_label: Get Data
  tool_shed_url: https://toolshed.g2.bx.psu.edu/
- name: sra_tools
  owner: iuc
  revisions: 83c7d564b128
  tool_panel_section_label: Get Data
  tool_shed_url: https://toolshed.g2.bx.psu.edu/

Feedback

2 stars	1

July 2024

• 2 stars: Liked: SARF metadata querying Disliked: File links don't work