Data sources and data formats for NGS and bioinformatics.

1. Overview

Bioinformatics projects usually combine several data sources and file types. A typical NGS analysis may start with raw FASTQ files, align them to a reference genome, use a gene annotation file to count reads, compare the results across metadata-defined sample groups, and interpret findings using variant, pathway or disease databases.

2. Main types of bioinformatics data sources

Different resources serve different purposes. Some store raw sequencing reads, some provide reference genomes, some curate variants or diseases, and others provide processed expression or clinical datasets.

3. Raw read repositories

Raw read repositories store sequencing data submitted by researchers and consortia. These repositories are important for reanalysis, methods development, benchmarking and meta-analysis.

4. Reference genomes

A reference genome provides the coordinate system for alignment, variant calling, gene annotation and visualization. The chosen genome assembly must match all downstream files.

5. Genome annotations

Genome annotation files define genes, transcripts, exons, promoters, repeats, regulatory regions or target intervals. They are essential for read counting, variant annotation and genomic-interval analysis.

6. Variant and population databases

Variant databases support filtering and interpretation. Population databases help identify common variants. Clinical and disease databases help interpret relevance, but they should be used with context and version tracking.

7. Clinical and cancer-genomics resources

Cancer and clinical-genomics interpretation often combines variant calls with curated databases, cohort resources and literature. These resources differ in scope, licensing and evidence model.

8. Omics and functional interpretation resources

Processed omics resources can help validate findings, compare studies and interpret gene lists.

9. Metadata and sample sheets

Metadata describes what each file and sample represents. In many projects, metadata quality is as important as sequencing quality.

Recommended sample-sheet columns

• sample_id: stable, unique identifier.
• fastq_1 and fastq_2: file names or paths for paired-end reads.
• group: biological condition or class.
• replicate: biological replicate identifier.
• batch: sequencing run, library-preparation batch or processing batch.
• organism: species or strain.
• reference: genome assembly and annotation version.
• library_type: RNA-seq strandedness, single-cell chemistry, capture kit or panel information.

sample_id	group	batch	fastq_1	fastq_2
S1	control	A	S1_R1.fastq.gz	S1_R2.fastq.gz
S2	control	A	S2_R1.fastq.gz	S2_R2.fastq.gz
S3	treated	B	S3_R1.fastq.gz	S3_R2.fastq.gz
S4	treated	B	S4_R1.fastq.gz	S4_R2.fastq.gz

10. Raw sequence formats: FASTQ and FASTA

FASTQ and FASTA are foundational sequence formats.

@read_001
ACGTACGTACGT
+
FFFFFFFFFFFF

>chr1_example
ACGTACGTACGTACGTACGT

11. Alignment formats: SAM, BAM and CRAM

Alignment files store where sequencing reads map to a reference genome or transcriptome.

12. Variant formats: VCF and BCF

Variant files store genomic differences relative to a reference. They can represent single-nucleotide variants, indels, genotypes, allele depths, filters, annotations and sometimes structural variants.

##fileformat=VCFv4.3
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	S1
chr1	12345	rsExample	A	G	99	PASS	DP=42	GT:DP	0/1:42

13. Annotation and interval formats: GTF, GFF, BED and interval lists

Annotation and interval files describe genomic features and regions.

14. Signal tracks, matrices and tabular formats

Many downstream results are stored as matrices, tracks or tables.

15. Compression, indexing and checksums

Bioinformatics files are often large. Compression and indexing make storage and access more efficient.

md5sum sample_R1.fastq.gz > sample_R1.fastq.gz.md5
sha256sum reference.fa.gz > reference.fa.gz.sha256

md5sum -c sample_R1.fastq.gz.md5

16. Recommended project organization

A consistent folder structure helps prevent confusion between raw data, reference files and derived results.

project_name/
├── data/
│   ├── raw_fastq/
│   ├── external/
│   └── metadata/
├── references/
│   ├── genome/
│   ├── annotation/
│   └── indexes/
├── results/
│   ├── qc/
│   ├── alignments/
│   ├── variants_or_counts/
│   └── figures/
├── scripts/
├── envs/
├── logs/
└── reports/

Recommended rules

• Keep raw files read-only when possible.
• Use clear sample IDs that match the metadata table.
• Record database names, versions, genome assemblies and download dates.
• Separate reference files from project-specific results.
• Keep scripts, logs and environment files with the project.

17. Basic quality checks before analysis

Before running a full workflow, verify that files are present, readable, correctly compressed and consistent with metadata.

18. Example command-line checks

The following examples show common format inspection commands. Adapt file names and paths to your project.

# Show one FASTQ record
zcat sample_R1.fastq.gz | head -n 4

# Count reads in FASTQ
echo $(( $(zcat sample_R1.fastq.gz | wc -l) / 4 ))

# Show FASTA headers
grep '^>' reference.fa | head

# Count FASTA sequences
grep -c '^>' reference.fa

# Alignment summary
samtools flagstat sample.bam

# Header and reference names
samtools view -H sample.bam | head
samtools idxstats sample.bam | head

# Index BAM
samtools index sample.bam

# View VCF header
bcftools view -h variants.vcf.gz | head

# Count variants
bcftools view -H variants.vcf.gz | wc -l

# List samples
bcftools query -l variants.vcf.gz

# Index compressed VCF
bcftools index variants.vcf.gz

# Show sample sheet columns
head -n 1 samples.tsv

# Count samples by group
tail -n +2 samples.tsv | cut -f 2 | sort | uniq -c

# Check duplicate sample IDs
tail -n +2 samples.tsv | cut -f 1 | sort | uniq -d

19. Data format cheat sheet

Useful data-source and format links

These resources are useful starting points for downloading data and checking format specifications.

Frequently asked questions