Last updated: 2025-09-10

Checks: 7 0

Knit directory: footprint_clustering/

This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20250530)

The command set.seed(20250530) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: f2089fc

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version f2089fc. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated: 


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/prepare_input_data_K562.Rmd) and HTML (docs/prepare_input_data_K562.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File Version Author Date Message
Rmd f2089fc kevinlkx 2025-09-10 wflow_publish("analysis/prepare_input_data_K562.Rmd")
html 7d0b2e4 kevinlkx 2025-06-03 Build site.
Rmd 715d7d8 kevinlkx 2025-06-03 wflow_publish("analysis/prepare_*")
html ffe5da9 kevinlkx 2025-06-03 Build site.
Rmd 7bd0886 kevinlkx 2025-06-03 wflow_publish("analysis/prepare_input_data_K562.Rmd")

Install TOP R package

# install.packages("devtools")
devtools::install_github("HarteminkLab/TOP")

Input data

Here, we show an example procedure with several steps for preparing input data.

Load R packages

library(TOP)
library(data.table)

Step 1: Find TF motif matches using FIMO software

To scan for TF motif matches,
we use the FIMO software from the MEME suite.

Download hg38 reference genome FASTA file and save it as hg38.fa.

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/ref_genome
cd /project2/xinhe/kevinluo/footprint_clustering/data/ref_genome
wget https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/analysisSet/hg38.analysisSet.fa.gz
gunzip -c hg38.analysisSet.fa.gz > hg38.fa

Generate the chrom.sizes file which will be needed later.

index_fa('/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.fa', chromsize_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes')

Download the motif files (in MEME format) from JASPAR.

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/motifs/
cd /project2/xinhe/kevinluo/footprint_clustering/data/motifs/

wget https://jaspar.elixir.no/download/data/2024/CORE/JASPAR2024_CORE_non-redundant_pfms_meme.zip
unzip JASPAR2024_CORE_non-redundant_pfms_meme.zip -d JASPAR2024_CORE_non-redundant_pfms_meme

Run FIMO

Step 2: Get candidate TF binding sites

We take motif matches obtained from FIMO as candidate binding sites, and add 100 bp flanking regions on both sides of the motifs, then filter candidate sites by FIMO p-value and PWM score, and filter the candidate sites falling in ENCODE blacklist regions.

Download ENCODE blacklist from ENCODE portal and save as blacklist.hg38.bed.gz.

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/ENCODE_blacklist
cd /project2/xinhe/kevinluo/footprint_clustering/data/ENCODE_blacklist
wget https://www.encodeproject.org/files/ENCFF356LFX/@@download/ENCFF356LFX.bed.gz
mv ENCFF356LFX.bed.gz blacklist.hg38.bed.gz

Obtain the candidate sites

Using the following script to run step 1 and step 2 for different TF motifs:

Rscript ~/projects/footprint_clustering/code/get_motif_sites.R \
  --tf="CTCF" --motif="MA0139.2" \
  --threshP=1e-5 --threshPWM=10 --flank=100 \
  --outdir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/'

Rscript ~/projects/footprint_clustering/code/get_motif_sites.R \
  --tf="REST" --motif="MA0138.3" \
  --threshP=1e-5 --threshPWM=10 --flank=100 \
  --outdir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/'

Step 3: Count DNase-seq and ATC-seq genome-wide cleavage

We use DNase-seq reads from K562 cell line (ENCODE ID: ENCSR000EMT).

We first sort and index the BAM file, and obtain the total number of mapped reads from the idxstats file, which will be used later when normalizing read counts by library sizes.

module load samtools

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/DNaseseq/K562
cd /project2/xinhe/kevinluo/footprint_clustering/data/DNaseseq/K562

# Download the BAM file from ENCODE
wget https://www.encodeproject.org/files/ENCFF257HEE/@@download/ENCFF257HEE.bam

# Sort the bam file
samtools sort ENCFF257HEE.bam -o DNaseseq_K562_alignments_sorted_hg38.bam

rm ENCFF257HEE.bam
# This BAM file has already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/DNaseseq/K562/DNaseseq_K562_alignments_sorted_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)
count_genome_cuts(bam_file='/project2/xinhe/kevinluo/footprint_clustering/data/DNaseseq/K562/DNaseseq_K562_alignments_sorted_hg38.bam', 
                  chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes', 
                  data_type='DNase',
                  outdir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                  outname='K562.DNase')

We use ATAC-seq reads from K562 cell line (ENCODE ID: ENCSR483RKN) for example.

We first sort and index the BAM file, and obtain the total number of mapped reads from the idxstats file, which will be used later when normalizing read counts by library sizes.

module load samtools

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/ATACseq/K562
cd /project2/xinhe/kevinluo/footprint_clustering/data/ATACseq/K562

# Download the BAM file from ENCODE
wget https://www.encodeproject.org/files/ENCFF512VEZ/@@download/ENCFF512VEZ.bam
wget https://www.encodeproject.org/files/ENCFF987XOV/@@download/ENCFF987XOV.bam

# Rename the bam file
mv ENCFF512VEZ.bam ATACseq_K562_alignments_rep1_ENCFF512VEZ_hg38.bam
mv ENCFF987XOV.bam ATACseq_K562_alignments_rep2_ENCFF987XOV_hg38.bam

samtools merge ATACseq_K562_alignments_merged_hg38.bam ATACseq_K562_alignments_rep1_ENCFF512VEZ_hg38.bam ATACseq_K562_alignments_rep2_ENCFF987XOV_hg38.bam

samtools sort ATACseq_K562_alignments_merged_hg38.bam -o ATACseq_K562_alignments_merged_sorted_hg38.bam

rm ATACseq_K562_alignments_merged_hg38.bam
# This BAM file has already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ATACseq/K562/ATACseq_K562_alignments_merged_sorted_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)
count_genome_cuts(bam_file='/project2/xinhe/kevinluo/footprint_clustering/data/ATACseq/K562/ATACseq_K562_alignments_merged_sorted_hg38.bam', 
                  chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes', 
                  data_type='ATAC',
                  outdir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                  outname='K562.ATAC')

Step 4: Get DNase- or ATAC-seq count matrices around candidate sites, then normalize, bin and transform the counts

Get DNase-seq read counts matrix around candidate sites:

CTCF

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF_MA0139.2_1e-5.candidate.sites.rds')

count_matrix <- get_sites_counts(sites,
                                 genomecount_dir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                                 genomecount_name='K562.DNase')
saveRDS(count_matrix, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF.K562.DNase.counts.mat.rds')

REST

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST_MA0138.3_1e-5.candidate.sites.rds')

count_matrix <- get_sites_counts(sites,
                                 genomecount_dir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                                 genomecount_name='K562.DNase')
saveRDS(count_matrix, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST.K562.DNase.counts.mat.rds')

Get ATAC-seq read counts matrix around candidate sites:

CTCF

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF_MA0139.2_1e-5.candidate.sites.rds')

count_matrix <- get_sites_counts(sites,
                                 genomecount_dir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                                 genomecount_name='K562.ATAC')
saveRDS(count_matrix, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF.K562.ATAC.counts.mat.rds')

REST

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST_MA0138.3_1e-5.candidate.sites.rds')

count_matrix <- get_sites_counts(sites,
                                 genomecount_dir='/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/',
                                 genomecount_name='K562.ATAC')
saveRDS(count_matrix, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST.K562.ATAC.counts.mat.rds')

Prepare CTCF ChIP-seq data

CTCF ChIP-seq from ENCSR000EGM (Stanford)

Download CTCF K562 ChIP-seq BAM files (ENCODE ID: ENCSR000EGM).

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562
cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq BAM files
wget https://www.encodeproject.org/files/ENCFF172KOJ/@@download/ENCFF172KOJ.bam
wget https://www.encodeproject.org/files/ENCFF265ZSP/@@download/ENCFF265ZSP.bam

# Rename the BAM files
mv ENCFF172KOJ.bam CTCF_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam
mv ENCFF265ZSP.bam CTCF_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam

Download CTCF ChIP-seq peaks from

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq peaks
wget https://www.encodeproject.org/files/ENCFF660GHM/@@download/ENCFF660GHM.bed.gz

# Rename the peak files
mv ENCFF660GHM.bed.gz CTCF.K562.ChIPseq.peaks.bed.gz

Sort and index the BAM files and obtain the number of mapped reads.

# The BAM files have already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

Count ChIP-seq reads around candidate sites (merge ChIP-seq replicates), and normalize to the reference ChIP-seq library size (default: 20 million).

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF_MA0139.2_1e-5.candidate.sites.rds')

sites_chip <- count_normalize_chip(sites,
chip_bam_files=c('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam',
'/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam'),
                                   chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes')

Add binary ChIP labels from ChIP-seq peaks

sites_chip_labels <- add_chip_peak_labels_to_sites(sites_chip,chip_peak_file='/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF.K562.ChIPseq.peaks.bed.gz')

saveRDS(sites_chip_labels, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF.K562.sites.chip.labels.rds')

CTCF ChIP-seq from ENCSR000DWE (UW)

Download CTCF K562 ChIP-seq BAM files (ENCODE ID: ENCSR000DWE).

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq BAM files
wget https://www.encodeproject.org/files/ENCFF800GVR/@@download/ENCFF800GVR.bam
wget https://www.encodeproject.org/files/ENCFF196QVZ/@@download/ENCFF196QVZ.bam

# Rename the BAM files
mv ENCFF800GVR.bam CTCF_K562_ChIPseq_UW_rep1_ENCFF800GVR_hg38.bam
mv ENCFF196QVZ.bam CTCF_K562_ChIPseq_UW_rep2_ENCFF196QVZ_hg38.bam

Download CTCF ChIP-seq peaks from

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq peaks
wget https://www.encodeproject.org/files/ENCFF736NYC/@@download/ENCFF736NYC.bed.gz

# Rename the peak files
mv ENCFF736NYC.bed.gz CTCF.K562.ChIPseq.UW.peaks.bed.gz

Sort and index the BAM files and obtain the number of mapped reads.

# The BAM files have already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_UW_rep1_ENCFF800GVR_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_UW_rep2_ENCFF196QVZ_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

Count ChIP-seq reads around candidate sites (merge ChIP-seq replicates), and normalize to the reference ChIP-seq library size (default: 20 million).

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF_MA0139.2_1e-5.candidate.sites.rds')

sites_chip <- count_normalize_chip(sites,
                                   chip_bam_files=c('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_UW_rep1_ENCFF800GVR_hg38.bam',
                                                    '/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_UW_rep2_ENCFF196QVZ_hg38.bam'),
                                   chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes',
                                   bedtools_path = "~/softwares/bin/bedtools")

Add binary ChIP labels from ChIP-seq peaks

sites_chip_labels <- add_chip_peak_labels_to_sites(sites_chip,chip_peak_file='/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF.K562.ChIPseq.UW.peaks.bed.gz')

saveRDS(sites_chip_labels, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF.K562.UW.sites.chip.labels.rds')

CTCF ChIP-seq from ENCSR000AKO (Broad)

Download CTCF K562 ChIP-seq BAM files (ENCODE ID: ENCSR000AKO).

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq BAM files
wget https://www.encodeproject.org/files/ENCFF834MPG/@@download/ENCFF834MPG.bam
wget https://www.encodeproject.org/files/ENCFF173QSV/@@download/ENCFF173QSV.bam

# Rename the BAM files
mv ENCFF834MPG.bam CTCF_K562_ChIPseq_Broad_rep1_ENCFF834MPG_hg38.bam
mv ENCFF173QSV.bam CTCF_K562_ChIPseq_Broad_rep2_ENCFF173QSV_hg38.bam

Download CTCF ChIP-seq peaks from

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq peaks
wget https://www.encodeproject.org/files/ENCFF769AUF/@@download/ENCFF769AUF.bed.gz

# Rename the peak files
mv ENCFF769AUF.bed.gz CTCF.K562.ChIPseq.Broad.peaks.bed.gz

Sort and index the BAM files and obtain the number of mapped reads.

# The BAM files have already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_Broad_rep1_ENCFF834MPG_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_Broad_rep2_ENCFF173QSV_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

Count ChIP-seq reads around candidate sites (merge ChIP-seq replicates), and normalize to the reference ChIP-seq library size (default: 20 million).

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF_MA0139.2_1e-5.candidate.sites.rds')

sites_chip <- count_normalize_chip(sites,
                                   chip_bam_files=c('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_Broad_rep1_ENCFF834MPG_hg38.bam',
                                                    '/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF_K562_ChIPseq_Broad_rep2_ENCFF173QSV_hg38.bam'),
                                   chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes',
                                   bedtools_path = "~/softwares/bin/bedtools")

Add binary ChIP labels from ChIP-seq peaks

sites_chip_labels <- add_chip_peak_labels_to_sites(sites_chip,chip_peak_file='/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/CTCF.K562.ChIPseq.Broad.peaks.bed.gz')

saveRDS(sites_chip_labels, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/CTCF.K562.Broad.sites.chip.labels.rds')

Prepare REST ChIP-seq data

Download REST K562 ChIP-seq BAM files (ENCODE ID: ENCSR137ZMQ).

mkdir -p /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562
cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq BAM files
wget https://www.encodeproject.org/files/ENCFF116CTI/@@download/ENCFF116CTI.bam
wget https://www.encodeproject.org/files/ENCFF778MNM/@@download/ENCFF778MNM.bam

# Rename the BAM files
mv ENCFF116CTI.bam REST_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam
mv ENCFF778MNM.bam REST_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam

Download REST ChIP-seq peaks

cd /project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562

# Download the ChIP-seq peaks
wget https://www.encodeproject.org/files/ENCFF761YYL/@@download/ENCFF761YYL.bed.gz

# Rename the peak files
mv ENCFF761YYL.bed.gz REST.K562.ChIPseq.peaks.bed.gz

Sort and index the BAM files and obtain the number of mapped reads.

# The BAM files have already been sorted, so we skip the sorting step. 
sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/REST_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

sort_index_idxstats_bam('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/REST_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam', sort=FALSE, index=TRUE, idxstats=TRUE)

Count ChIP-seq reads around candidate sites (merge ChIP-seq replicates), and normalize to the reference ChIP-seq library size (default: 20 million).

sites <- readRDS('/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST_MA0138.3_1e-5.candidate.sites.rds')

sites_chip <- count_normalize_chip(sites,
chip_bam_files=c('/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/REST_K562_ChIPseq_rep1_ENCFF430XCG_hg38.bam',
'/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/REST_K562_ChIPseq_rep2_ENCFF794BPW_hg38.bam'),
                                   chrom_size_file='/project2/xinhe/kevinluo/footprint_clustering/data/ref_genome/hg38.chrom.sizes')

Add binary ChIP labels from ChIP-seq peaks

sites_chip_labels <- add_chip_peak_labels_to_sites(sites_chip,chip_peak_file='/project2/xinhe/kevinluo/footprint_clustering/data/ChIPseq/K562/REST.K562.ChIPseq.peaks.bed.gz')

saveRDS(sites_chip_labels, '/project2/xinhe/kevinluo/footprint_clustering/processed_data/hg38/REST.K562.sites.chip.labels.rds')

sessionInfo()