Merge pull request #410 from deeptools/develop

Develop

Author: joachimwolff

bin/hicCompartmentsPolarization

@@ -0,0 +1,7 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from hicexplorer.hicCompartmentsPolarization import main
+
+if __name__ == "__main__":
+    main()

bin/hicValidateLocations

@@ -0,0 +1,8 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+from hicexplorer.hicValidateLocations import main
+
+if __name__ == "__main__":
+    main()
+

docs/content/News.rst

@@ -1,6 +1,37 @@
 News and Developments
 =====================
 
+Release 3.1
+-----------
+**9 July 2019**
+
+- KR correction improvements: It is now able to process larger data sets like GM12878 primary+replicate on 10kb resolution.
+- Adding script for validation of loop locations with protein peak locations
+- Adding script hicCompartmentsPolarization: Rearrange the average interaction frequencies using the first PC values to represent the global compartmentalisation signal
+
+
+Release 3.0.2
+-------------
+**28 June 2019**
+
+- Pinning dependencies to:
+
+   - hicmatrix version 9: API changes in version 10
+   - krbalancing version 0.0.4: API changes in version 0.0.5
+   - matplotlib version 3.0: Version 3.1 raises 'Not implemented error' for unknown reasons.
+
+- Set fit_nbinom to version 1.1: Version 1.0 Had deprecated function call of scipy > 1.2.
+- Small documentation fixes and improvements.
+
+
+Release 3.0.1
+-------------
+**5 April 2019**
+
+- Fixes KR balancing correction factors
+- Deactivates log.debug
+
+
 Release 3.0
 -----------
 **3 April 2019**
@@ -13,14 +44,14 @@ Release 3.0
 
 
 Release 2.2.3
----------------------
+-------------
 **22 March 2019**
 
 - This bug fix release patches an issue with cooler files, hicBuildMatrix and the usage of a restriction sequence file instead of fixed bin size.
 
 
 Release 2.2.2
----------------------
+--------------
 **27 February 2019**
 
 - This bug fix release removes reference to hicExport that were forgotten to delete in 2.2. Thanks @BioGeek for this contribution.

docs/content/example_usage.rst

@@ -111,7 +111,7 @@ diagnostic plot as follows:
 
 .. code-block:: bash
 
-   $ hicCorrectMatrix diagnostic_plot -m hic_matrix.h5 -o hic_corrected.h5
+   $ hicCorrectMatrix diagnostic_plot -m hic_matrix.h5 -o hic_corrected.png
 
 
 The plot should look like this:
@@ -235,7 +235,7 @@ The A / B compartments can be plotted with :ref:`hicPlotMatrix`.
 
    $ hicPlotMatrix -m pearson_all.h5 --outFileName pca1.png --perChr --bigwig pca1.bw
 
-//.. figure:: ../images/eigenvector1_lieberman.png
-//    :scale: 90 %
-//    :align: center
+.. figure:: ../images/eigenvector1_lieberman.png
+    :scale: 60 %
+    :align: center
 

docs/content/installation.rst

@@ -8,26 +8,26 @@ Requirements
 -------------
 
 * Python 3.6
-* numpy >= 1.15
-* scipy >= 1.1
-* matplotlib >= 3.0
-* pysam >= 0.14
-* intervaltree >= 2.1
-* biopython >= 1.72
-* pytables >= 3.4
+* numpy >= 1.16
+* scipy >= 1.2
+* matplotlib == 3.0
+* pysam >= 0.15
+* intervaltree >= 3.0
+* biopython >= 1.73
+* pytables >= 3.5
 * pyBigWig >= 0.3
 * future >= 0.17
-* six >= 1.11
+* six >= 1.12
 * jinja2 >= 2.10
-* pandas >= 0.23
-* unidecode >= 1.0
-* hicmatrix = 9
-* pygenometracks >= 2.1
-* psutil >= 5.4.8
-* hic2cool >= 0.5
-* cooler >= 0.8.3
-* krbalancing >= 0.0.3 (Needs the library eigen; openmp is recommended for linux users. No openmp support on macOS.)
-* fit_nbinom >= 1.0
+* pandas >= 0.24
+* unidecode >= 1.1
+* hicmatrix = 10
+* pygenometracks >= 3.0
+* psutil >= 5.6
+* hic2cool >= 0.7
+* cooler >= 0.8.5
+* krbalancing >= 0.0.5 (Needs the library eigen; openmp is recommended for linux users. No openmp support on macOS.)
+* fit_nbinom >= 1.1
 
 
 **Warning:** Python 2.7 support is discontinued. Moreover, the support for pip is discontinued too. 

docs/content/list-of-tools.rst

@@ -0,0 +1,8 @@
+.. _hicCompartmentsPolarization:
+
+hicCompartmentsPolarization
+============================
+
+.. argparse::
+   :ref: hicexplorer.hicCompartmentsPolarization.parse_arguments
+   :prog: hicCompartmentsPolarization

docs/content/tools/hicCompartmentsPolarization.rst

@@ -24,4 +24,21 @@ The iterative correction can be used via:
 
 .. code:: bash
 
-    $ hicCorrectMatrix correct --matrix matrix.cool --correctionMethod ICE --chromosomes chrUextra chr3LHet --iterNum 500  --outFileName corrected_ICE.cool --filterThreshold -1.5 5.0
+    $ hicCorrectMatrix correct --matrix matrix.cool --correctionMethod ICE --chromosomes chrUextra chr3LHet --iterNum 500  --outFileName corrected_ICE.cool --filterThreshold -1.5 5.0
+
+
+HiCExplorer version 3.1 changes the way data is transfered from Python to C++ for the KR correction algorithm. With these changes 
+the following runtime and peak memory usage on Rao 2014 GM12878 primary + replicate data is possible:
+
+- KR on 25kb: 165 GB, 1:08 h 
+- ICE on 25kb: 224 GB, 3:10 h 
+- KR on 10kb: 228 GB, 1:42 h
+- ICE on 10kb: 323 GB, 4:51 h
+
+- KR on 1kb: 454 GB, 16:50 h
+- ICE on 1kb: >600 GB, > 2.5 d (we interrupted the computation and strongly recommend to use KR on this resolution)
+
+For HiCExplorer versions <= 3.0 KR performs as follows:
+
+- KR on 25kb: 159 GB, 57:11 min
+- KR on 10kb: >980 GB, -- (out of memory on 1TB node, we do not have access to a node with more memory on our cluster)

docs/content/tools/hicCorrectMatrix.rst

@@ -52,7 +52,7 @@ The ``zscore_matrix.h5`` file contain a z-score matrix that is useful to quickly
 
     $ hicFindTADs -m myHiCmatrix.h5 \ 
     --outPrefix myHiCmatrix_min10000_max40000_step1500_thres0.01_delta0.01_fdr \
-    --TAD_sep_score_prefix myHiCmatrix_min10000_max40000_step1500_thres0.001_delta0.01_fdr_zscore_matrix.h5
+    --TAD_sep_score_prefix myHiCmatrix_min10000_max40000_step1500_thres0.001_delta0.01_fdr
     --thresholdComparisons 0.01 \
     --delta 0.01 \
     --correctForMultipleTesting fdr \
@@ -180,14 +180,14 @@ The process to identify boundaries is as follows:
  * everything between 2 consecutive boundaries is a TAD
 
 For the computation of the p-values, the distribution of the z-scores at the 'diamond' above the local minimum is compared
-with the distribution of z-scores that are `min_depth` downstream using the Wilcoxon rank-sum test. Simarlty, the
-distribution of z-scores is computed with the z-scores `min_dep` upstream of the local mininum. The smallest of the
+with the distribution of z-scores that are `min_depth` downstream using the Wilcoxon rank-sum test. Similarly, the
+distribution of z-scores is computed with the z-scores `min_depth` upstream of the local minimum. The smallest of the
 two p-values is assigned to the local minimum.
 
 If `min_depth` is not given, this is computed as bin size * 30
 (if the bins are smaller than 1000), as bin size * 10 if the bins are between
 1000 and 20.000 and as bin size * 5 if the bin size is bigger than 20.000.
 
-If `min_depth` is not given, this is computed as bin size * 60
+If `max_depth` is not given, this is computed as bin size * 60
 (if the bins are smaller than 1000), as bin size * 40 if the bins are between
 1000 and 20.000 and as bin size * 10 if the bin size is bigger than 20.000.

docs/content/tools/hicFindTADs.rst

@@ -0,0 +1,29 @@
+.. _hicValidateLocations:
+
+hicValidateLocations
+=====================
+
+.. argparse::
+   :ref: hicexplorer.hicValidateLocations.parse_arguments
+   :prog: hicValidateLocations
+
+
+hicValidateLoops is a tool to compare the detect loops from hicDetectLoops (or from any other software as long as the data format is followed, see below) 
+with known peak protein locations to validate if the computed loops do have the expected anchor points. Loops are usually bound by CTCF or Cohesin, 
+therefore it is important to know if the detect loops have protein peaks at their X and Y position.
+
+.. figure:: ../../images/loops_bonev_cavalli.png
+
+    Loops in Hi-C, graphic from Bonev & Cavalli, Nature Reviews Genetics 2016
+
+
+Data format
+===========
+
+The data format of hicDetectLoops output is:
+
+chr_x start_x end_x chr_y start_y end_y p-value
+
+As protein input narrowPeak or broadPeak files are tested. However, as long as the protein data contains in the first three columns the
+chromosome, start and end it should work too.
+