Cc analysis - Revision history

Kay: /* Example */

2023-08-11T09:24:37Z

Example

← Older revision		Revision as of 11:24, 11 August 2023
Line 55:		Line 55:
	4 5 0.8432		4 5 0.8432
	</pre>		</pre>
	Please note that the CCs are rounded to 4 valid digits. This introduces a bit of noise.		Please note that the CCs are rounded to 4 valid digits. This introduces a bit of noise. In addition, the solution in this example is not overdetermined; 5 is just the minimum number of objects that can be represented in 2-dimensional space when given 10 unique correlation coefficients.
	<pre>		<pre>
	bash-4.2$ cc_analysis -dim 2 cc.dat solution.dat		bash-4.2$ cc_analysis -dim 2 cc.dat solution.dat

Kay: /* The program */

2023-08-11T09:22:09Z

The program

← Older revision		Revision as of 11:22, 11 August 2023
Line 22:		Line 22:
	# as long as the problem is well-determined, the vectors can be calculated. Unknown CCs between data sets (e.g. in case of crystallographic data sets that don't have common reflections) can be estimated from the dot product of their vectors. Well-determination means: each data set has to be related (directly or indirectly i.e through others) to any other by at least as many CCs as the desired number of dimensions is. A necessary condition for this is that each data set has at least as many relations to others (input lines to cc_analysis involving this data set) as the number of dimensions is. It is of course better if more relations are specified!		# as long as the problem is well-determined, the vectors can be calculated. Unknown CCs between data sets (e.g. in case of crystallographic data sets that don't have common reflections) can be estimated from the dot product of their vectors. Well-determination means: each data set has to be related (directly or indirectly i.e through others) to any other by at least as many CCs as the desired number of dimensions is. A necessary condition for this is that each data set has at least as many relations to others (input lines to cc_analysis involving this data set) as the number of dimensions is. It is of course better if more relations are specified!

	== The program ==		== The Fortran program ==
	<code>cc_analysis</code> calculates the vectors from the pairwise correlation coefficients. The (low) dimension must be specified, and a file with lines specifying the correlation coefficients must be provided.		<code>cc_analysis</code> calculates the vectors from the pairwise correlation coefficients. The (low) dimension must be specified, and a file with lines specifying the correlation coefficients must be provided.

Line 29:		Line 29:
	<input.dat> has lines with items: i j corr [ncorr]		<input.dat> has lines with items: i j corr [ncorr]
	-b option: <input.dat> is a binary file (4 bytes for each item)		-b option: <input.dat> is a binary file (4 bytes for each item)
	-w option: calculate weights from of correlated items (4th item on input line)		-w option: calculate weights from number of correlated items (4th item on input line)
	-z option: use Fisher z-transformation		-z option: use Fisher z-transformation
	-f option: skip some calculations (fast)		-f option: skip some calculations (fast)

Kay: /* Example */

2023-07-25T08:03:36Z

Example

← Older revision		Revision as of 10:03, 25 July 2023
Line 59:		Line 59:
	bash-4.2$ cc_analysis -dim 2 cc.dat solution.dat		bash-4.2$ cc_analysis -dim 2 cc.dat solution.dat
	CC_ANALYSIS version 29.10.2018 (K. Diederichs). No redistribution please!		CC_ANALYSIS version 29.10.2018 (K. Diederichs). No redistribution please!

	compiler: Intel(R) Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 18.0.5.274 Build 20180823 options: -msse4.2 -L/usr/local/src/arpack/ -larpack_ifort -lmkl_lapack95_lp64 -lmkl_blas95_lp64 -mkl -static-intel -qopenmp-link=static -sox -traceback -qopenmp -align array64byte -assume buffered_io -o /home/dikay/bin/cc_analysis

	Linux turn31.biologie.uni-konstanz.de 4.18.14-1.el7.elrepo.x86_64 #1 SMP Sat Oct 13 10:29:59 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux		Linux turn31.biologie.uni-konstanz.de 4.18.14-1.el7.elrepo.x86_64 #1 SMP Sat Oct 13 10:29:59 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Line 165:		Line 163:
	Finished outputting 2-dimensional representative vectors! =)		Finished outputting 2-dimensional representative vectors! =)
	</pre>		</pre>
	The 5 lines at the bottom give the solution. The coordinates agree with those of the Fortran program within a rms deviation of 0.0055~~; however they are mirrored~~ across the x axis ~~and thus represent an~~ inverted solution.		The 5 lines at the bottom give the solution. The coordinates agree with those of the Fortran program within a rms deviation of 0.0055 after mirroring across the x axis (inverted solution) and rotating by half a degree.

Kay: /* The program */ fix link

2023-07-25T07:40:26Z

The program: fix link

← Older revision		Revision as of 09:40, 25 July 2023
Line 38:		Line 38:
	* the number of vectors must be > 2*(low dimension). Typical number of dimensions is 2 or 3, but depending on the problem it could of course be much more.		* the number of vectors must be > 2*(low dimension). Typical number of dimensions is 2 or 3, but depending on the problem it could of course be much more.

	Python code is available [https://~~strucbio.biologie~~.uni-konstanz.de/pub/cc_analysis.py] under GPL.		Python code is available as [https://wiki.uni-konstanz.de/pub/cc_analysis.py] under GPL.

	== Example ==		== Example ==

Kay: /* Example */

2023-01-13T20:19:12Z

Example

← Older revision		Revision as of 22:19, 13 January 2023
Line 41:		Line 41:

	== Example ==		== Example ==
			Suppose we have 5 objects (e.g. images or crystallographic data sets), and measure their correlation coefficients against each other. These must be written to a file (named cc.dat in this example), and given as input to cc_analysis:
	<pre>		<pre>
	bash-4.2$ cat cc.dat		bash-4.2$ cat cc.dat

Kay: improve

2023-01-13T19:11:16Z

improve

← Older revision		Revision as of 21:11, 13 January 2023
Line 16:		Line 16:

	It further turns out that the resulting arrangement of vectors (that minimize the function <math>\Phi(\mathbf{x})</math> given above) has the properties		It further turns out that the resulting arrangement of vectors (that minimize the function <math>\Phi(\mathbf{x})</math> given above) has the properties
	# 0 <= length <= 1 for each vector; short vectors have a low signal-to-noise ratio, long vectors a good signal-to-noise ratio, and vectors of length 1 represent ideal (prototypical) data sets. In fact, the length of each vector is CC* (as defined in [http://dx.doi.org/10.1126/science.1218231 Karplus & Diederichs (2012)]), and its ~~exact~~ relation to the signal-to-noise ratio is given in eqn. 4.		# 0 <= length <= 1 for each vector; short vectors have a low signal-to-noise ratio, long vectors a good signal-to-noise ratio, and vectors of length 1 represent ideal (prototypical) data sets. In fact, the length of each vector is CC* (as defined in [http://dx.doi.org/10.1126/science.1218231 Karplus & Diederichs (2012)]), and its relation to the signal-to-noise ratio is given in eqn. 4.
	# vectors point in the same direction (i.e. they lie on a radial line) if their data sets only differ by random noise		# vectors point in the same direction (i.e. they lie on a radial line) if their data sets only differ by random noise
	# vectors representing systematically different (heterogeneous) data sets enclose an angle whose cosine is the CC of their respective ideal data sets (eqn. 5).		# vectors representing systematically different (heterogeneous) data sets enclose an angle whose cosine is the CC of their respective ideal data sets (eqn. 5).
	# if all CCs are known, the solution is unique in terms of lengths of vectors, and angles between them. However, a rotated (around the origin) or inverted (through the origin) arrangement of the vectors leaves the functional unchanged, because these transformations do not change lengths and angles.		# if all CCs are known, the solution is unique in terms of lengths of vectors, and angles between them. However, a rotated (around the origin) or inverted (through the origin) arrangement of the vectors leaves the functional unchanged, because these transformations do not change lengths and angles.
	# as long as the problem is ~~over~~-determined, the vectors can be calculated. Unknown CCs between data sets (e.g. in case of crystallographic data sets that don't have common reflections) can be estimated from the dot product of their vectors. ~~Over~~-determination means: each data set has to be related (directly or indirectly i.e through others) to any other by at least as many CCs as the desired number of dimensions is.		# as long as the problem is well-determined, the vectors can be calculated. Unknown CCs between data sets (e.g. in case of crystallographic data sets that don't have common reflections) can be estimated from the dot product of their vectors. Well-determination means: each data set has to be related (directly or indirectly i.e through others) to any other by at least as many CCs as the desired number of dimensions is. A necessary condition for this is that each data set has at least as many relations to others (input lines to cc_analysis involving this data set) as the number of dimensions is. It is of course better if more relations are specified!

	== The program ==		== The program ==

Kay: update descri

2022-12-07T08:11:54Z

update descri

← Older revision		Revision as of 10:11, 7 December 2022
Line 3:		Line 3:
	Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.		Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.

	If N is the number of data sets and <math>cc_{ij}</math> denotes the correlation coefficients between data sets i and j, cc_analysis minimizes		How is the low-dimensional representation found? If N is the number of data sets and <math>cc_{ij}</math> denotes the correlation coefficients between data sets i and j, cc_analysis minimizes

	<math>\Phi(\mathbf{x} )=\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}\left(cc_{ij}-x_{i}\cdot x_{j}\right)^{2}</math>		<math>\Phi(\mathbf{x} )=\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}\left(cc_{ij}-x_{i}\cdot x_{j}\right)^{2}</math>

Kay: Improve description

2022-12-07T08:08:21Z

Improve description

← Older revision		Revision as of 10:08, 7 December 2022
Line 9:		Line 9:
	as a function of the vector <math>\bf{x}</math>, the column vector of the N low-dimensional vectors <math>\it{\{{x_{k}\}}}</math>. This can be performed by minimizing from random starting positions, or more elegantly and efficiently by obtaining starting positions through Eigen decomposition after estimating the missing values of the matrix of correlation coefficients.		as a function of the vector <math>\bf{x}</math>, the column vector of the N low-dimensional vectors <math>\it{\{{x_{k}\}}}</math>. This can be performed by minimizing from random starting positions, or more elegantly and efficiently by obtaining starting positions through Eigen decomposition after estimating the missing values of the matrix of correlation coefficients.

	As the resulting vectors x are in low-dimensional space, and the data sets reside in high-dimensional space, the procedure may be considered as ''multidimensional scaling'' ~~- there are other procedures in multidimensional scaling, but this particular one~~ has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. ~~Alternatively, we~~ can think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.		As the resulting vectors <math>\it{\{{x_{k}\}}}</math> are in low-dimensional space, and the data sets reside in high-dimensional space, the procedure may be considered as ''multidimensional scaling''. This procedure has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. We can also think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.

	== Properties of cc_analysis ==		== Properties of cc_analysis ==
Line 15:		Line 15:
	It turns out that the dot product (also called scalar product, or inner product) of low-dimensional vectors (representing the high-dimensional data sets) is very appropriate for approximating the CCs. This is because a CC is itself a dot product (eqn. 3 of [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)).		It turns out that the dot product (also called scalar product, or inner product) of low-dimensional vectors (representing the high-dimensional data sets) is very appropriate for approximating the CCs. This is because a CC is itself a dot product (eqn. 3 of [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)).

	It further turns out that ~~if the dot product is used, then~~ the resulting arrangement of vectors (that minimize ~~a functional defined in eqn. 1~~) has the properties		It further turns out that the resulting arrangement of vectors (that minimize the function <math>\Phi(\mathbf{x})</math> given above) has the properties
	# 0 <= length <= 1 for each vector; short vectors have a low signal-to-noise ratio, long vectors a good signal-to-noise ratio, and vectors of length 1 represent ideal (prototypical) data sets. In fact, the length of each vector is CC* (as defined in [http://dx.doi.org/10.1126/science.1218231 Karplus & Diederichs (2012)]), and its exact relation to the signal-to-noise ratio is given in eqn. 4.		# 0 <= length <= 1 for each vector; short vectors have a low signal-to-noise ratio, long vectors a good signal-to-noise ratio, and vectors of length 1 represent ideal (prototypical) data sets. In fact, the length of each vector is CC* (as defined in [http://dx.doi.org/10.1126/science.1218231 Karplus & Diederichs (2012)]), and its exact relation to the signal-to-noise ratio is given in eqn. 4.
	# vectors point in the same direction (i.e. they lie on a radial line) if their data sets only differ by random noise		# vectors point in the same direction (i.e. they lie on a radial line) if their data sets only differ by random noise

Kay: show and define formula which is minimized

2022-12-07T08:02:42Z

show and define formula which is minimized

← Older revision		Revision as of 10:02, 7 December 2022
Line 3:		Line 3:
	Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.		Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.

	As the ~~result~~ (the vectors) are in low-dimensional space, and the data sets reside in high-dimensional space, the procedure may be considered as ''multidimensional scaling'' - there are other procedures in multidimensional scaling, but this particular one has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. Alternatively, we can think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.		If N is the number of data sets and <math>cc_{ij}</math> denotes the correlation coefficients between data sets i and j, cc_analysis minimizes

			<math>\Phi(\mathbf{x} )=\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}\left(cc_{ij}-x_{i}\cdot x_{j}\right)^{2}</math>

			as a function of the vector <math>\bf{x}</math>, the column vector of the N low-dimensional vectors <math>\it{\{{x_{k}\}}}</math>. This can be performed by minimizing from random starting positions, or more elegantly and efficiently by obtaining starting positions through Eigen decomposition after estimating the missing values of the matrix of correlation coefficients.

			As the resulting vectors x are in low-dimensional space, and the data sets reside in high-dimensional space, the procedure may be considered as ''multidimensional scaling'' - there are other procedures in multidimensional scaling, but this particular one has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. Alternatively, we can think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.

	== Properties of cc_analysis ==		== Properties of cc_analysis ==

Kay at 07:17, 7 December 2022

2022-12-07T07:17:12Z

← Older revision		Revision as of 09:17, 7 December 2022
Line 1:		Line 1:
	The purpose of cc_analysis is to find a representation in low-dimensional space (e.g. 2D, 3D, 4D) that displays the relations of data sets to each other. For this, the procedure needs the values of correlation coefficients (CCs) calculated between the data sets (e.g. crystallographic intensities, or pixels of images). (Not all theoretically possible pairwise CCs are needed, but of course, the more the better.)		The purpose of cc_analysis is to find a representation in low-dimensional space (e.g. 2D, 3D, 4D) that displays the relations of data sets to each other. For this, the procedure needs the values of correlation coefficients (CCs) calculated between the data sets (e.g. crystallographic intensities, or pixels of images). (Not all theoretically possible pairwise CCs are needed, but of course, the more the better.)

	Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.		Since the data sets are composed of many measurements, they could be thought of as residing in a high-dimensional space: in case of crystallography, that dimension is the number of unique reflections; in case of images, the number of pixels.

	As the result (the vectors) are in low-dimensional space, and the data sets ~~are~~ in high-dimensional space, the procedure may be considered as ''multidimensional scaling'' - there are other procedures in multidimensional scaling, but this particular one has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. Alternatively, we can think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.		As the result (the vectors) are in low-dimensional space, and the data sets reside in high-dimensional space, the procedure may be considered as ''multidimensional scaling'' - there are other procedures in multidimensional scaling, but this particular one has first been described in [http://journals.iucr.org/d/issues/2017/04/00/rr5141/index.html Diederichs, Acta D (2017)]. Alternatively, we can think of the procedure as ''unsupervised learning'', because it "learns" from the given CCs, and predicts the unknown CCs - or rather, the relations of even those data sets that have nothing (crystallography: no reflections; imaging: no pixels) in common.

	== Properties of cc_analysis ==		== Properties of cc_analysis ==