I am working with a large (complex) Hermitian matrix and I am trying to diagonalize it efficiently using Python/Scipy.

Using the `eigh`

function from `scipy.linalg`

it takes about 3s to generate and diagonalize a roughly 800x800 matrix and compute all the eigenvalues and eigenvectors.

The eigenvalues in my problem are symmetrically distributed around 0 and range from roughly -4 to 4. I only need the eigenvectors corresponding to the negative eigenvalues, though, which turns the range I am looking to calculate into [-4,0).

My matrix is sparse, so it's natural to use the `scipy.sparse`

package and its functions to calculate the eigenvectors via `eigsh`

, since it uses much less memory to store the matrix.

Also I can tell the program to only calculate the negative eigenvalues via `which='SA'`

. The problem with this method is, that it takes now roughly 40s to compute half the eigenvalues/eigenvectors. I know, that the ARPACK algorithm is very inefficient when computing small eigenvalues, but I can't think of any other way to compute all the eigenvectors that I need.

Is there any way, to speed up the calculation? Maybe with using the shift-invert mode? I will have to do many, many diagonalizations and eventually increase the size of the matrix as well, so I am a bit lost at the moment.

I would really appreciate any help!

This question is probably better to ask on http://scicomp.stackexchange.com as it's more of a general math question, rather than specific to Scipy or related to programming.

If you need all eigenvectors, it does not make very much sense to use ARPACK. Since you need N/2 eigenvectors, your memory requirement is at least `N*N/2`

floats; and probably in practice more. Using `eigh`

requires `N*N+3*N`

floats. `eigh`

is then within a factor of 2 from the minimum requirement, so the easiest solution is to stick with it.

If you can process the eigenvectors "on-line" so that you can throw the previous one away before processing the next, there are other approaches; look at the answers to similar questions on scicomp.

Similar Questions

When dealing with sparse matrices, how do I convert Matrix Market format into CRS (Compressed Row Storage)?

I noticed Pandas now has support for Sparse Matrices and Arrays. Currently, I create DataFrame()s like this: return DataFrame(matrix.toarray(), columns=features, index=observations) Is there a way to

Anyone know of any good sparse matrix library? I need it for doing kronecker products and multiplication on large sparse matrices (10,000 x 10,000). Right now we are using R, which handles them prett

(using MATLAB) I have a large coordinate matrix and a large sparse adjacency matrix for which coordinates are connected to each other. I had asked previously on SO, how to efficiently compute these di

I have an RDD as such: byUserHour: org.apache.spark.rdd.RDD[(String, String, Int)] I would like to create a sparse matrix of the data for calculations like median, mean, etc. The RDD contains the row_

I have a fairly large sparse matrix that, I reckon, would occupy 1Gb when loaded into memory. I don't need access to the whole matrix at all times, so some kind of memory mapping would work; it doesn'

I am creating a matrix from a Pandas dataframe as follows: dense_matrix = numpy.array(df.as_matrix(columns = None), dtype=bool).astype(np.int) And then into a sparse matrix with: sparse_matrix = scip

Given a large sparse matrix (say 10k+ by 1M+) I need to find a subset, not necessarily continuous, of the rows and columns that form a dense matrix (all non-zero elements). I want this sub matrix to b

I have a sparse matrix: from scipy import sparse a = sparse.diags([1,4,9],[-1,0,1],shape =(10,10),format =csr) I want to take the square root of each of the elements in the sparse matrix I look up

I have the following matrix which I believe is sparse. I tried converting to dense using the x.dense format but it never worked. Any suggestions as to how to do this?, thanks. mx=[[(0, 2), (1, 1), (2,

I've been wondering about this question for quite a while but cannot find a reference: How does Matlab transpose a sparse matrix so fast, given that it is stored in CSC (compressed sparse column) form

I am trying to do a Matrix-Vector Multiplication on GPU (using Cuda). I loaded the matrix on my C++ code (CPU), and then I perform the matrix-vector multiplication using cuda. I am also using shared m

I have to implement sparse matrix and do some decompositions like Cholesky Decomposition, LU Decomposition, QR Decomposition on it. Actually I found a library called JAMA which is capable of doing th

I'm trying to figure out how to iterate through a scipy sparse matrix by column. I'm trying to compute the sum of each column, then weight the members of that column by that sum. What I want to do is

That is, when I do A\b for a very large, symmetric and sparse A, what algorithm does matlab use?

I want to initialise a sparse matrix (for use with scipy minimum_spanning_tree if that matters) from a list of matrix coordinates and values. That is, I have: coords - Nx2 array of coordinates to be s

I am starting dealing with sparse matrices so I'm not really proficient on this topic. My problem is, I have a simple coo-occurrences matrix from a word list, just a 2-dimensional co-occurrence matrix

I initialize an empty sparse matrix using S = scipy.sparse.lil_matrix((n,n),dtype=int) As expected print S doesn't show anything, since nothing has been assigned. Yet if I test: print S[0,0]==0 I r

I have a 3007 x 1644 dimensional matrix of terms and documents. I am trying to assign weights to frequency of terms in each document so I'm using this log entropy formula http://en.wikipedia.org/wiki/

Given a sparse binary matrix A (csr, coo, whatever) I want to make a plot such that I can see the position (i,j) = white in the figure if A(i,j) = 1, and (i,j) = black if A(i,j) = 0; For a dense numpy

I want to initial a sparse matrix with numpy array. The numpy array contains NaN as zero for my program, the code to initial a sparse matrix as following: a= np.array([[np.NaN,np.NaN,10]]) zero_a= np.

Say that I have a sparse matrix in scipy.sparse format. How can I extract a diagonal other than than the main diagonal? For a numpy array, you can use numpy.diag. Is there a scipy sparse equivalent? F

I am trying to use the very recent capability of the RcppArmadillo package (version 0.3.910.0 with R 3.0.1 and evrerything up to date) for conversion of a sparse matrix from the Matrix package (class

I need to insert the zero elements in any sparse matrix in the Matrix Market format (but already without the headers). The first column is the number of the ROW, the second columns is the number of th

I'm currently working with sparse matrices, and I have to compare the computation time of sparse matrix-matrix multiplication with full matrix-matrix multiplication. The issue is that sparse matrix co

How can a sparse matrix - matrix product be calculated? I know the 'classic' / mathematical way of doing it, but it seems pretty inefficient. Can it be improved? I thought about storing the first matr

I'm populating a sparse matrix in R and have written the update in a for loop but was hoping to get some pointers to make it quicker. Here is some example code: library(Matrix) rowId <- rep(c(101:1

I'm writing a lexer generator as a spare time project, and I'm wondering about how to go about table compression. The tables in question are 2D arrays of short and very sparse. They are always 256 cha

I have never use R ,but now I need import a sparse matrix to do association rule in R My import data is a sparse matrix like this: i j x 1 2 3 1 2 3 5 1 3 3 1

I'm trying to use large 10^5x10^5 sparse matrices but seem to be running up against scipy: n = 10 ** 5 x = scipy.sparse.rand(n, n, .001) gets ValueError: Trying to generate a random sparse matrix su

I want to pass a sparse matrix to a shared library from MATLAB, do some operation there (written in C), and then return it. I can pass a dense matrix and use, pretty easy. But, I have no idea how to p

Let's say I have A = [3 0 2; ... 0 0 1; ... 1 1 0] A = sparse(A); The output of which is below: ans = (1,1) 3 (3,1) 1 (3,2) 1 (1,3) 2 (2,3) 1 Question: is there an easy command to generate the fol

How can you take the log base 10 of every element in a sparse matrix (COO)? >>print type(X) <class 'scipy.sparse.coo.coo_matrix'> I've tried this but it doesn't work: import math X.data =

I have a large 500x53380 sparse matrix and trying to dichotomize it. I have tried using event2dichot under sna package but no success because it requires an adjacency matrix or network object. I al

I am working on an algorithm that uses diagonal and first off-diagonal blocks of a large (will be e06 x e06) block diagonal sparse matrix. Right now I create a dict that stores the blocks in such a wa

I am using MATLAB. I have very large sparse matrices, and I want to perform a logical or bsxfun on each column of this matrix. There is a single for loop where in it is a single operation of logical t

I have to process a large sparse matrix whose size is 6004*17842 (doc*terms). The function find() has been tried to get its rows, cols and values and the result has been save in ascii form. But the te

I need to do matrix operations (mainly multiply and inverse) of a sparse matrix SparseMat in OpenCV. I noticed that you can only iterate and insert values to SparseMat. Is there an external code I can

I have an assignment where Im supposed to finish the implementation on a generic sparse matrix. Im stuck on the addition part. The matrix is only going to support numbers so I had it extend Number hop

Say I have a sparse non-rectangular matrix A: >> A = round(rand(4,5)) A = 0 1 0 1 1 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 I would like to obtain the matrix B where the non-zero entries of A are replace

I'm working with some rather large sparse matrices (from 5000x5000 to 20000x20000) and need to find an efficient way to concatenate matrices in a flexible way in order to construct a stochastic matrix

I got a problem when using octave sparse matrix. max(speye(65536)(:)) will result in a 0x0 variable. However, speye(65535) and speye(65537) works. How that happens? My octave version is 3.2.4 in Fedo

I have a numpy array, X: type(X) >>> <class 'scipy.sparse.csc.csc_matrix'> I am interested in finding the indices of the rows where there are non-zero entries, in the 0th column. I tri

I'm working in Matlab and I have the next problem: I have a B matrix of nx2 elements, which contains indexes for the assignment of a big sparse matrix A (almost 500,000x80,000). For each row of B, the

I have a list of 100k items and each item has a list of indices. I am trying to put this into a boolean sparse matrix for vector multiplication. My code isn't running as fast as I would like, so I am

I am using Boost's uBLAS in a numerical code and have a 'heavy' solver in place: http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?LU_Matrix_Inversion The code works excellently, however,

I need to perform a set of operations on a scipy sparse matrix in a Cython method. To efficiently apply these I need access to lil_matrix representation. The lil (linked-list sparse matrix) data repre

I'm trying to write a matrix to sparse matrix conversion program. But, there is a problem in my scan_matrix function.I couldn't figure out where is the problem.According to Nemiver(debugging program)

I am new to the use of sparse matrices, but now need to utilize one in my work to save space. I understand that the following matrix: 10 0 0 0 -2 0 3 9 0 0 0 3 0 7 8 7 0 0 3 0 8 7 5 0 0 8 0 9 9 13 0 4

I have a numpy array such as: array = [0.2, 0.3, 0.4] (this vector is actually size 300k dense, I'm just illustrating with simple examples) and a sparse symmetric matrix created using Scipy such as f