I am trying to run the full SVD of a large (120k x 600k) and sparse (0,1% of non-null values) matrix M. Due to memory limitation all my previous attempts failed (with SVDLIBC, Octave, and R) and I am (almost) resigned to exploring other approaches to my problem (LSA).

However, at the moment, I am only interested in the eigenvalues of the diagonal matrix S and not in the left/right singular vectors (matrices U and V).

Is there a way to compute those singular values without storing in memory the dense matrix M and/or the singular vector matrices U and V?

Any help will be greatly appreciated.

[EDIT] My server configuration: 3,5GHz/3,9GHz (6 cores / 12 threads) 128GB of RAM

Looking for the meaning of that values (elements of matrix S from a SVD decomposition) in wikipedia we get:

The non-zero singular values of M (found on the diagonal entries of Σ) are the square roots of the non-zero eigenvalues of both M*M and MM*

So you can look for the eigenvalues of the matrix A*A' (120k x 120k) without explicitly build the matrix, of course.

By the way, I dont think you are interested in ALL the eigenvalues (or singular values) for a matrix with such a dimensions. I do not think that any algorithm will give enough accurate results.

How comfortable are you with Fortran? I think you should be able to complete the computations using prebuilt packages available here and/or here. Also, if you're open to C++ and a decomposition using randomized and re-orthonormalized matrices, you can try the code at the google code project called redsvd. (I can't post the link because I don't have the requisite reputation for three links, but you can search for redsvd and find it readily.)

Similar Questions

It seems to me that in SAGE the only difference between creating a dense matrix and a sparse matrix is by the flag passed to the constructor (sparse = True). In particular, this means that if I want

I have two sparse matrices, m1 and m2: > m1 <- Matrix(data=0,nrow=2, ncol=1, sparse=TRUE, dimnames=list(c(b,d),NULL)) > m2 <- Matrix(data=0,nrow=2, ncol=1, sparse=TRUE, dimnames=list(c

I am attempting to perform fastclust on a very large set of distances, but running into a problem. I have a very large csv file (about 91 million rows so a for loop takes too long in R) of similaritie

I am trying to find the eigenvalues/vectors for the following matrix: 1 0 0 0 1 0 1 1 0 using the code: e_vals, e_vecs = la.eig(A) print eigenvector of A = , e_vals print eigenvalue of A = \n, e_v

I've a Sparse matrix in CSR Sparse format in python and I want to import it to MATLAB. MATLAB does not have a CSR Sparse format. It has only 1 Sparse format for all kind of matrices. Since the matrix

I am trying to cPickle a large scipy sparse matrix for later use. I am getting this error: File tfidf_scikit.py, line 44, in <module> pickle.dump([trainID, trainX, trainY], fout, protocol=-1)

I am working on a project that involves the computation of the eigenvectors of a very large sparse matrix. To be more specific I have a Matrix that is the laplacian of a big graph and I am interested

If I had a square matrix that is 1,000 by 1,000 could Lapack calculate the eigenvectors and eigenvalues for this matrix? And if it can how long would it take? Also what about for a 10,000 by 10,000 ma

I have a 5000 *5000 sparse matrix with 4 different values. I want to visualise the nonzero elements with 4 different colors such that I can recognise the ratio of this values and the relationships bet

I have a large m *n sparse matrix Y. I would like to normalize each row of Y, so that each row has zero mean. I first tried this. But the mean of each row is also subtracted from the zero entries, wh

I am trying to use SVD decomposition to calculate a covariance matrix however it is not working correctly. I have built the following function. Can anyone help point me to the mistake I have made. Tha

I know there are quite a few good ways to store a sparse matrix without taking up much memory. But I'm wondering whether there is a good way to store a sparse matrix during the construction of it? Her

is there an easy way to shuffle a sparse matrix in python? This is how I shuffle a non-sparse matrix: index = np.arange(np.shape(matrix)[0]) np.random.shuffle(index) return matrix[index] How can I d

In a scipy program I'm creating a dia_matrix (sparse matrix type) with 5 diagonals. The centre diagonal the +1 & -1 diagonals and the +4 & -4 diagonals (usually >> 4, but the principle i

I have an original matrix that looks something like this: (32, 69901) 1 (108, 69901) 1 (32, 69902) 1 (108, 69903) 1 (108, 69904) 1 (432, 69905) 1 (432, 69906) 1 (432, 69907) 1 I want to separate the

I'm trying to build and update a sparse matrix as I read data from file. the matrix is of size 100000X40000 what is the most efficient way of updating multiple entries of the sparse matrix. specifical

Basically, I am just trying to do a simple matrix multiplication, specifically, extract each column of it and normalize it by dividing it with its length. #csc sparse matrix self.__WeightMatrix__ = s

I am examining java version sparse matrix multiplication program which is from JGF benchmark. I run this program in many kinds of cpu frequency. I also do some profile for this program. I classify it

I'm working in Matlab and I have the next problem: I have a B matrix of nx2 elements, which contains indexes for the assignment of a big sparse matrix A (almost 500,000x80,000). For each row of B, the

What is the fastest way to convert 1/0 sparse matrix to 0/1 sparse matrix without using todense() method? Example: Source matrix looks like: matrix([[1, 1, 0, 0, 0, 0, 0, 0, 1, 1], [1, 1, 0, 0, 1, 1,

Let A be a sparse matrix in coordinate format [row(int) col(int) val(float)]. If a upper triangular sparse matrix of A is needed, then the same can be obtained using logical indexing like: A = A(A(:,1

I am working on an algorithm that uses diagonal and first off-diagonal blocks of a large (will be e06 x e06) block diagonal sparse matrix. Right now I create a dict that stores the blocks in such a wa

I have two large sparse matrices: In [3]: trainX Out[3]: <6034195x755258 sparse matrix of type '<type 'numpy.float64'>' with 286674296 stored elements in Compressed Sparse Row format> In [

In my Python code, I was computing SVD of some data using numpy.linalg.svd: from numpy import linalg (_, _, v) = linalg.svd(m) V matrix returned by this was: [[ 0.4512937 -0.81992002 -0.35222884] [-0

I want to do SVD on a sparse matrix by using scipy: from svd import compute_svd print(The size of raw matrix: +str(len(raw_matrix))+ * +str(len(raw_matrix[0]))) from scipy.sparse import dok_matrix

Does anybody know of a sparse SVD solver for c++? My problem involves some badly conditioned matrices that might have zeroed columns/rows. My data is stored in a uBLAS matrix which is the Harwell-Boei

I have difficulties to determine the stationary distribution of a markov model. I start to understand the theory and connections: Given a stochastic matrix, to dermine the stationary distribution we n

I have time data with irregular intervals and I need to convert it to a sparse matrix for use with a graphing library. The data is currently in the following format: { :series1 => [entry, entry, en

I want to save two sparse matrix Y and R to a mat file. However, when I run the following code, I found out that the twomatrices.mat contains two full matrices instead of sparse matrices. Do does .mat

Does anyone know how to perform svd operation on a sparse matrix in python? It seems that there is no such functionality provided in scipy.sparse.linalg.

I process rather large matrices in Python/Scipy. I need to extract rows from large matrix (which is loaded to coo_matrix) and use them as diagonal elements. Currently I do that in the following fashio

I have a very big and sparse matrix, represented as a CSV file (67 GB). Is it possible to load and work with this matrix in Matlab? I can use a 64bit version on a MAC OS computer, 8GB RAM. I have read

After learning about the options for working with sparse matrices in R, I want to use the Matrix package to create a sparse matrix from the following data frame and have all other elements be NA. s r

I am trying to do some k-means clustering on a very large matrix. The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of 1 values per row). The whole thing does not f

Any way of computing in opencv the eigenvalues and eigenvectors of an asymmetric matrix?

I have three large matrices: I, G, and G^2. These are 4Million x 4Million matrices and they are sparse. I would like to check if they are linearly independent and I would like to do this in R. For sm

I am using Scipy to construct a large, sparse (250k X 250k) co-occurrence matrix using scipy.sparse.lil_matrix. Co-occurrence matrices are triangular; that is, M[i,j] == M[j,i]. Since it would be high

I have a large sparse matrix (lil) implemented in Python with scipy, which comprises of Users on one axis, and songs they played on the other. So each row is a linked list of the songs that user has p

I have a very large and very sparse matrix, composed of only 0s and 1s. I then basically handle (row-column) pairs. I have at most 10k pairs per row/column. My needs are the following: Parallel inser

I am looking for a C library to solve linear and, if possible, nonlinear matrix equation of the form Ax = b. It is important to me, that the packages are not too big and free of charge. Speed does not

I am attempting to cluster a set of data points that are represented as a sparse scipy matrix, X. That is, >>> print type(X) <class 'scipy.sparse.csr.csr_matrix'> >>> print X.s

I have an adjacency weight matrix of a graph. I want to call the bellman ford algorithm of the matlabbgl libaray. How can I convert the matrix to a sparse format?

When trying to run the cor() function on sparse matrices (of either type dgCMatrix or dgTMatrix) I get the following error: Error in cor(x) : supply both 'x' and 'y' or a matrix-like 'x' Converting m

I have a sparse matrix created from R's Matrix package. I would like to iterate over each entry in the matrix and perform an operation, saving the result in another sparse matrix with the same indexes

I create a sparse matrix in scala breeze, ie using http://www.scalanlp.org/api/breeze/linalg/CSCMatrix.html. Now I want to get a column slice from it. How to do this? Edit: there are some further requ

I am using the cusp library with CUDA to use sparse matrix. Can't I use it in a struct in C like: #include <cusp/coo_matrix.h> #include <cusp/multiply.h> #include <cusp/print.h> #in

I have the following sparse matrix that contains O(N) elements boost::numeric::ublas::compressed_matrix<int> adjacency (N, N); I could write a brute force double loop to go over all the entries

I've read several topics, but I'm lost. I'm quite new to this. I want to store huge sparse matrix and have several idea's but can choose between them. Here's my needs: Adjacency matrix of approx. 50

I'm trying to transpose a sparse matrix in c++. I'm struggling with the traversal of the new transposed matrix. I want to enter everything from the first row of the matrix to the first column of the n

I'm trying to come up with fast algorithm to find result of operation, where L - is symmetric n x n matrix with real numbers. A - is sparse n x m matrix, m < n. Each row has one and only one non