I have a very large and sparse matrix of size 180GB(text , 30k * 3M) containing only the entries and no additional data. I have to do matrix multiplication , inversion and some similar linear algebra operations over it. I tried octave and simple single-threaded C code for the multiplication but my system RAM of 40GB gets used up very fast and then I can find the program starts thrashing. Is there any other options available to me. I am not familiar with MathLab or any other matrix operational library that can help me in doing so.

When I run a simple matrix multiplication of two matrices with 10 rows and 3 M cols, and its transpose, it gives the following error :

```
memory exhausted or requested size too large for range of Octave's index type
```

I am not sure whether the same would work on Matlab or not. For sparse matrix representation and matrix multiplication, is there another library or code.

Have you considered performing your processing in blocks? Transposition and multiplications work very well with block matrix processing (see https://en.wikipedia.org/wiki/Block_matrix) and that will get you around any limitations about the indices.

This wouldn't help you with matrix inversion though unless you can decompose your matrix in blocks when blocks that aren't on the diagonal are completely empty, which isn't stated in your assumptions.

if there are few enough nonzero entries, I suggest creating a sparse matrix S with appropriate dimensions and max nonzero entries; see matlab create sparse matrix. Then as @oleg komarov described, load the matrix in blocks and assign the nonzero entries from each block into the correct address in the sparse matrix S. I feel that if your matrix is sparse enough, then loading it is really the only difficulty you face. I had similar issues with large transfer operators.

Octave has a limit in both the memory resources of about 2GB and the maximum number of indices a matrix can hold of about 2^32 (for 32 bits Octave). MatLab doesn't have such a memory limit, since it will use all of your memory resources, swapping file included. Thus you could try with MatLab by setting a huge swapfile, you may then compute your operations (but it will anyway take quite along time...).

If you are interested by other approaches, you may take a look into out-of-core computing which aims to promote new methods to process huge datasets that cannot reside all in memory, but rather store it on disk and load efficiently the bits that are necessary.

For a practical approach, you may take a look into Blaze for Python (notice: still in development!).

Similar Questions

When dealing with sparse matrices, how do I convert Matrix Market format into CRS (Compressed Row Storage)?

I noticed Pandas now has support for Sparse Matrices and Arrays. Currently, I create DataFrame()s like this: return DataFrame(matrix.toarray(), columns=features, index=observations) Is there a way to

Anyone know of any good sparse matrix library? I need it for doing kronecker products and multiplication on large sparse matrices (10,000 x 10,000). Right now we are using R, which handles them prett

(using MATLAB) I have a large coordinate matrix and a large sparse adjacency matrix for which coordinates are connected to each other. I had asked previously on SO, how to efficiently compute these di

I have an RDD as such: byUserHour: org.apache.spark.rdd.RDD[(String, String, Int)] I would like to create a sparse matrix of the data for calculations like median, mean, etc. The RDD contains the row_

I am trying to create a large sparse matrix, 10^5 by 10^5 in R, but am running into memory issues. > Matrix(nrow=1e5,ncol=1e5,sparse=TRUE) Error in Matrix(nrow = 1e+05, ncol = 1e+05, sparse = TRUE)

I am creating a matrix from a Pandas dataframe as follows: dense_matrix = numpy.array(df.as_matrix(columns = None), dtype=bool).astype(np.int) And then into a sparse matrix with: sparse_matrix = scip

Given a large sparse matrix (say 10k+ by 1M+) I need to find a subset, not necessarily continuous, of the rows and columns that form a dense matrix (all non-zero elements). I want this sub matrix to b

I have a sparse matrix: from scipy import sparse a = sparse.diags([1,4,9],[-1,0,1],shape =(10,10),format =csr) I want to take the square root of each of the elements in the sparse matrix I look up

I have the following matrix which I believe is sparse. I tried converting to dense using the x.dense format but it never worked. Any suggestions as to how to do this?, thanks. mx=[[(0, 2), (1, 1), (2,

I've been wondering about this question for quite a while but cannot find a reference: How does Matlab transpose a sparse matrix so fast, given that it is stored in CSC (compressed sparse column) form

I am trying to do a Matrix-Vector Multiplication on GPU (using Cuda). I loaded the matrix on my C++ code (CPU), and then I perform the matrix-vector multiplication using cuda. I am also using shared m

I have to implement sparse matrix and do some decompositions like Cholesky Decomposition, LU Decomposition, QR Decomposition on it. Actually I found a library called JAMA which is capable of doing th

I'm trying to figure out how to iterate through a scipy sparse matrix by column. I'm trying to compute the sum of each column, then weight the members of that column by that sum. What I want to do is

That is, when I do A\b for a very large, symmetric and sparse A, what algorithm does matlab use?

I want to initialise a sparse matrix (for use with scipy minimum_spanning_tree if that matters) from a list of matrix coordinates and values. That is, I have: coords - Nx2 array of coordinates to be s

I am starting dealing with sparse matrices so I'm not really proficient on this topic. My problem is, I have a simple coo-occurrences matrix from a word list, just a 2-dimensional co-occurrence matrix

I initialize an empty sparse matrix using S = scipy.sparse.lil_matrix((n,n),dtype=int) As expected print S doesn't show anything, since nothing has been assigned. Yet if I test: print S[0,0]==0 I r

I have a 3007 x 1644 dimensional matrix of terms and documents. I am trying to assign weights to frequency of terms in each document so I'm using this log entropy formula http://en.wikipedia.org/wiki/

Given a sparse binary matrix A (csr, coo, whatever) I want to make a plot such that I can see the position (i,j) = white in the figure if A(i,j) = 1, and (i,j) = black if A(i,j) = 0; For a dense numpy

I want to initial a sparse matrix with numpy array. The numpy array contains NaN as zero for my program, the code to initial a sparse matrix as following: a= np.array([[np.NaN,np.NaN,10]]) zero_a= np.

Say that I have a sparse matrix in scipy.sparse format. How can I extract a diagonal other than than the main diagonal? For a numpy array, you can use numpy.diag. Is there a scipy sparse equivalent? F

I am trying to use the very recent capability of the RcppArmadillo package (version 0.3.910.0 with R 3.0.1 and evrerything up to date) for conversion of a sparse matrix from the Matrix package (class

I need to insert the zero elements in any sparse matrix in the Matrix Market format (but already without the headers). The first column is the number of the ROW, the second columns is the number of th

I'm currently working with sparse matrices, and I have to compare the computation time of sparse matrix-matrix multiplication with full matrix-matrix multiplication. The issue is that sparse matrix co

How can a sparse matrix - matrix product be calculated? I know the 'classic' / mathematical way of doing it, but it seems pretty inefficient. Can it be improved? I thought about storing the first matr

I'm populating a sparse matrix in R and have written the update in a for loop but was hoping to get some pointers to make it quicker. Here is some example code: library(Matrix) rowId <- rep(c(101:1

I'm writing a lexer generator as a spare time project, and I'm wondering about how to go about table compression. The tables in question are 2D arrays of short and very sparse. They are always 256 cha

I have never use R ,but now I need import a sparse matrix to do association rule in R My import data is a sparse matrix like this: i j x 1 2 3 1 2 3 5 1 3 3 1

I am trying to cPickle a large scipy sparse matrix for later use. I am getting this error: File tfidf_scikit.py, line 44, in <module> pickle.dump([trainID, trainX, trainY], fout, protocol=-1)

I want to pass a sparse matrix to a shared library from MATLAB, do some operation there (written in C), and then return it. I can pass a dense matrix and use, pretty easy. But, I have no idea how to p

Let's say I have A = [3 0 2; ... 0 0 1; ... 1 1 0] A = sparse(A); The output of which is below: ans = (1,1) 3 (3,1) 1 (3,2) 1 (1,3) 2 (2,3) 1 Question: is there an easy command to generate the fol

How can you take the log base 10 of every element in a sparse matrix (COO)? >>print type(X) <class 'scipy.sparse.coo.coo_matrix'> I've tried this but it doesn't work: import math X.data =

I have a large 500x53380 sparse matrix and trying to dichotomize it. I have tried using event2dichot under sna package but no success because it requires an adjacency matrix or network object. I al

I am working on an algorithm that uses diagonal and first off-diagonal blocks of a large (will be e06 x e06) block diagonal sparse matrix. Right now I create a dict that stores the blocks in such a wa

How can we display a sparse matrix? Someone who wanted to help me said: use linked lists. Also I have read some where that we should create class like this : SpMatrix { int non_zero_value; int i,j; }

I have to process a large sparse matrix whose size is 6004*17842 (doc*terms). The function find() has been tried to get its rows, cols and values and the result has been save in ascii form. But the te

I need to do matrix operations (mainly multiply and inverse) of a sparse matrix SparseMat in OpenCV. I noticed that you can only iterate and insert values to SparseMat. Is there an external code I can

I have an assignment where Im supposed to finish the implementation on a generic sparse matrix. Im stuck on the addition part. The matrix is only going to support numbers so I had it extend Number hop

I am looking for a good (in the best case actively maintained) C++ matrix library. Thereby it should be templated, because I want to use a complex of rationals as numerical type. The matrices what I a

I have a large csv of similarities between keywords that I would like to convert it to a triangular distance matrix (because it is very large and sparse would be even better) to perform hierarchical c

I got a problem when using octave sparse matrix. max(speye(65536)(:)) will result in a 0x0 variable. However, speye(65535) and speye(65537) works. How that happens? My octave version is 3.2.4 in Fedo

I'd like to accumulate indexed elements in a matrix, like table and tapply function in R. I found sparse(i,j,s,m,n) fit my need perfectly, As the document says:Any elements of s that have duplicate v

I'm working in Matlab and I have the next problem: I have a B matrix of nx2 elements, which contains indexes for the assignment of a big sparse matrix A (almost 500,000x80,000). For each row of B, the

I have a list of 100k items and each item has a list of indices. I am trying to put this into a boolean sparse matrix for vector multiplication. My code isn't running as fast as I would like, so I am

I am using Boost's uBLAS in a numerical code and have a 'heavy' solver in place: http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?LU_Matrix_Inversion The code works excellently, however,

Can anyone tell me a smart way how to calculate the distance to the non-zero neighbors in a sparse matrix? The reason to do this is that I want to identify holes in a large, multidimensional matrix.

I'm trying to write a matrix to sparse matrix conversion program. But, there is a problem in my scan_matrix function.I couldn't figure out where is the problem.According to Nemiver(debugging program)

I am new to the use of sparse matrices, but now need to utilize one in my work to save space. I understand that the following matrix: 10 0 0 0 -2 0 3 9 0 0 0 3 0 7 8 7 0 0 3 0 8 7 5 0 0 8 0 9 9 13 0 4

I have a numpy array such as: array = [0.2, 0.3, 0.4] (this vector is actually size 300k dense, I'm just illustrating with simple examples) and a sparse symmetric matrix created using Scipy such as f