I am working with large sparse matrices (sparse). I have a large sparse matrix of values which needs to be included into a larger sparse matrix. I have an array of `logicals`

which indicates which rows and columns are to be filled up with the smaller matrix values. In this application the smaller of the two matrices is a graph stored as an adjacency matrix and the logicals indicate the node id positions in the larger matrix.

A small toy example to demonstrate what I am doing currently:

```
zz = sparse(4,4); %create a sparse matrix of the final size desired
rr = rand(3,3); %a matrix of values
logical_inds = logical([1 0 1 1]); %a logical array to index with the dimension size of zz
zz(logical_inds,logical_inds) = rr(:,:) %'rr' values are mapped into the subset of 'zz'
```

I see that the 2^{nd} column of `zz`

are zeros, and that the 2^{nd} row are zero values as well. This is the output desired.

In my program get a warning that this "sparse indexing is likely to be slow", and it is. Occasionally when the matrices are very large the program terminates at this line.

How can I create this matrix (`zz`

) with the sparse method? I am unsure how to create the row column indexes from the mask of logicals I have, and how to turn the values of `rr`

into an array ordered appropriately for this new indexing.

**in general `rr`

is very sparse although the mask of logicals addresses the full matrix

To create this matrix with the `sparse`

function the logical indices will need to be converted into row and column indices, **so this may end up being slower...**

Here the locations of ones in the logical vector are found and then a matrix is created containing the row and column indices for the non zeros in the sparse matrix.

Finally the sparse function is used to create the sparse matrix with the elements of `rr`

in these locations (`rr(:)`

is used to convert it into a column vector)

```
ind_locs = find(logical_inds);
ind = combvec(ind_locs,ind_locs);
zz = sparse(ind(1,:),ind(2,:),rr(:))
```

I think the problem is mostly due to implicit resizing during the allocation. Here's why I think that:

```
%# test parameters
N = 5000; %# Size of 1 dimension of the square sparse
L = rand(1,N) > 0.95; %# 5% of rows/cols will be non-zero values
M = sum(L);
rr = rand(M); %# the "data" to fill the sparse with
%# Method 1: direct logical indexing
%# (your original method)
zz1 = sparse(N,N);
tic
zz1(L,L) = rr;
toc
%# Method 2: test whether the conversion to logical col/row indices matters
zz2 = sparse(N,N);
inds = zz1~=0;
tic
zz2(inds) = rr;
toc
%# Method 3: test whether the conversion to linear indices matters
zz3 = sparse(N,N);
inds = find(inds);
tic
zz3(inds) = rr;
toc
%# Method 4: test whether implicit resizing matters
zz4 = spalloc(N,N, M*M);
tic
zz4(inds) = rr;
toc
```

Results:

```
Elapsed time is 3.988558 seconds. %# meh M1 (original)
Elapsed time is 3.916462 seconds. %# meh M2 (expanded logicals)
Elapsed time is 4.003222 seconds. %# meh M3 (converted row/col indices)
Elapsed time is 0.139986 seconds. %# WOW! M4 (pre-allocated memory)
```

So apparently (and amazingly), it would seem that MATLAB does not grow the existing sparse *before* the allocation (as you'd expect), but actually loops through the row/col indices and grows the sparse *during* the iteration. Therefore, it would seem one has to "help" MATLAB a bit:

```
%# Create initial sparse
zz1 = sparse(N,N);
%# ...
%# Do any further operations until you can create rr:
%# ...
rr = rand(M); %# the "data" to fill the sparse with
%# Now that the size of the data is known, re-allocate space for the sparse:
tic
[i,j] = find(zz1); %# indices
[m,n] = size(zz1); %# Sparse size (you can also use N of course)
zz1 = sparse(i,j,nonzeros(zz1), m,n, M*M);
zz1(L,L) = rr; %# logical or integer indices, doesn't really matter
toc
```

Results (for the same `N`

, `L`

and `rr`

):

```
Elapsed time is 0.034950 seconds. %# order of magnitude faster than even M4!
```

See also this question+answers.

Similar Questions

I am new to matlab. I want to do the following: Generate an array of a thousand replications of a random draws between three alternatives A,B and C, where at every draw, each alternative has the same

I'm working with a 30*26000 size matrix that has NaNs at the beginning and at the end. NaNs are also sprinkled throughout each row. I can fill in the NaNs with linear interpolation but that will leave

I am using MATLAB to load a text file that I want to make a sparse matrix out of. The columns in the text file refer to the row indices and are double type. I need them to be integers to be able to us

I have to create a very big 3D matrix (such as: 500000x60x60). Is there any way to do this in matlab? When I try omega = zeros(500000,60,60,'single'); I get an out-of-memory error. The sparse functio

In MATLAB I have a very large matrix (matrix A). Now I would like to find the row-index of the row which contain certain values in the second column. These values - which I'm looking for in Matrix A -

I want to pass a sparse matrix to a shared library from MATLAB, do some operation there (written in C), and then return it. I can pass a dense matrix and use, pretty easy. But, I have no idea how to p

I want to sum the elements of the matrix M according to the the values in the matrix R into the array d. Theoretically, it's cannot be serialized, because the action of summing into one array (D) requ

I've been wondering about this question for quite a while but cannot find a reference: How does Matlab transpose a sparse matrix so fast, given that it is stored in CSC (compressed sparse column) form

This seems to be a very common problem of mine: data = [1 2 3; 4 5 6]; mask = [true false true]; mask = repmat(mask, 2, 1); data(mask) ==> [1; 4; 3; 6] What I wanted was [1 3; 4 6]. Yes I can just

I have a large matrix that I would like to convert to sparse CSR format. When I do: import scipy as sp Ks = sp.sparse.csr_matrix(A) print Ks Where A is dense, I get (0, 0) -2116689024.0 (0, 1) 39462

I'm trying to dynamically construct a 2-D matrix with numpy based on the values of an array, like this: In [113]: A = np.zeros((5,5),dtype=bool) In [114]: A Out[114]: array([[False, False, False, Fals

I have a large sparse graph that I am representing as an adjacency matrix (100k by 100k or bigger), stored as an array of edges. An example with a (non-sparse) 4 by 4 matrix: 0 7 4 0 example_array = [

I have a large 4D data set and need to create a smaller 4D array from it. I am fairly new to python and am use to IDL or matlab. I read in my values then using the where function I find the index numb

I have a large sparse matrix X in scipy.sparse.csr_matrix format and I would like to multiply this by a numpy array W making use of parallelism. After some research I discovered I need to use Array in

I'm currently working on a scipy sparse csr matrix. I would like to delete all rows in the matrix that contain 0 in the data array of the matrix (the data array is the 1s and 2s you can see in the exa

In Octave/MATLAB, say I have: all = { [1,1] = one [1,2] = two [1,3] = three [1,4] = four [1,5] = five [1,6] = six [1,7] = seven [1,8] = eight [1,9] = nine [1,10] = ten } some = { [1,1] = someword [1,2

I need to do matrix operations (mainly multiply and inverse) of a sparse matrix SparseMat in OpenCV. I noticed that you can only iterate and insert values to SparseMat. Is there an external code I can

I have a question about how can I find and replace some values by area. I have three matrixs. It has the same length = 1700*1700. The third matrix, called 'mask' has two different values= 0 and 255. a

My m file is: x=0:0.01:2*pi; y=sin(x); plot(x,y) I want build this program by .exe format. I want run this .exe in a computer that doesn't have matlab.

I process rather large matrices in Python/Scipy. I need to extract rows from large matrix (which is loaded to coo_matrix) and use them as diagonal elements. Currently I do that in the following fashio

This question already has an answer here: Removing rows and columns from MATLAB matrix quickly 3 answers I have an array X (array of vectors) which is formed by 2 lines and 3 columns. a1=[1 2

I have something like the following: X = [1 5; 2 5; 3 7; 4 7; 5 2; 6 7]; X = 1 5 2 5 3 7 4 7 5 2 6 7 b = [1;4;6]; b = 1 4 6 Is there a way to lookup the values (from X), input as b, to output the fol

I have a matrix NxN. I want to extract values only from odd columns in this matrix. http://photoload.ru/data/38/5d/d2/385dd20f148fd21a08de36a9c03e69a1.jpg And after generate new matrix by this values

I am trying to find an efficient way that lets me increase the top k values of a sparse matrix by some constant value. I am currently using the following code, which is quite slow for very large matri

I have large array. Now I need a matrix with 8 elements in every row. My array looks like this: A= Columns 1 through 18 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 1 0 0 Columns 19 through 36 0 0 0 0 0 0 0 0 0 1

In the context of a finite element problem, I have a 12800x12800 sparse matrix. I'm trying to solve the linear system just using MATLAB's \ operator to solve and I get an out of memory error using mld

I would like to generate 3-dimenstional matrix M of values sampled uniformly from the array of discrete values [0.2 0.4 0.6 0.8 1.0], with replacement. M should be of the dimension n*m*m, where n=4 an

Suppose I have a numpy array a = np.array([0, 8, 25, 78, 68, 98, 1]) and a mask array b = [0, 1, 1, 0, 1] Is there an easy way to get the following array: [8, 25, 68] - which is first, second and for

I am using MATLAB. I have very large sparse matrices, and I want to perform a logical or bsxfun on each column of this matrix. There is a single for loop where in it is a single operation of logical t

This question already has an answer here: 2D logical matrix from vector of coordinates (Basic matlab) 2 answers I would like to use values from a matrix to index an array. I will use a 3x2 matr

I would like to get the minimum nonzero values per row in a sparse matrix. Solutions I found for dense matrices suggested masking out the zero values by setting them to NaN or Inf. However, this obvio

I have a very large and very sparse matrix, composed of only 0s and 1s. I then basically handle (row-column) pairs. I have at most 10k pairs per row/column. My needs are the following: Parallel inser

Possible Duplicate: How can I sort a 2-D array in MATLAB with respect to one column? Sort a matrix with another matrix I have a vector 'A' of 429 values and a matrix 'B' of 429x200 values. Rows in A

I'm involved in the resolution of a system of the type Ax = b, where A is a square sparse matrix, x is the vector of the unknows (I have to compute it) and b is a vector of all zeros excpet for the la

In MATLAB, it is very convenient to create a pentadiagonal sparse matrix using commands like this: I = eye(m); % create identity matrix e = ones(m,1); % create an array of all 1's T = spdiags([e -4*e

I am writing a MEX-file that involves reading a sparse matrix from Matlab, but Matlab crashes every time I ran the program. Then I write a test file, finding that the problem is exactly in the mxGetIr

I've a large D Matrix of size MxNxK. Given a binary mask B of size MxN, I would like to split the matrix D into two submatrices: D0 and D1, in such a way that matrix D0 has the values of matrix D asso

consider the code below, Am trying to make a Sparse_matrix that contains an array of pointer. In struct Sparse_matrix, Is this a proper way to create a pointer array when size is unknown? What's the

In scipy, we can construct a sparse matrix using scipy.sparse.lil_matrix() etc. But the matrix is in 2d. I am wondering if there is an existing data structure for sparse 3d matrix / array (tensor) in

I want to read data from a text file in Matlab. The file describes a surface consists of two parts: 1)A header with information on x,y dimensions in pixels and nm etc. 2)Matrix with height values (512

I have a code that slices a numpy array into a circle. I wish to recover only the values included in a certain range of angles from the circle and mask the array. For example: mask the original array

Is it possible to apply for example numpy.exp or similar pointwise operators to all elements in a scipy.sparse.lil_matrix or another sparse matrix format? import numpy from scipy.sparse import lil_mat

I have two arrays of different size A and B. I am comparing A to B and want to find for each element in A the first element in B that is smaller/larger. I would like to use something like ismember(A,B

I am using the C++ Eigen 3 library in my program. In particular, I need to multiply two Eigen sparse matrix and store the result into another Eigen sparse matrix. However, I noticed that if the some e

I have a matrix that is outputted like this: maximums = [ -9.9043877608991468201413092380493, 426.34796945271797204125533010993] [ 9.3758615553048990076305298649689, 441.87005169359418197397861057075

If have an array (5000x1 double) in matlab workspace. I put the 'from workspace' block in simulink window for input of another block, But when run the program this error occurred: Invalid matrix-form

I have a matrix: raw = 'alon' 'tomer' 'ilana' 'T1' '2' '53' '24' 'NaN' 'ilana' 'oren' '6' 'a' 'g' 'g' 'gsh' 'NaN' 'dikla' 'aba' 'mama' 'a' 'h' 'ghfs' 'bfsbf' 'NaN' '4' 'NaN' 'NaN' 'nn' 'NaN' 'NaN' 'ha

I have a v = 2x30 matrix and I need to take make another matrix to take all points of v whose y coordinates are positive

First Array is Array ( [2] => Course 1 [3] => C2 [4] => COMPUTATIONAL MATHEMATICS -I [5] => BASIC ELECTRONICS [6] => DATA STRUCTURE [7] => COMMUNICATION SKILL [8] => SYSTEMS PROGR

I have a 1034_by_1034 sparse matrix (scipy.sparse.csr.csr_matrix), which basically represents the adjacency matrix of a graph. I want to check if some elements are ones or not. But I found this to be