I'm wondering what the best way is to iterate nonzero entries of sparse matrices with scipy.sparse. For example, if I do the following:

```
from scipy.sparse import lil_matrix
x = lil_matrix( (20,1) )
x[13,0] = 1
x[15,0] = 2
c = 0
for i in x:
print c, i
c = c+1
```

the output is

```
0
1
2
3
4
5
6
7
8
9
10
11
12
13 (0, 0) 1.0
14
15 (0, 0) 2.0
16
17
18
19
```

so it appears the iterator is touching every element, not just the nonzero entries. I've had a look at the API

http://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.lil_matrix.html

and searched around a bit, but I can't seem to find a solution that works.

Thanks in advance for your help.

Try `filter(lambda x:x, x)`

instead of `x`

.

Edit: bbtrb's method (using coo_matrix) is much faster than my original suggestion, using nonzero. Sven Marnach's suggestion to use `itertools.izip`

also improves the speed. Current fastest is `using_tocoo_izip`

:

```
import scipy.sparse
import random
import itertools
def using_nonzero(x):
rows,cols = x.nonzero()
for row,col in zip(rows,cols):
((row,col), x[row,col])
def using_coo(x):
cx = scipy.sparse.coo_matrix(x)
for i,j,v in zip(cx.row, cx.col, cx.data):
(i,j,v)
def using_tocoo(x):
cx = x.tocoo()
for i,j,v in zip(cx.row, cx.col, cx.data):
(i,j,v)
def using_tocoo_izip(x):
cx = x.tocoo()
for i,j,v in itertools.izip(cx.row, cx.col, cx.data):
(i,j,v)
N=200
x = scipy.sparse.lil_matrix( (N,N) )
for _ in xrange(N):
x[random.randint(0,N-1),random.randint(0,N-1)]=random.randint(1,100)
```

yields these `timeit`

results:

```
% python -mtimeit -s'import test' 'test.using_tocoo_izip(test.x)'
1000 loops, best of 3: 670 usec per loop
% python -mtimeit -s'import test' 'test.using_tocoo(test.x)'
1000 loops, best of 3: 706 usec per loop
% python -mtimeit -s'import test' 'test.using_coo(test.x)'
1000 loops, best of 3: 802 usec per loop
% python -mtimeit -s'import test' 'test.using_nonzero(test.x)'
100 loops, best of 3: 5.25 msec per loop
```

The fastest way should be by converting to a `coo_matrix`

:

```
cx = scipy.sparse.coo_matrix(x)
for i,j,v in zip(cx.row, cx.col, cx.data):
print "(%d, %d), %s" % (i,j,v)
```

I had the same problem and actually, if your concern is only speed, the fastest way (more than 1 order of magnitude faster) is to convert the sparse matrix to a dense one (x.todense()), and iterating over the nonzero elements in the dense matrix. (Though, of course, this approach requires a lot more memory)

tocoo() materializes the entire matrix into a different structure, which is not the preferred MO for python 3. You can also consider this iterator, which is especially useful for large matrices.

```
from itertools import chain, repeat
def iter_csr(matrix):
for (row, col, val) in zip(
chain(*(
repeat(i, r)
for (i,r) in enumerate(comparisons.indptr[1:] - comparisons.indptr[:-1])
)),
matrix.indices,
matrix.data
):
yield (row, col, val)
```

I have to admit that I'm using a lot of python-constructs which possibly should be replaced by numpy-constructs (especially enumerate).

**NB**:

```
In [43]: t=time.time(); sum(1 for x in rather_dense_sparse_matrix.data); print(time.time()-t)
52.48686504364014
In [44]: t=time.time(); sum(1 for x in enumerate(rather_dense_sparse_matrix.data)); print(time.time()-t)
70.19013023376465
In [45]: rather_dense_sparse_matrix
<99829x99829 sparse matrix of type '<class 'numpy.float16'>'
with 757622819 stored elements in Compressed Sparse Row format>
```

So yes, enumerate is somewhat slow(ish)

For the iterator:

```
In [47]: it = iter_csr(rather_dense_sparse_matrix)
In [48]: t=time.time(); sum(1 for x in it); print(time.time()-t)
113.something something
```

So you decide whether this overhead is acceptable, in my case the tocoo caused `MemoryOverflows`

's.

IMHO: such an iterator should be part of the csr_matrix interface, similar to items() in a dict() :)

To loop a variety of sparse matrices from the `scipy.sparse`

code section I would use this small wrapper function (note that for Python-2 you are encouraged to use `xrange`

and `izip`

for better performance on large matrices):

```
from scipy.sparse import *
def iter_spmatrix(matrix):
""" Iterator for iterating the elements in a ``scipy.sparse.*_matrix``
This will always return:
>>> (row, column, matrix-element)
Currently this can iterate `coo`, `csc`, `lil` and `csr`, others may easily be added.
Parameters
----------
matrix : ``scipy.sparse.sp_matrix``
the sparse matrix to iterate non-zero elements
"""
if isspmatrix_coo(matrix):
for r, c, m in zip(matrix.row, matrix.col, matrix.data):
yield r, c, m
elif isspmatrix_csc(matrix):
for c in range(matrix.shape[1]):
for ind in range(matrix.indptr[c], matrix.indptr[c+1]):
yield matrix.indices[ind], c, matrix.data[ind]
elif isspmatrix_csr(matrix):
for r in range(matrix.shape[0]):
for ind in range(matrix.indptr[r], matrix.indptr[r+1]):
yield r, matrix.indices[ind], matrix.data[ind]
elif isspmatrix_lil(matrix):
for r in range(matrix.shape[0]):
for c, d in zip(matrix.rows[r], matrix.data[r]):
yield r, c, d
else:
raise NotImplementedError("The iterator for this sparse matrix has not been implemented")
```

Similar Questions

I am trying to extract values from a datagrid, by iterating through all the rows of the datagrid foreach (DataRow drv in PGIPortfolio.Items) { // DataRow row = drv.Row; string acname = drv[Portfolio

so let's say i have a vector, wich contains 4 elemens [string elements]. I need to loop through the vector first , then through each element , loop through an array of chars [vowels] and to count how

I have a vector K=[3,4,5,...] where the number of elements of K = n. I also have a 3xn identity matrix. I would like to place the numbers from K on the diagonal of the identity matrix i.e for n=3 K=[3

Could someone tell me what I am doing wrong here? I have the following jQuery. It is supposed to iterate through each .box, get the attribute 'fade' and apply the individual fade times to each .box. H

I tried to measure the difference of different matrix-vector-multiplication schemes in Fortran. I have actually written the following code: http://pastebin.com/dmKXdnX6 The 'optimized version' is mean

This question already has an answer here: Performance issue for vector::size() in a loop 8 answers When i looked at the typical way of iterating through a vector... for (size_t i = 0; i < my

Let's say I have a boost interprocess vector in shared memory. One process may be iterating through it. If another process wants to concurrently clear the vector, what do I need to do so that I don't

I have a program that maintains an integer vector. I have three functions that add an integer, remove an integer and check whether an integer is already in the vector. The problem is with the last one

in a service application I am iterating through the Windows application event log to parse Events in order react depanding on the entry message. In the case that the event log is full (Windows usually

I have a 100*8 data matrix where each row is a vector of values at 8 different time points. I am interested to know how to plot the following matrix in R to get the graph closely similar to the one be

I have a list that looks like this: A = [(1,1,1,2,2), (1,1,3,2,2), (1,1,6,2,2), (1,1,5,2,2), (1,1,2,5,2), (2,1,1,1,2) ...] I'm running the elements through a simple for-loop like so: n = len(A); for

I've a JSON response that I'd like to iterate through and capture the values of incident number, the longtitude, latitude as a group and send it to an SMS messaging service. I've tried different appro

Is there a way, in R, to save a whole vector into one value of a matrix or data frame, without having to combine it into a single value first? For example, if I had a vector.. pk<-c(0.021477,0.0211

How can I iterate through my arguments to print these lines out for each of the arguments in my function, instead of typing each of them out? def validate_user(surname, username, passwd, password, err

I'm optimizing a function and I want to get rid of slow for loops. I'm looking for a faster way to multiply each row of a matrix by a vector. Any ideas? EDIT: I'm not looking for a 'classical' multipl

I know this has been asked and answered a couple times already, but I'm still confused about how to reference the current object when iterating over a jQuery array. For example, the following code giv

I am trying to iterate through an SQLite database and perform checks or operations on the objects in the list. I need to use a database because the eventual number of objects will be quite large and a

Short Description: I am iterating over a vector calling a virtual function on every object in the vector in order to execute a sequence of actions. The vector is of the base class as is the iterator.

How to Declare array in stored procedure. Adding values to array through query. and iterating through a loop for array elements in mssql stored procedure .

I'm trying to iterate through objects in Twig, but am having no such luck. Here's what I've tried so far. Here's my default controller method: public function ReturnPhonesAction() { $phonequery = $thi

I have a basic matrix mat and I hope to get an R object x = (mat, mat, ...) where mat is repeated for 100 times. If this is possible, then I can pass x to a function which takes a vector of matrix nam

In my controller, I'm returning returning View with a list of products. return View(ProductList) In my view I want to get all the values of the product list WITHOUT iterating through them either with

Possible Duplicate: How do you concatenate the rows of a matrix into a vector in MATLAB? Hi, Does anyone know what is the best way to create one row matrix (vector) from M x N matrix by putting all

Is it possible to iterate a vector from the end to the begin? for (vector<my_class>::iterator i = my_vector.end(); i != my_vector.begin(); /* ?! */ ) { } Or is that only possible with something

I need to go through this vector and delete the duplicates. A segmentation fault is occurring somewhere within this code. My guess is that it has something to do with deleting elements while the itera

I'm having some trouble iterating through ArrayList. I have a class named Row which extends ArrayList<String>. I have another class named Table which extends ArrayList<Row>. I am trying to

I have a vector that holds items that are either active or inactive. I want the size of this vector to stay small for performance issues, so I want items that have been marked inactive to be erased fr

Possible Duplicate: What are the most widely used C++ vector/matrix math/linear algebra libraries, and their cost and benefit tradeoffs? just need a publicly downloadable vector/matrix class that ha

I already tried using PHP 5.3 with the dbase extension, but it doesn't work reliably for large databases over 2 GB. I need a way to iterate through a subsection of a large DBF and have the ability to

Is there an elegant method of iterating through a multidimensional vector? Say, for instance, you have a 6D vector, though I think a 2D would suffice. Something like vector< vector< int > my

How does one multiply a matrix and a vector element wise in core.matrix? I am trying to find a core.matrix equivalent of the following octave code: A = [1 2 3 4; 5 6 7 8] B = [2; 3] A .* B ans = 2 4 6

I have a 309 digit integer, I want to iterate through its characters. Currently I am using: require 'openssl' e = 11650501396272635679426984666718814747389912110044906944384450682388585921107384352390

I'm having a really hard time trying to find a way to iterate through this JSON object in the way that I'd like. I'm using only Javascript here. First, here's the object { dialog: { dialog_trunk_1

Say I have a vector that looks like this: 1/2 a + 1/3 b b + c 2a + c 1/3c + 4d Mathematically this can be factorised into matrix and a vector: Matrix: 1/2 1/3 0 0 0 1 1 0 2 0 1 0 0 0 1/3 4 Vector: a

I want to do something like this: Matrix m; // (4x4 matrix) Vector4 v; // (4-elements vector) m(0) = v; // replace first 4-elements matrix row by vector v v = m(0); // replace vector v by first 4-elem

OVERVIEW I am facing performance slowdown while iterating MANY times through a calculator class. Iterations take about 3mn each at the beginning and take longer and longer as the iteration count grows

I'm trying to use numbapro to write a simple matrix vector multiplication below: from numbapro import cuda from numba import * import numpy as np import math from timeit import default_timer as time n

When working with 3d graphics, sample shaders USUALLY use the following operation for vector position transformation: result = mul(matrix, vector); This obviously means the same as: result = mul(vect

I have a task to create a simple game in Smalltalk, a language which I'm largely unfamiliar with. The game is Marble Solitaire, and it involves a 7x7 matrix with the pieces removed in a 2x2 square in

I'm parsing a json from a website and trying to get some data from it. However, it is giving me undefined values when iterating through a collection of objects. If it's a badly formatted json, unfortu

I can't seem to send a vector of a struct that I serialized with msgpack through ZeroMQ. It's a vector of this struct: struct MyData { MyData() : id(0), x(0), y(0), a(0) {} MyData(const Obj &r) :

So suppose I have some kind of tree structure with nodes and children. I want to travel through the tree and shout when I see certain nodes. Look at the following, this is ActionScript but it's simila

I have been advised not to store arrays in my DB, but instead just IDs. In my project, I am storing IDs for line-items. If I don't have a line_item OBJECT, but only IDs, could I still iterate through

I have the following code for iterating through the arrayList: <form:forEach items=${list} var=list> <tr> <td><c:out value=${list} /><td> </tr> </form:for

I'm trying to bruteforce through some list iterations in python (I'm a pynoob) and I can't seem to understand why this is such an issue. My data structure looks like this: pprint.pprint(list) [[13557

Does anyone know how to do array matrix multiplication in matlab? i.e. I have two 3 dimensional arrays consisting of sets of matrices in the first 2 dimensions and I would like to multiply each matrix

I am trying to loop through a JSON Object (DATA) containing three nested JSON Objects (NEWUSERS,NEWUSERDATA,NEWRELATIONS), using a switch function to choose the appropriate function for a MySQL insert

I try to create a Graph class and want to print all Edges to the screen. But I have a problem iterating a 2d vector. class Graph { public: Graph(void); ~Graph(void); vector<Node*> nodes; void ad

Matrix multiplication is a very basic task in image processing and OpenCV takes care of with a overloaded * operator. A STL vector of points can be converted into Mat by casting. vector<Point2f>

I have this matrix: A = [1 3 5 7 9 10]; And this vector: B = zeros(1,10); Now I want to change the elements in the ranges of [1:3],[5:7] and [9:10] to 1 . So, to get this: C = [1 1 1 0 1 1 1 0 1 1];