I use some C++ code to take a text file from a database and create a dgcMatrix type sparse matrix from the `Matrix`

package. For the first time, I'm trying to build a matrix that has more than 2^31-1 non-sparse members, which means that the index vector in the sparse matrix object must also be longer than that limit. Unfortunately, vectors seem to use 32-bit integer indices, as do NumericVectors in Rcpp.

Short of writing an entire new data type from the ground up, does R provide any facility for this? I don't think I can use too exotic a solution as I need `glmnet`

to recognize the resultant object.

In recent versions of R, vectors are indexed by the `R_xlen_t`

type, which is 64 bits on 64 bits platforms and just `int`

on 32 bit platforms.

Rcpp so far still uses `int`

everywhere. I would encourage you to request the feature on their issue list. It is not hard, but needs systematic involvement of someone with skills, time and willingness. The development version of `Rcpp11`

uses the correct type, perhaps they can use that as a model.

Note however that even though R uses 64 bit unsigned integers on 64 bit plaforms, you are in fact limited to the range that can be handled by the `double`

type, which is what R will give you if you ask for the `length`

of a vector. R has no 64 bit integer type that it can represent natively, so when you ask for the length of a vector you either get an `int`

or a `double`

depending on the value.

Similar Questions

I am using Scipy to construct a large, sparse (250k X 250k) co-occurrence matrix using scipy.sparse.lil_matrix. Co-occurrence matrices are triangular; that is, M[i,j] == M[j,i]. Since it would be high

In scipy, we can construct a sparse matrix using scipy.sparse.lil_matrix() etc. But the matrix is in 2d. I am wondering if there is an existing data structure for sparse 3d matrix / array (tensor) in

I have a large sparse matrix X in scipy.sparse.csr_matrix format and I would like to multiply this by a numpy array W making use of parallelism. After some research I discovered I need to use Array in

I've got a scipy.sparse_matrix A and I want to zero-out a decently-sized fraction of the elements. (In the matrices I'm working with today, A has about 70M entries and I want to zero-out about 700K of

If I am using the sparse.lil_matrix format, how can I remove a column from the matrix easily and efficiently?

I'm reading through instructions of Matrix package in R. But I couldn't understand the p argument in function: sparseMatrix(i = ep, j = ep, p, x, dims, dimnames, symmetric = FALSE, index1 = TRUE, give

This Creating sparse matrix in MEX has a good example on mxCreateSparse. But this function return a double sparse matrix instead of single. If I want to return a single sparse matrix, what should I do

I am trying to compute the m first eigenvectors of a large sparse matrix in R. Using eigen() is not realistic because large means N > 106 here. So far I figured out that I should use ARPACK from th

is there an easy way to shuffle a sparse matrix in python? This is how I shuffle a non-sparse matrix: index = np.arange(np.shape(matrix)[0]) np.random.shuffle(index) return matrix[index] How can I d

I want to augment the scipy.sparse.csr_matrix class with a few methods and replace a few others for personal use. I am making a child class which inherits from csr_matrix, as such: class SparseMatrix(

I am using a sparse matrix format implemented in scipy as csr_matrix. I have a mat variable which is in csr_matrix format and all its elements are non-negative. However, when I use mat + mat operation

I have a dataset (x) contains 1 10 20 30 34 38 59 83 ... I have a big matrix nx1. I want to assign a value 1 for each row in x. For example mat[1:10,1] = 1 mat[20:30,1] = 1 etc... In R, the size o

I have this sparse matrix of size 20 millionx20million in matlab. I want to get around 40000 specific rows from this matrix. If I do new_data = data_original(index,:) where index consists of the rows

I got a problem when using octave sparse matrix. max(speye(65536)(:)) will result in a 0x0 variable. However, speye(65535) and speye(65537) works. How that happens? My octave version is 3.2.4 in Fedo

For example , M is a sparse Matrix , and track_list is the colnames of the matrix. library(Matrix) M <- Matrix(0,nrow = 3,ncol = 4) M[1,2] = 1 M[2,3] = 1 M[3,2] = 1 track_list = c('a','b','c','d')

In numpy if you want to calculate the sinus of each entry of a matrix (elementise) then a = numpy.arange(0,27,3).reshape(3,3) numpy.sin(a) will get the job done! If you want the power let's say to 2

In the context of a finite element problem, I have a 12800x12800 sparse matrix. I'm trying to solve the linear system just using MATLAB's \ operator to solve and I get an out of memory error using mld

I'm writing a sparse matrix solver using the Gauss-Seidel method. By profiling, I've determined that about half of my program's time is spent inside the solver. The performance-critical part is as fol

Over the course of finite difference discretization of an elliptic equation and application of Neumann BCs on all sides of the 2D domain I have a large sparse matrix. I need to find null space of its

I have sparse CSR matrices (from a product of two sparse vector) and I want to convert each matrix to a flat vector. Indeed, I want to avoid using any dense representation or iterating over indexes. S

I am implementing a sparse matrix based on the Stack class, and I'm getting the following error: Sparse.java:6: Sparse is not abstract and does not override abstract method pop() in Stack public clas

Question one: Are there specialized databases to store dense and sparse matrices ? I googled but didn't find any... The matrix in question is huge (10^5 by 10^5) but it's sparse, which means that most

I have a fairly large sparse matrix that, I reckon, would occupy 1Gb when loaded into memory. I don't need access to the whole matrix at all times, so some kind of memory mapping would work; it doesn'

I'm trying to write a matrix to sparse matrix conversion program. But, there is a problem in my scan_matrix function.I couldn't figure out where is the problem.According to Nemiver(debugging program)

I have a matrix with 21492 columns and 8532 rows. i am running flexmix package to calculate Gaussian univariate distributions. When i run the following commands the process will crash due to lack of m

I have a large sparse matrix, implemented as a lil sparse matrix from sci-py. I just want a statistic for how sparse the matrix is once populated. Is there a method to find out this?

I have a sparse matrix: from scipy import sparse a = sparse.diags([1,4,9],[-1,0,1],shape =(10,10),format =csr) I want to take the square root of each of the elements in the sparse matrix I look up

How do you populate an empty matrix with the values of another matrix? The empty matrix: > m1 <- matrix(ncol=8, nrow=8) > rownames(m1) <- c('a','b','c','d','e','f','g','h') > colnames(m

I have an RDD as such: byUserHour: org.apache.spark.rdd.RDD[(String, String, Int)] I would like to create a sparse matrix of the data for calculations like median, mean, etc. The RDD contains the row_

I've got a sparse Matrix in R that's apparently too big for me to run as.matrix() on (though it's not super-huge either). The as.matrix() call in question is inside the svd() function, so I'm wonderin

We have an application that stores a sparse matrix. This matrix has entries that mostly exist around the main diagonal of the matrix. I was wondering if there were any efficient algorithms (or existin

I am filling a sparse matrix P (230k,290k) with values coming from a text file which I read line by line, here is the (simplified) code while ... C = textscan(text_line,'%d','delimiter',',','EmptyValu

I have a sparse matrix W, when I use linalg.pinv(W), it throws some errors: Traceback (most recent call last): File /Users/ad9075/PycharmProjects/bednmf/test.py, line 14, in testNmfRun self.factor =

Question: How can I split 1 sparse matrix into 2, based on the values in a list? That is, I have a sparse matrix X: >>print type(X) <class 'scipy.sparse.csr.csr_matrix'> that I visualize

I am trying to do some k-means clustering on a very large matrix. The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of 1 values per row). The whole thing does not f

I have a question pertaining to R. I have some sequentially numbered matrices (all of the same dimensions) and I want to search them all and produce a final matrix that contains (for each matrix elem

I don`t know how to solve this problem in Fundamentals of data structure in C ed.2nd ch2.5 On a computer with w bits per word, how much storage is needed to represent a sparse matrix, A, with t nonzer

I'm trying to manipulate some data in a sparse matrix. Once I've created one, how do I add / alter / update values in it? This seems very basic, but I can't find it in the documentation for the sparse

I have a NxM matrix and I want to compute the NxN matrix of Euclidean distances between the M points. In my problem, N is about 100,000. As I plan to use this matrix for a k-nearest neighbor algorithm

I'm using the bigmemory package. I want to calculate w. My v length is 478000 and k length is 240500. The two matrix multiplication is w very large. I run the code by loop, but it still is running an

When I use large sparse matrix, it's better to use compressed matrix like CCS, CRS and so on. I tried to use ScalaNLP, la4j, colc to calc 100,000*100,000 sparse matrix. There are some problems. Breez

In another post regarding resizing of a sparse matrix in SciPy the accepted answer works when more rows or columns are to be added, using scipy.sparse.vstack or hstack, respectively. In SciPy 0.12 the

consider the code below, Am trying to make a Sparse_matrix that contains an array of pointer. In struct Sparse_matrix, Is this a proper way to create a pointer array when size is unknown? What's the

In SparseSuiteQR, all of the examples I can find use stdin or a file read to create a sparse matrix. Could someone provide a simple example of how to create one directly in C++? Even better, in the CH

I'm wondering whether there is a way to simply add a dense vector to all the rows of a sparse matrix represented as a csr_matrixin scipy.sparse and returning a sparse matrix, ie trying to sum only the

I am looking for sparse matrix representation that allow for efficient row and column swaping. The classic representation (by compressed row,compressed column or triplets) seems to only allow to perfo

I'm trying to find a way to subtract a column of a scipy.sparse matrix from a numpy vector but I can't seem to find a way to do it without changing the shape of the vector. This is what I have so far:

For the purpose I used the solution from that thread link by now, however it gives memory error as expected since my matrix A size is 6 million to 40000 matrix. Therefore I am looking for any other so

I wanted to repeat the rows of a scipy csr sparse matrix, but when I tried to call numpy's repeat method, it simply treats the sparse matrix like an object, and would only repeat it as an object in an

This question has two parts (maybe one solution?): Sample vectors from a sparse matrix: Is there an easy way to sample vectors from a sparse matrix? When I'm trying to sample lines using random.sample