(using `MATLAB`

) I have a large coordinate matrix and a large sparse adjacency matrix for which coordinates are connected to each other. I had asked previously on SO, how to *efficiently* compute these distances in this SO question, but I have run into **memory problems** now which are a much more severe issue.

I used this MATLAB function to compute the distance matrix `Dists = pdist2(fruchterman_graph(:,:),fruchterman_graph(:,:),'euclidean');`

but it fails on large networks for both speed and finally memory.

This is the code which runs only on smallish graphs (not hundreds of thousands):

```
coordinate = kamada_graph;
[n, d] = size(kamada_graph);
assert(d == 2);
resi = sparse(adj* spdiags((1:n)',0,n,n));
resj = sparse(spdiags((1:n)',0,n,n) * adj);
res = sparse(n,n);
f = find(adj);
res(f) = sqrt((coordinate(resi(f), 1) - coordinate(resj(f), 1)) .^ 2 +...
(coordinate(resi(f), 2) - coordinate(resj(f), 2)) .^ 2);
```

This on *large* graphs creates **??? Error using ==> find Matrix is too large to return linear indices. Use [i,j] = find(S) for sparse matrix. Error in ==> modularize_graphs at 49 [f] = find(adj);**

I changed the line which is referred to to be:

[i,j] = find(ajd);

res(i,j) = sqrt((coordinate(resi(i,j), 1) - coordinate(resj(i,j), 1)) .^ 2 +... (coordinate(resi(i,j), 2) - coordinate(resj(i,j), 2)) .^ 2);

and on the smallish network now (~500 vertices) the error: **??? Out of memory. Type HELP MEMORY for your options. Error in ==> modularize_graphs at 50 res(i,j) = sqrt((coordinate(resi(i,j), 1) - coordinate(resj(i,j), 1)) .^ 2 +...**

Is there anyway to compute the distance matrix using the adjacency matrix and a coordinate matrix, (N,2) of x,y values, without falling into memory issues and possibly making it not too slow as well?

The desired output is a matrix of distances, the distances between all points that are connected according to the `Adj`

adjacency matrix.

To calculate pointwise distances with using a minimum amount of memory, you can always iterate through your adjacency matrix on an element-wise basis:

```
%# assuming a square and symmetric adjacency matrix
nCoords = size(adjMat,1);
%# there are at most (n^2-n) distances
%# if this runs out of memory already, there
%# is a way to only store existing distances to save
%# even more memory
distances = NaN((nCoords^2-nCoords)/2,1);
ct = 0;
for row = 1:nCoords
for col = 1:row-1
ct = ct+1;
if adjacencyMatrix(row,col)
distances(ct) = sum( (coordinate(row,:)-coordinate(col,:)).^2 );
end
end
end
distances = sqrt(distances);
```

With the sparse approach, you may want to try the following (I don't think you need `resi`

and `resj`

, unless I totally misunderstand your problem).

```
[row,col]=find(adjacencyMatrix);
distances = sqrt(sum( (coordinate(row,:) - coordinate(col,:)).^2 ,2));
sparseDistanceMatrix = sparse(row,col,distances);
```

Similar Questions

I am looking at dimensioning a large cluster (10k cores) that needs to support both compute bound deep analytics as well as I/O bound big data, and I want to hear from some folks that have built a big

I have large arrays which I am doing fairly simple linear algebra on. I have achieved good speed ups by vectorising the operations but i want to know how MATLAB treats the subarrays. I pre-allocate ar

I am getting 3 camera input to MATLAB with Infinite Trigger. But after minutes, cameras fill up my memory, and I have to restart MATLAB. How can I deal with this memory problem?

I'm a newbie to Matlab and just stumped how to do a simple task that can be easily performed in excel. I'm simply trying to get the percent change between cells in a matrix. I would like to create a f

Below is my code for calculating Euclidean distance between vectors, and a snippet of my transformed data set (vectors). import itertools import numpy as np vect=[[2, 1, 1, 1, 1, 3, 4, 2, 5, 1], [1, 5

I'm new to Matlab processing, and I would like to read and process a large video (more than 200k frames) inside a for loop (or without it). In particular, i would like to: read the video with Video

I've inherited 3 SBS (now GE Fanuc) C2k single board computers connected over a CompactPCI backplane, and need to get shared memory up and running between them. My understanding is that in previous ye

I'm having a brain lapse, but I just can't get this to work. I have an array of distances: import numpy as np zvals = np.linspace(-5,5,10) d = np.array([(0,0,z) for z in zvals]) I want to compute the

This question already has an answer here: MATLAB, Filling in the area between two sets of data, lines in one figure 4 answers I'm trying (without success so far) to colorize or to shade the sur

I have around 644 nodes in my graph database(Neo4j) . I need to compute distances between all these 644 nodes and display it visually in the GUI. I want to pre-compute and store the distances between

Two matrices, A and B: A = [1 2 3 9 7 5 4 9 4 1 4 7] B = [1 2 3 1 4 7] All rows of matrix B are members of matrix A. I wish to delete the common rows of A and B from A without sorting. I have tried s

<a id=clickme>here</a> when I click on the link,I want to get the coordinate where click happens. how to do that with jQuery?

I came across an interesting question in my textbook, but not further answer or details were supplied :( Given some points, A, B, C etc and some distance relationships between those points: A -> B

I am running VBA code on a large spreadsheet. How do I clear the memory between procedures/calls to prevent an out of memory issue occurring? Thanks

Is it possible to compute the numerical hessian matrix for this function with respect to W_i,C, epsilon_i easily Matlab? I have computed a hessian by manually take a derivative, but I want to verify i

I have a file with ~2 billion lines of text (~200gigs). I want to produce a new file containing the same text lines, but shuffled randomly by line. I can't hold all the data in memory. Is there a good

I was wondering if there is already a framework or a jquery plugin which i can use to add constraints to form fields. With constraints in this case I don't want to say that field x is an e-mail fiel

I have a large number of coordinate points and I would like to find the distance from start to finish. I can manage to find the distance between the first and last point, but when there is a circular

I'm trying to write a small DSL parser using fslex and fsyacc. The input is composed of interleaving chunks of two different languages which require different lexing rules. How do I write my fslex fil

If memory were not scarce, how would you implement a sort in a language with libraries for representing and sorting sets

I'd like to sync a large list of items between the client and the server. Since the list is pretty big I can't sync it in a single request so, how can I ensure the list to be synched with a reasonable

Excuse the very simple/stupid question, but... Say I have the following (in matlab): x_A = rand(1,30); y_A = rand(1,30); x_B = rand(1,40); y_B = rand(1,40); scatter(x_A,y_A,'r') hold on scatter(x_B,y_

I'm working on a Java program that take an input file containing a bunch of depencencies between tasks like this : C --> D A --> B A --> D F --> G B --> C E --> F C --> D mean

In numpy, what's the most efficient way to compute x.T * x, where x is a large (200,000 x 1000) dense float32 matrix and .T is the transpose operator? For the avoidance of doubt, the result is 1000 x

I'm trying to compute the Fourier coefficients for a waveform using MATLAB. The coefficients can be computed using the following formulas: T is chosen to be 1 which gives omega = 2pi. However I'm ha

The R function dist computes and returns the distance matrix computed by using the specified distance measure to compute the distances between the rows of a data matrix. However, I want the distance

Why does the code below cause Matlab to keep increasing the amount of memory used? Nothing is being stored from one iteration to the next? Yet Matlab keeps using up my system memory until the whole ma

I am trying to write something that will determine the distance between to sets of lat/lon coordinates. I am using the following code which I found on this site: public static double distance (double

How can I compute a base 2 logarithm without using the built-in math functions in C#? I use Math.Log and BigInteger.Log repeatedly in an application millions of times and it becomes painfully slow. I

When parsing a large file I get the following error Caught: java.lang.OutOfMemoryError: Java heap space How do you parse large files in Groovy without exceeding heap size? example code that fails with

How do you establish a constraint where, one column (not the primary key) has to have the same value as another table's column. I'm not quite sure how to phrase it so here's an example: Ex: I have thr

I am working on matlab deploytool which generates a .Net dll. When I call the function from the dll it crashes because out of memory exception. It runs correctly without any exceptions in matlab. How

Using physical engine Box2d C++, how to check the possibility of moving in a straight line a body to coordinate without collision (that is, how to know whether the obstacles to the body to coordinate)

I'm trying to find a factor using matlab that requires me to compute the Fourier transform of an input signal. The problem was stated to me this way: fbin = 50HZ 0 <= n <= 1999 alpha = F {Blackm

I'm running SQL 2008 R2 64 Bit on a Windows 2008 64 bit OS server. The server has 7.5 GB RAM. SQL Management Studio, connected to server, right click on server > Properties > Memory - Use AWE i

How can I find the minimum y coordinate for elliptical curve y^2 = x^3 + ax + b on a finite field F(p) in SAGE where a and b are large about the order of 10^15 and the integer p is very large about th

I am trying to figure out how to compute large powers of huge numbers in matlab to do RSA encryption. For example: A 50+ digit integer raised to the power of 999999.

I have a text file containing a large number of rows of data with columns separated by spaces. How can I read in this data with MATLAB? I have tried the following code without success: fid = fopen('fi

This question already has an answer here: What is the best way get the symmetric difference between two sets in java? 5 answers I have 2 sets s1 and s2 one condition is s1 > s2. And my requi

I need to handle massive (tens of millions) of MATLAB structs; I needed a dozen or so fields so I reckoned memory won't be an issue, until I discovered this ( explanation ) >> s=[]; >> s.f

Can I add paragraphs with SharpPDF, without having to specify the exact coordinates? Can't I just place paragraphs one under the other? Please tell me if you used the library.

I need to verify if a coordinate LAT/LNG point is between other two points (a segment line like a road). I followed THIS topic with no success. function inRange(start, point, end) { start_x = start.la

Rather than ask how to plot big data sets, I want to wrap plot so that code that produces a lot of plots doesn't get hammered when it is plotting a large object. How can I wrap plot with a very simple

I have a quick graph theory question. I have a 13 x 13 adjacency matrix in Matlab. I've already taken just the lower diagonal (this is an undirected graph) and subtracted off the identity matrix (so

Given two sets of keywords, where each keyword got start and end offset (e.g. keyword abc starts at offset 23 and ends at offset 25), I would like to efficiently find matching pairs between those se

How can I comment a group of lines without commenting each line? ie like 'c' /* printf(hello); printf(there); */ in Matlab the only way I know to do this is to comment each line %disp('hello') %d

At the moment I am dealing with large amounts of float/double datasets to be used for calculation. I have a set of files to compare Data A to Data B and I would like to compute the Euclidean distance

I'm trying to deal with an Excel file which comprises 1 million rows. However, when I open it in MATLAB, only 10,000 rows are displayed.... Could anyone tell me how to import full data using MATLAB?

At the moment i am using the pdist function in Matlab, to calculate the euclidian distances between various points in a three dimensional cartesian system. I'm doing this because i want to know which

How to display an error when the uploaded files are exceeding post_max_size php? print_r($_FILES); I get an empty array when I have exceeded the post_max_size array() I got this from php.net but I d