Fortran - Efficient arrays for artificial neural networks?

Author

Efficient arrays for artificial neural networks?

sk8terg1rl

2007-03-16, 8:19 am

Hi group,

I've been toying with the idea of spending some upcoming free time of
mine this summer writing my own image recognition program, based
primarily on neural networks.

I've already written a simple one that counts the number of 1's & 0's
in a binary string. After a bit of training, it gets it right most of
the time. It's learning technique is simple - it starts off with a
fixed topology (number of layers, neurons per layer) and randomly
changes weights until the error is minimised.

In the past I stored the neural data (number of neurons per layer,
which is connected to which, interconnect weights) by having a 3D cube
array {x;y;z} where x = layer number, y = number of neurons in that
layer, z = neuron number it is connected to followed by its connection
weight.

So {1; 5; 10,0.1,11,0.5} is a neuron in layer 1, index number 5,
connected to neurons 10 with weight 0.1 and 11 with weight 0.5

This method is obviously unwieldy as the array will be very sparse.
Once the network has to start processing kilo-megabytes (instead of
tens of bytes worth of bit strings) of input data, how the network is
put on memory will count a lot.

I'm not a very experienced programmer, ANNs are just a "hobby" of mine
and I'm not a genius, so please go easy with the technical
explanations :-)

I'm looking for a storage scheme that:
(1) Stores the data efficiently with minimal memory wastage.
(2) Allows the layer and interconnect densities to grow/shrink as
necessary. This makes me think continuous array (de)allocation is
necessary.
(3) Allows an input signal to be quickly filtered through the network
(i.e. no searching or sorting necessary to find out which neuron
connects to which other one)
(4) Can be translated easily into a 1D vector.

Thanks group :-)
skate x

Beliavsky

2007-03-16, 7:07 pm

On Mar 16, 6:32 am, "sk8terg1rl" <sk8terg1rl_2...@yahoo.co.uk> wrote:
> Hi group,
>
> I've been toying with the idea of spending some upcoming free time of
> mine this summer writing my own image recognition program, based
> primarily on neural networks.
>
> I've already written a simple one that counts the number of 1's & 0's
> in a binary string. After a bit of training, it gets it right most of
> the time. It's learning technique is simple - it starts off with a
> fixed topology (number of layers, neurons per layer) and randomly
> changes weights until the error is minimised.
>
> In the past I stored the neural data (number of neurons per layer,
> which is connected to which, interconnect weights) by having a 3D cube
> array {x;y;z} where x = layer number, y = number of neurons in that
> layer, z = neuron number it is connected to followed by its connection
> weight.
>
> So {1; 5; 10,0.1,11,0.5} is a neuron in layer 1, index number 5,
> connected to neurons 10 with weight 0.1 and 11 with weight 0.5
>
> This method is obviously unwieldy as the array will be very sparse.
> Once the network has to start processing kilo-megabytes (instead of
> tens of bytes worth of bit strings) of input data, how the network is
> put on memory will count a lot.
>
> I'm not a very experienced programmer, ANNs are just a "hobby" of mine
> and I'm not a genius, so please go easy with the technical
> explanations :-)
>
> I'm looking for a storage scheme that:
> (1) Stores the data efficiently with minimal memory wastage.

You can store "jagged" 2-D arrays using arrays of derived types, with
each derived type having an allocatable component (or pointer
component in strict Fortran 95). A jagged 3-D array is a 1-D array of
jagged 2-D arrays. Below is an example of allocating a jagged 3-D
array. There may be more efficient approaches.

module sparse_tensor_mod
! manipulate "jagged" matrices or tensors
implicit none
private
public :: vector,vec,mat,tensor,set_mat,alloc_tens
or
type vec
integer, allocatable :: ii(:)
end type vec
type mat
type(vec), allocatable :: ivec(:)
end type mat
type tensor
type(mat), allocatable :: imat(:)
end type tensor
contains
function vector(ivec) result(xx)
integer, intent(in) :: ivec(:)
type(vec) :: xx
allocate (xx%ii(size(ivec)))
xx%ii = ivec
end function vector
!
elemental function num_mat(xx) result(n)
! return the # of matrices in tensor xx
type(tensor), intent(in) :: xx
integer :: n
n = size(xx%imat)
end function num_mat
!
elemental function num_rows(xx) result(nrows)
! return the # of rows in matrix xx
type(mat), intent(in) :: xx
integer :: nrows
nrows = size(xx%ivec)
end function num_rows
!
elemental function get_val(itens,i1,i2,i3) result(j)
! return element (i1,i2,i3) of tensor itens
type(tensor), intent(in) :: itens
integer , intent(in) :: i1,i2,i3
integer :: j
j = itens%imat(i1)%ivec(i2)%ii(i3)
end function get_val
!
subroutine set_row(irow,ivec,xx)
! set row irow of matrix xx to ivec
integer , intent(in) :: irow,ivec(:)
type(mat), intent(in out) :: xx
integer :: n
n = size(ivec)
if (allocated(xx%ivec(irow)%ii)) deallocate (xx%ivec(irow)%ii)
allocate (xx%ivec(irow)%ii(n))
xx%ivec(irow)%ii = ivec
end subroutine set_row
!
subroutine set_mat(vec_vec,xx,display)
! set matrix elements to values in array of vectors vec_vec
type(vec), intent(in) :: vec_vec(:)
type(mat), intent(out) :: xx
logical , intent(in), optional :: display
integer :: i,nvec
nvec = size(vec_vec)
call alloc_mat_rows(nvec,xx)
do i=1,nvec
call set_row(i,vec_vec(i)%ii,xx)
end do
if (present(display)) then
if (display) call display_mat(xx)
end if
end subroutine set_mat
!
function matrix(vec_vec) result(xx)
type(vec), intent(in) :: vec_vec(:)
type(mat) :: xx
call set_mat(vec_vec,xx)
end function matrix
!
subroutine alloc_mat_rows(nrows,xx)
integer , intent(in) :: nrows
type(mat), intent(out) :: xx
allocate (xx%ivec(nrows))
end subroutine alloc_mat_rows
!
subroutine alloc_mat_col(ncol,idef,xx)
integer , intent(in) :: ncol(:)
type(mat), intent(in out) :: xx
integer , intent(in) :: idef
integer :: irow,nrows
nrows = size(ncol)
call alloc_mat_rows(nrows,xx)
do irow=1,nrows
allocate (xx%ivec(irow)%ii(ncol(irow)))
xx%ivec(irow)%ii = idef
end do
end subroutine alloc_mat_col
!
subroutine display_mat(xx)
type(mat), intent(in) :: xx
integer :: i
do i=1,num_rows(xx)
write (*,*) xx%ivec(i)%ii
end do
end subroutine display_mat
!
subroutine alloc_tensor(xmat,idef,xtens,display)
type(mat) , intent(in) :: xmat
integer , intent(in) :: idef
type(tensor), intent(out) :: xtens
logical , intent(in), optional :: display
integer :: i,nmat
nmat = num_rows(xmat)
allocate (xtens%imat(nmat))
do i=1,nmat
call alloc_mat_col(xmat%ivec(i)%ii,idef,xtens
%imat(i))
end do
if (present(display)) then
if (display) call display_tensor(xtens)
end if
end subroutine alloc_tensor
!
subroutine display_tensor(xx)
type(tensor), intent(in) :: xx
integer :: i
do i=1,num_mat(xx)
write (*,"(/,1x,'matrix',1x,i0)") i
call display_mat(xx%imat(i))
end do
end subroutine display_tensor
end module sparse_tensor_mod

program xsparse_tensor
! driver for sparse_tensor_mod
use sparse_tensor_mod, only: vector,mat,tensor,set_mat,alloc_tensor
implicit none
integer, parameter :: idef = 0
type(mat) :: dim_rows
type(tensor) :: xtens
call set_mat([vector([3,1]),vector([2,1,4])],
dim_rows,display=.true.)
call alloc_tensor(dim_rows,idef,xtens,display
=.true.)
end program xsparse_tensor

output with g95 or gfortran (modulo spacing):

2 1
2 1 4

matrix 1
0 0
0

matrix 2
0 0
0
0 0 0 0

sk8terg1rl

2007-03-19, 7:13 pm

Thanks Beliavsky. I didn't know about jagged arrays :-)

Jan Vorbrüggen

2007-03-23, 4:05 am

Look in the archives of the comp.* newsgroups - in particular comp.benchmarks
- for a discussion of the optimization the Sun C compiler performs on the SPEC
CPU2000 benchmark art. This optimization is precisely about modifying the data
structures of a neural network simulation in such a way that it runs much
faster. It must be admitted, though, that the original code was more or less
pessimal, and that it was a quite a feat, given the restrictions of the C
standard, to cut into thin slices and re-assemble the art code such that it
actually works within the constraints.

Jan

Sponsored Links