The model of distributed calculators makes it possible a parallel calculation of the correlated Nparticle system with a complex multiparticle interaction (longrange ionic and shortrange repulsive, two and threeparticle covalent interactions) with MPI and CUDA technologies. The computational model is based on the mathematical model of heterogeneous descriptors developed by the authors, that allows shift the focus from the describing the physical interactions in the system to the description of data flow between the descriptors. The results of computer experiments, which compare the time of the simulation on the cluster of 16 calculators and GPU NVIDIA are given. The model of distributed calculators was being tested with the software package of RIS «MDSLAGMELT»[1].
Nowadays, computer modeling (CM) is widely used in various fields of modern science. In particular, in physical chemistry we can study the properties and structure of materials and their relationships.
Moleculardynamic (MD), MonteCarlo and quantumchemical methods are applied there and allow to define different sets of properties. The moleculardynamics method allows to define the whole complex of properties (structural, thermodynamic, transport) and to investigate the interrelations of nanostructure and physicalchemical properties [1–5].
The size of the simulated system for MD modeling is extremely important. A significant increase in the size of the system provides the practical relevance of the results. The calculation of the systems with 105–107 particles requires a large amount of time and computer resources and it makes impossible carrying out CM without highperformance computing [6, 7]. To solve this problem, the authors have developed a model of distributed calculators based on distributed computing methods for correlated Nparticle system [8–10].
Physical phenomena that are adequately described by the classical and quasiclassical theory can be simulated (using models of particles) by molecular dynamics method. The term «model of particles» is the general one for a class of computing models in which the discrete description of the physical phenomena includes cooperating particles. Each modeling particle has a set of constant and variable attributes.
In this case moleculardynamic simulation represents the numerical solution of the Cauchy’s boundary task, which means that the initial system state in a bounded region of space (calculation area) is specified at the time t = 0 and the boundary conditions are reserved on it. Modeling is tracking the time evolution of this configuration. The main part of calculation is the cycle on a time step in which the state of physical system changes on time for a small step Δt.
The current condition of the physical system is defined by the attributes of the final ensemble of particles, and the evolution of the system is defined by the interaction laws of these particles. The most of the moleculardynamics systems relates to the class of longrange potentials, or considering only shortrange covalent interactions.
The subject of this work is an investigation of the polymerizing systems with multiparticle interactions which means uniting some types of interactions – twoparticle contribution (longrange ionic and shortrange repulsive) and multiparticle ones (two and threeparticle covalent interactions). The description of this class of models is given in Table 1 [10].
The ionic model is a part of ioniccovalent model though for modeling ionic connections it can be used only independently. In the ionic model (IM) potential functions are built for the ion system. The IM includes models of shortrange repulsive and longrange Coulomb interaction. The last one is calculated by the Ewald method or the fast multipole method.
The ioniccovalent model is more realistic to describe the interactions in the polymerizing systems. The model is based on the polymer theory of the slags, which suggests the presence of small nondestructive structural groups (elementary structure groups – ESG), which form the structural grid in the polymerizing melts.
Table 1
The interparticle interaction models are used in the information research system «MDSlagMelt» [10]
The Ionic model (IM) 

The Shortrange model 
The Longrange model 
The Ioniccovalent model (ICM) 

– ICM of ESG (elementary structure group) 

– ICM of BI (bridge bonds) 

– ICME (environmental effect) 

The techniques covered net of covalent bonds 
The ICM unites the ionic model, the models of two and threeparticle covalent interaction in the elementary structure groups, the threeparticle covalent interaction for bridge bonds, the environmental effects for the network forming atomsoxygen bounds inside the ESG from the nearest neighbors outside ESG.
The features of the above mentioned physicochemical models require the development of a mathematical model that allows realizing highperformance computing for correlated systems with complex interactions containing 105–107 particles.
For solution the authors have developed a model of heterogeneous descriptors for distributed MD simulation of the correlated system of Nparticles.
The basic elements of the model are the object and the descriptor, which provide the possibility of calculations distribution without detail specification of all interactions between the particles
We define an object as a set of particles descriptions and relations between them for the system in the initial state, which is built by specific rules and provides the possibility of the system decomposition for distributing and parallel calculations.
The objects are initialized by means of heterogeneous descriptors which contain polytypic elements of the description of the allocated object which is necessary for the distribution of calculations.
Thus, the system is a set of the objects characterized by heterogeneous descriptors the calculation of which can be distributed on separate calculators, combining results on a certain scheme.
On the basis of physicochemical models and the analysis of the program code of a local MD application, the authors have constructed a set of descriptors allowing to share the application code into independent blocks between calculators. The descriptors can be divided into two classes: oneparticle descriptors and aggregators (two – and threeparticle descriptors).
Both classes of descriptors include the possibility of parallel computing on different calculators. However, if the oneparticle descriptors can be distributed on the calculators independently, then the aggregators (containing the elements describing the cross relations of different orders between oneparticle descriptors and/or aggregators) are «dependant» on the oneparticle descriptor. The aggregators are computed on the same computer with a «parental» oneparticle descriptor.
The notation of local moleculardynamic model in model of descriptors for distribution MD simulating is given in Table 2.
Thus, the set of objects and their relationship between them are devided into different types of subset according to heterogeneous descriptor.
More detailed characteristic of descriptors elements are given below.
{D1(i)} – the class of oneparticle descriptors assuming distribution. It is expedient to divide this descriptor into two subset.
{D1s(i)} – a subset of the descriptors, containing constant scalar attributes of particles. Here D1s(i) = <I, σ q, m, tip,>, where i – particle number, tip – particle type, m – weight, q – charge, σ – rigid sphere radius.
{D1v(i)} – a subset of the descriptors containing variable vector attributes of particles. Here
where – radius vector; – speed; – acceleration; – full force effecting on an iparticle at a given timestep.
{D∑(i)} – the class of heterogeneous descriptors providing creation of different orders relations between descriptors of different types. Some elements of the descriptors in the class are the storage, so these descriptors are called aggregators.
Three types of subset aggregators are defined by the authors.
1. {D∑2(i)} – a subset of the twoparticle aggregators containing scalar and vector elements. They provide the calculation of cross relations, on the basis of oneparticle descriptors, and also contain scalar and vector stores. The aggregators depend on oneparticle descriptors (called «brunches»). Data for calculation are received from «parental» descriptor. The aggregator realizes consequent calculation of all pair relations between fixed i object and others to form storages.
Table 2
Distributed descriptor model of moleculardynamic modeling Nparticle correlation system
Classes of descriptors and different types of descriptors in the class 
Type of descriptors 
Type of elements 
distribution 
D1(i) 
oneparticle : 
attributes of particles : 
independent 
D1s(i) = <i, tip, m, q, σ> 
scalar 
constant 


vector 
variable 

D∑(i) 
aggregators : 
cross relationship between descriptors 
depends on «parental» descriptor 

twoparticle 
{D1(i)}⋂{D1(j)} 


threeparticle 
{D1(i)}⋂{D1(j)} {D1(k)} 

D(Property) = <T, P, E, C,γV, αP,bT, χa, msda, h, x, l> 
multiparticle (macroscopic properties) 
{D1(i)}⋂ D∑2(i)⋂ D∑3(i) 
not distributed 
Here
,
where i, j – particles numbers; – force acting between i and j particles; φ2 – twoparticle potential; – the vector storage including different types of contributions to twoparticle force effecting on an iparticle; Σφ2 – the scalar storage containing values of different power contributions to total twoparticle energy.
2. {D∑3(i)} – a subset of threeparticle aggregators which build third order cross relations on the basis of oneparticle descriptors. Each aggregator contains the variable vector and scalar elements which are calculated on special algorithms, providing formation of limited selections of particles (elementary groups) from all range of numbers. The aggregators calculate the relations between triplets of particles within these selections. Like twoparticle aggregators it contains scalar and vector stores being brunches in relation to a pair of onepartial descriptors.
Here
,
where i, j, k – triplets of particles from elementary groups; – threeparticle interaction; φ3 – three particle potential; – the vector storage including different types of contributions to the three particle force acting on an iparticle; Σφ3 – the scalar storage containing values of different power contributions to total three particle energy.
3. {D(Property)} – aggregators subset containing elements, describing macroscopic properties of the system, based on two particle and three particle aggregators. Here
D (Property) = <T, P, E, C, γV, αP, ,bT, χa, msda, h, x, l>,
where a – type of particles; T – temperature; P – pressure; the С – heat capacity; E – energy; γV – coefficient of thermal pressure; αP – coefficient of thermal expansion, bT – isothermal compressibility; χa – velocity autocorrelation function; msda – mean square displacement; Da – diffusion coefficient; h, x, l – viscosity factor, coefficient of conductivity and thermal conductivity. These aggregators cannot be distributed.
The method of heterogeneous descriptors make it possible to transfer the accent from physical interactional in the system to information description of data flows redistribution between descriptors and use this approach to form computing model of redistributed calculation on the bases of heterogeneous descriptors.
The model of distributed calculators is based on a masterslave methodology. The master controls the slaves, separates a set of particles into subsets which are transferred to the slaves for calculation. The master receives data from the slaves and processes the results. The slaves calculate the Nparticle system on the CPU or the GPU, using MPI and CUDA technologies.
The model of distributed calculators is given in Fig. 1.
The model assumes the implementation of distributed computing a heterogeneous computing environment. The calculators have different performance and type, such as the GPU and CPU [11].
On the first step of modeling the all set of particles is transmitted to every calculator. The master generates a subset of particles that will be transferred to the calculators for processing. At run time of work, each slave receives data from the master about the calculated subset of the particle. The slave provides interactions calculation describing descriptors D∑2(i) и D∑3(i), associated with the i particle, according to a given scheme. When each slave completes the calculation, it passes the results to master which gathers and processes data.
Fig. 1. The model of distributed calculators using the CUDA and MPI technologies
The most pressing is to ensure parallel calculation of objects characterized by the descriptors D∑2(i), since the implementation of this part simulation has the largest computating time (a quadratic dependence on the number of particles N2). Flowchart of the calculators work based on the model of heterogeneous descriptors is given in Fig. 2.
Fig. 2. Flowchart of the calculators work based on the model of heterogeneous descriptors
The formal description of parallel calculation correlated Nparticle system using CUDA and MPI, is given in Fig. 3 and 4 in the form of pseudocode.
To make parallel calculation of correlated Nparticle system using the technology MPI calculators are placed in communication field of MPI_COMM_WORLD. Each calculator is given a unique number, stored in the variable rank. Data on the calculators quantity are stored in a variable size.
MPI_BCAST function (a variable containing the information of the quantity of descriptors in a subset, quantity of values containing in the variable, variable type, transmitter, communication area, error number) sends to each calculator the number of descriptors for further combining. The calculators independently determine their own subset of the descriptors, using variables StrSml and FshSml. The variables correspond to the start and final value of the subset descriptors and set computing field for calculator.
Fig. 3. Pseudocode parallel calculation of correlated Nparticle system with MPI technology
After completion of computing calculators call the function MPI_GATHER (new values of elements of a subset descriptors {Dlv(i)}, number of values, data type, variable for the gather subsets of descriptors on the master, variable contains number of descriptor elements, variable type, communication area, error number) that sends a subset of the oneparticle descriptors {Dlv(i)}to the master for their combining.
The calculation of the correlated Nparticle system at CUDA is divided into two parts. In sequential part there are set of variables(memory GPU). In the GPU memory values of the elements of oneparticle descriptors are placed. The subsets of descriptors distributed between processor threads and when the parallel calculation is completed the new values of the elements {Dlv(i)} are combined in RAM on the master.
Descriptions are distributed between threads in parallel part. Then each thread conducts its independent calculation and stores the data in memory GPU.
To estimate the efficiency of the distributed calculators model a number of experiments are carried out. The spared time for modeling using MPI and CUDA technologies are compared. Descriptors are filled with test values of the elements. For the model using MPI technology organized cluster containing 16 calculators. Testing of the model using CUDA technology was carried out on the processor GForce GTS 450. The results of computer simulations to model distributed calculators are given in Table 3.
According to the results of the simulation the least time is observed at the calculation of the system GPU GTS 450. At the calculation using MPI technology it is not possible to achieve the acceleration equal to the number of processors because frequent calls to slow speed RAM. In the code on the GPU provides accommodation of variables in a quick shared and constant memory, which reduces the simulation time.
The model of heterogeneous descriptors allows to create a computing model, which is used MPI and CUDA technology for GPU and CPU and allows to obtain experimental results of practical value.
Fig. 4. Pseudocode parallel calculation of correlated Nparticle system with CUDA technology
Table 3
The results of computer simulations to model distributed calculators
Number particle 
Local version 
MPI 
CUDA 

P = 1 
P = 2 
P = 8 
P = 16 
P = 1 

Calculation time in seconds 

50176 
7,6 
5,91 
3,72 
2,31 
0,14 
250880 
180,4 
131,93 
88,45 
54,68 
3,53 
401408 
548,2 
421,64 
282,47 
156,34 
9,11 
The model of distributed calculators was being tested win the software package of RIS «MDSLAGMELT» [1], [12].
The study was supported by The Ministry of education and science of Russian Federation, project 14.132.21.1792.
The work was submitted to the International Scientific Conference «Modern high technologies» SpainFrance, July 27  on August 3, 2013, came to the editorial office оn 24.07.2013.
Библиографическая ссылка
Voronova L.I., Voronova L.I., Trunov A.S., Voronov V.I., Voronov V.I. THE DISTRIBUTED CALCULATORS MODEL FOR MOLECULARDYNAMIC SIMULATION OF STRONG INTERACTION SYSTEMS // Международный журнал экспериментального образования. – 2013. – № 12. – С. 8288;URL: http://expeducation.ru/ru/article/view?id=4301 (дата обращения: 10.08.2020).