Sierra Toolkit  Version of the Day
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
Sierra Toolkit Rebalance Use Cases
Collaboration diagram for Sierra Toolkit Rebalance Use Cases:

Use Case 1: Unequal element weights

This use case demonstrates unequal element weights. A single-element-thick column of hex elements is constructed in the x-direction. Weights are assigned to elements such that a perfect rebalance is possible with Proc_Rank+1 elements placed on each processor. This is achieved when running on #procs < 4, but Zoltan does not produce the optimal new distribution for 4 or more procs.

The following hex8 mesh is used in this use case:

  Global node and element numbering
 <pre>
      3       7      11      15      19                 
      +-------+-------+-------+-------+                +-------+
     /       /       /       /       /|               /       /|
   4/      8/     12/     16/     20/ |              /       / |    Z  Y
   +-------+-------+-------+-------+  |     ......  +-------+  |    | / 
   |       |       |       |       |  +18           |       |  /    |/  
   |  e1   |  e2   |  e3   |  e4   | /              |  eN   | /     *--X       
   |       |       |       |       |/               |       |/         
   +-------+-------+-------+-------+                +-------+
   1       5      9       13      17                  
 </pre>
  where N = #elements = (#procs)(#procs+1)/2

  Local node numbering
 <pre>
      8       7
      +-------+
     /       /|
   5/      6/ |
   +-------+  |
   |       |  +3
   |  e1   | /
   |       |/
   +-------+
   1       2
 </pre>

The mesh is constructed on proc 0 using the HexFixture class and unequal element weights are assigned as follows:

// Assign weights so that a perfect rebalance is possible so long as the rebalancer can figure out
// to put p_rank+1 elements on proc = p_rank based on these weights.
unsigned nslabs = 0;
if( 0 == p_rank ) {
for ( unsigned l = 1 ; l <= p_size ; ++l ) {
for ( unsigned k = 0 ; k < nz ; ++k ) {
for ( unsigned j = 0 ; j < ny ; ++j ) {
for ( unsigned i = 0 ; i < l ; ++i ) {
const stk_classic::mesh::EntityId elem_id = 1 + nslabs + i + j*ngx + k*ngx*ny;
stk_classic::mesh::Entity * elem = bulk.get_entity(element_rank, elem_id);
double * const e_weight = stk_classic::mesh::field_data( weight_field , *elem );
*e_weight = double(ngx) / double(l);
}
}
}
nslabs += l;
}
}
// end assign weights

See UseCase_Rebal_1.cpp for the complete source listing.

Use Case 2: Node, Edge, Face and Element weights on subset

This use case demonstrates element weights comprised of contributions from nodes, edges, faces and elements over a subset of the mesh. A 3x3x3 cube of hex8 elements is constructed on proc 0 and weights are assigned to a subset of mesh entities. The mesh and weights are assigned as follows:

  Global node and element numbering
 <pre>

           +-------+-------+-------+               +                   +                 +
          /       /       /       /|              /|                  /|         
         /       /       /       / |             / |                 / |        
        +-------+-------+-------+  |            +  |                +  |    
       /       /       /       /|  +           /|  +               /   +   
      /       /       /       / | /|          / | /|              /    |  
     +-------+-------+-------+  |/ |         +  |/ |             +     | 
    /       /       /       /|  +  |        /|  +  |            /      |
   /       /       /       / | /|  +       / | /|  +           /       +
  +-------+-------+-------+  |/ | /|      +  |/ | /|          +        |        +
  |       |       |       |  +  |/ |      |  +  |/ |          |        |    
  |  e1   |  e2   |  e3   | /|  +  |      | /|  +  |          |        |        
  |       |       |       |/ | /|  +      |/ | /|  +          |        +                 +
  +-------+-------+-------+  |/ | /       +  |/ | /           +       /         
  |       |       |       |  +  |/        |  +  |/            |      /      
  |  e1   |  e2   |  e3   | /|  +         | /|  +             |     +      
  |       |       |       |/ | /          |/ | /              |    /      
  +-------+-------+-------+  |/           +  |/               +   /      
  |       |       |       |  +            |  +                |  +      
  |  e1   |  e2   |  e3   | /             | /                 | /      
  |       |       |       |/              |/                  |/      
  +-------+-------+-------+               +                   +                 +
x = 0
 </pre>
  Weight_elems = 1.0                 Z  Y      Local node numbering
  Weight_faces = 10.0                | /      
  Weight_edges = 100.0               |/            8       7       
  Weight_nodes = 1000.0              *--X          +-------+              
                                                  /       /|       
                                                5/      6/ |       
                                                +-------+  |   
                                                |       |  +3   
                                                |  e1   | /
                                                |       |/
                                                +-------+
                                                1       2
  

where all 27 elements are assigned weights along with the 9 faces, 12 edges and 4 nodes on the plane at x = 0.

// Assign entity weights
if( 0 == p_rank )
{
// Get the faces on the x=0 plane and give them a characteristic weight
stk_classic::mesh::EntityVector selected_nodes;
stk_classic::mesh::EntityVector selected_faces;
stk_classic::mesh::EntityVector one_face;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, k ) );
selected_nodes.push_back( fixture.node(0, j+1, k ) );
selected_nodes.push_back( fixture.node(0, j, k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, face_rank, one_face);
selected_faces.push_back(one_face[0]);
}
for( size_t iface = 0; iface < selected_faces.size(); ++iface )
{
stk_classic::mesh::Entity * face = selected_faces[iface];
double * const weight = stk_classic::mesh::field_data( weight_field, *face );
weight[0] = 10.0;
}
// Get the edges on the boundary of the x=0 plane and give them a characteristic weight
stk_classic::mesh::EntityVector selected_edges;
stk_classic::mesh::EntityVector one_edge;
for ( unsigned j = 0 ; j < ny; ++j )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, 0) );
selected_nodes.push_back( fixture.node(0, j+1, 0) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, nz) );
selected_nodes.push_back( fixture.node(0, j+1, nz) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, 0, k) );
selected_nodes.push_back( fixture.node(0, 0, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, ny, k) );
selected_nodes.push_back( fixture.node(0, ny, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, edge_rank, one_edge);
selected_edges.push_back(one_edge[0]);
}
for( size_t iedge = 0; iedge < selected_edges.size(); ++iedge )
{
stk_classic::mesh::Entity * edge = selected_edges[iedge];
double * const weight = stk_classic::mesh::field_data( weight_field, *edge );
weight[0] = 100.0;
}
// Finally, give the corner nodes of the x=0 plane a characteristic weight
selected_nodes.clear();
double * weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, 0, 0) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, ny, 0) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, 0, nz) );
weight[0] = 1000.0;
weight = stk_classic::mesh::field_data( weight_field, *fixture.node(0, ny, nz) );
weight[0] = 1000.0;
// Assign element weights
for( size_t i = 0; i < my_element_ids.size(); ++i )
{
stk_classic::mesh::Entity * elem = bulk.get_entity(element_rank, my_element_ids[i]);
double * const e_weight = stk_classic::mesh::field_data( weight_field , *elem );
*e_weight = 1.0;
}
//
// Get the elements on the x=0 plane and sum in weights from relations
selected_nodes.clear();
for ( unsigned j = 0 ; j < ny+1; ++j )
for ( unsigned k = 0 ; k < nz+1; ++k )
selected_nodes.push_back( fixture.node(0, j, k) );
std::vector<stk_classic::mesh::EntityRank> ranks;
ranks.push_back(face_rank);
ranks.push_back(edge_rank);
ranks.push_back(node_rank);
stk_classic::mesh::EntityVector selected_elems;
for ( unsigned j = 0 ; j < ny; ++j )
for ( unsigned k = 0 ; k < nz; ++k )
{
selected_nodes.clear();
selected_nodes.push_back( fixture.node(0, j, k ) );
selected_nodes.push_back( fixture.node(0, j+1, k ) );
selected_nodes.push_back( fixture.node(0, j, k+1) );
selected_nodes.push_back( fixture.node(0, j+1, k+1) );
stk_classic::mesh::get_entities_through_relations(selected_nodes, element_rank, one_face);
selected_elems.push_back(one_face[0]);
}
sum_element_weights_through_relations(selected_elems, weight_field, ranks);
}

The use case passes if the amount of imbalance following a rebalance is below 1.45 for 3 procs and below 1.1 for 2 or 4 procs.

See UseCase_Rebal_2.cpp for the complete source listing.

Use Case 3: Periodic Boundary via Constraint Relations

This use case sets up a 2D mesh of quad4 elements and then establishes a constraint realtion between the top and bottom of the mesh as would be needed to enforce periodic boundary conditions.

The following quad4 mesh is manually constructed on proc 0:

  Global node and element numbering
 <pre>

   21      22      23      24      25
   +-------+-------+-------+-------+   y = top
   |       |       |       |       |
   |  e13  |  e14  |  e15  |  e16  |
   |       |       |       |5      |
16 +-------+-------+-------+-------+ 20
   |       |       |       |       |      
   |  e9   |  e10  |  e11  |  e12  |      
   |       |       |       |       |      
11 +-------+-------+-------+-------+ 15
   |       |       |       |       |
   |  e5   |  e6   |  e7   |  e8   |
   |       |5      |       |       |  
 6 +-------+-------+-------+-------+ 10
   |       |       |       |       |  
   |  e1   |  e2   |  e3   |  e4   |    
   |       |       |       |       | 
   +-------+-------+-------+-------+   y = bottom
   1       2       3       4       5
 </pre>

Local node numbering:

       3       4
       +-------+     Y
       |       |     |
       |  e1   |     |
       |       |     *--> X
       +-------+
       1       2
     

and the two sets of nodes at y=bottom and y=top are related through constraint relations as follows:

// Assign constraint relations between nodes at top and bottom of mesh
{
const unsigned iy_bottom = 0;
const unsigned iy_top = ny;
stk_classic::mesh::PartVector add(1, &fem_meta.locally_owned_part());
for ( unsigned ix = 0 ; ix <= nx ; ++ix ) {
stk_classic::mesh::EntityId nid_bottom = 1 + ix + iy_bottom * nnx ;
stk_classic::mesh::EntityId nid_top = 1 + ix + iy_top * nnx ;
stk_classic::mesh::Entity * n_bottom = bulk_data.get_entity( node_rank, nid_bottom );
stk_classic::mesh::Entity * n_top = bulk_data.get_entity( node_rank, nid_top );
const stk_classic::mesh::EntityId constraint_entity_id = 1 + ix + nny * nnx;
stk_classic::mesh::Entity & c = bulk_data.declare_entity( constraint_rank, constraint_entity_id, add );
bulk_data.declare_relation( c , *n_bottom , 0 );
bulk_data.declare_relation( c , *n_top , 1 );
}
} // end snippet

The use case passes if the load imbalance of the new partition is below the nominal value of 1.5.

See UseCase_Rebal_3.cpp for the complete source listing.

Use Case 4: User-customization following default rebalance

This use case demonstrates additional user customization following a default rebalance in order to enforce constraints for new partitions. In this case, the constraint is that two quad4 elements sharing edge #7 be collocated on the same proc following rebalance. This is enforced using a greedy sideset class which inherits the determine_new_partition method.

The following quad4 mesh is used in this use case:

Global node and element numbering

    13      14      15      16
    +-------+-------+-------+
    |       |       |       |
    |  e7   |  e8   |  e9   |
    |       |       |       |
  9 +-------+-------+-------+ 12    Y
    |       |       |       |       |
    |  e4   |  e5   |  e6   |       |
    |       |       |       |       *--> X
  5 +-------+-------+-------+ 8
    |       |       |       | 
    |  e1   |  e2   |  e3   |
    |       |       |       | 
    +-------+-------+-------+ 
    1       2       3       4

Local node numbering

          3       4
          +-------+
          |       |
          |  e1   |
          |       |
          +-------+
          1       2

See UseCase_Rebal_4.cpp for the complete source listing.