mirror of
https://github.com/JuliaFEM/JuliaFEM.jl.git
synced 2026-09-17 17:22:10 +00:00
485 lines
14 KiB
Julia
485 lines
14 KiB
Julia
|
|
# This file is a part of JuliaFEM.
|
|||
|
|
# License is MIT: see https://github.com/JuliaFEM/JuliaFEM.jl/blob/master/LICENSE.md
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
Node-based COO assembly using block integration.
|
|||
|
|
|
|||
|
|
**NODAL ASSEMBLY PARADIGM**: Loop over nodes, not elements!
|
|||
|
|
|
|||
|
|
Each node:
|
|||
|
|
1. Finds all elements touching it (via inverse connectivity)
|
|||
|
|
2. For each touching element:
|
|||
|
|
- Prepares element geometry once (PreparedElement)
|
|||
|
|
- Computes only needed 3×3 blocks (compute_block!)
|
|||
|
|
3. Scatters blocks to COO triplets
|
|||
|
|
|
|||
|
|
# Key Differences from Element-Based Assembly
|
|||
|
|
|
|||
|
|
**Element-Based (traditional):**
|
|||
|
|
```julia
|
|||
|
|
for element in elements
|
|||
|
|
K_e = compute_element_stiffness(element) # Full N×N matrix of 3×3 blocks
|
|||
|
|
scatter(K_e) # Scatter all entries
|
|||
|
|
end
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Node-Based (this file):**
|
|||
|
|
```julia
|
|||
|
|
for node_i in nodes
|
|||
|
|
for element in elements_touching(node_i)
|
|||
|
|
prepared = prepare_element(element) # Geometry preprocessing
|
|||
|
|
for node_j in element.nodes
|
|||
|
|
K_ij = compute_block!(prepared, i, j) # Single 3×3 block
|
|||
|
|
scatter(K_ij, i, j) # Scatter one block
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# Advantages
|
|||
|
|
|
|||
|
|
1. **GPU-friendly**: One thread per node, no race conditions
|
|||
|
|
2. **Contact-ready**: Contact is naturally node-based
|
|||
|
|
3. **Matrix-free ready**: Can compute K*v without forming K
|
|||
|
|
4. **Cache-friendly**: Reuses PreparedElement for multiple blocks
|
|||
|
|
5. **Adaptive-ready**: Easy to refine/coarsen at node level
|
|||
|
|
|
|||
|
|
# Performance Expectations
|
|||
|
|
|
|||
|
|
- **CPU Single-thread**: ~1.5-2x slower than element-based (more kernel calls)
|
|||
|
|
- **CPU Multi-thread**: ~1.5-2x faster (better parallelization)
|
|||
|
|
- **GPU**: ~10-50x faster (massive parallelization, no atomics needed)
|
|||
|
|
|
|||
|
|
# References
|
|||
|
|
|
|||
|
|
- Golden standard: `docs/src/book/multigpu_nodal_assembly.md`
|
|||
|
|
- PreparedElement: `src/domains/continuum/integration.jl`
|
|||
|
|
- Block kernel: `src/domains/continuum/kernel.jl`
|
|||
|
|
|
|||
|
|
# Example
|
|||
|
|
|
|||
|
|
```julia
|
|||
|
|
# Setup
|
|||
|
|
mesh = create_cantilever_mesh(50, 10, 10)
|
|||
|
|
material = LinearElastic(E=210e9, ν=0.3)
|
|||
|
|
kernel = ContinuumKernel(
|
|||
|
|
ContinuumFormulation{FullThreeD}(),
|
|||
|
|
material,
|
|||
|
|
Displacement{3}()
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
# Create node-based assembler and cache
|
|||
|
|
assembler = NodeBasedCOOAssembler()
|
|||
|
|
cache = create_cache(assembler, mesh, kernel)
|
|||
|
|
|
|||
|
|
# Assemble (zero allocations after warmup!)
|
|||
|
|
assemble!(cache, assembler, kernel, mesh)
|
|||
|
|
|
|||
|
|
# Extract system
|
|||
|
|
K, f = extract_system(cache)
|
|||
|
|
|
|||
|
|
# Solve
|
|||
|
|
apply_dirichlet_bcs!(K, f, kernel, mesh, bc_dirichlet)
|
|||
|
|
u = K \\ f
|
|||
|
|
```
|
|||
|
|
"""
|
|||
|
|
|
|||
|
|
using SparseArrays
|
|||
|
|
using Tensors
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
NodeBasedCOOCache
|
|||
|
|
|
|||
|
|
Pre-allocated cache for node-based COO assembly.
|
|||
|
|
|
|||
|
|
Similar to COOCache but includes inverse connectivity mapping.
|
|||
|
|
|
|||
|
|
# Fields
|
|||
|
|
- `I::Vector{Int}`: Row indices (COO format)
|
|||
|
|
- `J::Vector{Int}`: Column indices (COO format)
|
|||
|
|
- `V::Vector{Float64}`: Values (COO format)
|
|||
|
|
- `f::Vector{Float64}`: Global force vector
|
|||
|
|
- `counter::Ref{Int}`: Current triplet count
|
|||
|
|
- `capacity::Int`: Maximum triplet capacity
|
|||
|
|
- `node_to_elements::NodeToElementsMap`: Inverse connectivity
|
|||
|
|
- `element_cache::ElementCache`: Cache for element operations
|
|||
|
|
- `ndofs::Int`: Total DOFs in system
|
|||
|
|
"""
|
|||
|
|
struct NodeBasedCOOCache{T<:AbstractTopology,B<:AbstractBasis,IPS}
|
|||
|
|
I::Vector{Int}
|
|||
|
|
J::Vector{Int}
|
|||
|
|
V::Vector{Float64}
|
|||
|
|
f::Vector{Float64}
|
|||
|
|
counter::Ref{Int}
|
|||
|
|
capacity::Int
|
|||
|
|
node_to_elements::NodeToElementsMap
|
|||
|
|
element_cache::ElementCache{T,B,IPS}
|
|||
|
|
ndofs::Int
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
NodeBasedCOOCache(mesh::AbstractMesh, kernel::ContinuumKernel)
|
|||
|
|
|
|||
|
|
Create cache for node-based assembly.
|
|||
|
|
|
|||
|
|
Builds inverse connectivity and allocates buffers.
|
|||
|
|
|
|||
|
|
# Arguments
|
|||
|
|
- `mesh`: Finite element mesh
|
|||
|
|
- `kernel`: Continuum kernel
|
|||
|
|
|
|||
|
|
# Returns
|
|||
|
|
- Pre-allocated node-based COO cache
|
|||
|
|
"""
|
|||
|
|
function NodeBasedCOOCache(mesh::AbstractMesh, kernel::ContinuumKernel)
|
|||
|
|
# Build inverse connectivity
|
|||
|
|
node_to_elements = NodeToElementsMap(mesh.connectivity)
|
|||
|
|
|
|||
|
|
# Estimate triplet count (same as element-based)
|
|||
|
|
ndofs_per_node = dofs_per_node(kernel)
|
|||
|
|
nnodes = length(mesh.nodes)
|
|||
|
|
ndofs = ndofs_per_node * nnodes
|
|||
|
|
|
|||
|
|
# Estimate: For each node, sum over touching elements
|
|||
|
|
# Each element contributes N blocks (N = nodes per element)
|
|||
|
|
# Each block = 3×3 = 9 triplets
|
|||
|
|
avg_elements_per_node = node_to_elements.nelements / nnodes
|
|||
|
|
N = length(first(mesh.connectivity)) # Nodes per element
|
|||
|
|
estimated_triplets = Int(ceil(1.2 * nnodes * avg_elements_per_node * N * 9))
|
|||
|
|
|
|||
|
|
# Allocate triplet arrays
|
|||
|
|
I = Vector{Int}(undef, estimated_triplets)
|
|||
|
|
J = Vector{Int}(undef, estimated_triplets)
|
|||
|
|
V = Vector{Float64}(undef, estimated_triplets)
|
|||
|
|
f = zeros(Float64, ndofs)
|
|||
|
|
counter = Ref(0)
|
|||
|
|
|
|||
|
|
# Create element cache (for prepare_element! and compute_block!)
|
|||
|
|
element_cache = ElementCache(mesh, kernel)
|
|||
|
|
|
|||
|
|
return NodeBasedCOOCache(I, J, V, f, counter, estimated_triplets,
|
|||
|
|
node_to_elements, element_cache, ndofs)
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
reset!(cache::NodeBasedCOOCache)
|
|||
|
|
|
|||
|
|
Reset cache for new assembly (zero force vector, reset counter).
|
|||
|
|
|
|||
|
|
Does NOT clear inverse connectivity (that's permanent structure).
|
|||
|
|
"""
|
|||
|
|
function reset!(cache::NodeBasedCOOCache)
|
|||
|
|
fill!(cache.f, 0.0)
|
|||
|
|
cache.counter[] = 0
|
|||
|
|
return nothing
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
assemble!(
|
|||
|
|
cache::NodeBasedCOOCache,
|
|||
|
|
assembler::NodeBasedCOOAssembler,
|
|||
|
|
kernel::ContinuumKernel,
|
|||
|
|
mesh::AbstractMesh
|
|||
|
|
) -> Nothing
|
|||
|
|
|
|||
|
|
Assemble global system using **node-based traversal**.
|
|||
|
|
|
|||
|
|
# Algorithm
|
|||
|
|
|
|||
|
|
```julia
|
|||
|
|
for node_i in 1:nnodes
|
|||
|
|
# Get all elements touching this node
|
|||
|
|
for elem_info in node_to_elements[node_i]
|
|||
|
|
element_id = elem_info.element_id
|
|||
|
|
local_i = elem_info.local_node_idx
|
|||
|
|
|
|||
|
|
# Prepare element geometry ONCE
|
|||
|
|
prepared = prepare_element!(cache.element_cache, kernel, element_id, mesh)
|
|||
|
|
|
|||
|
|
# Compute blocks for all nodes in this element
|
|||
|
|
for local_j in 1:N
|
|||
|
|
global_j = connectivity[element_id][local_j]
|
|||
|
|
|
|||
|
|
# Compute single 3×3 block
|
|||
|
|
K_ij = compute_block!(prepared, kernel.material, local_i, local_j)
|
|||
|
|
|
|||
|
|
# Scatter to triplets
|
|||
|
|
scatter_block_to_triplets!(cache, K_ij, node_i, global_j)
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# Key Operations
|
|||
|
|
|
|||
|
|
1. **prepare_element!** - Precompute geometry (Jacobian, gradients) once per element
|
|||
|
|
2. **compute_block!** - Compute single 3×3 stiffness block using prepared geometry
|
|||
|
|
3. **scatter_block_to_triplets!** - Add 9 triplets (i,j,value) for 3×3 block
|
|||
|
|
|
|||
|
|
# Zero-Allocation (After Warmup)
|
|||
|
|
|
|||
|
|
All arrays pre-allocated. Element preparation reuses cache buffers.
|
|||
|
|
|
|||
|
|
# Arguments
|
|||
|
|
- `cache`: Pre-allocated node-based COO cache
|
|||
|
|
- `assembler`: Node-based COO assembler
|
|||
|
|
- `kernel`: Continuum kernel
|
|||
|
|
- `mesh`: Finite element mesh
|
|||
|
|
"""
|
|||
|
|
function assemble!(
|
|||
|
|
cache::NodeBasedCOOCache,
|
|||
|
|
assembler::NodeBasedCOOAssembler,
|
|||
|
|
kernel::ContinuumKernel,
|
|||
|
|
mesh::AbstractMesh
|
|||
|
|
)
|
|||
|
|
# Reset cache
|
|||
|
|
reset!(cache)
|
|||
|
|
|
|||
|
|
nnodes = length(mesh.nodes)
|
|||
|
|
ndofs_per_node = dofs_per_node(kernel)
|
|||
|
|
|
|||
|
|
# NODAL LOOP: One iteration per node (GPU: one thread per node!)
|
|||
|
|
for node_i in 1:nnodes
|
|||
|
|
# Get all elements touching this node
|
|||
|
|
touching_elements = cache.node_to_elements.node_to_elements[node_i]
|
|||
|
|
|
|||
|
|
# Loop over touching elements
|
|||
|
|
for elem_info in touching_elements
|
|||
|
|
element_id = elem_info.element_id
|
|||
|
|
local_i = elem_info.local_node_idx # Position of node_i in element
|
|||
|
|
|
|||
|
|
# Prepare element geometry ONCE (reuses cache.element_cache)
|
|||
|
|
prepared = prepare_element!(cache.element_cache, kernel, element_id, mesh)
|
|||
|
|
|
|||
|
|
# Get element connectivity
|
|||
|
|
conn = mesh.connectivity[element_id]
|
|||
|
|
N = length(conn) # Nodes per element
|
|||
|
|
|
|||
|
|
# Compute blocks for all nodes j in this element
|
|||
|
|
for local_j in 1:N
|
|||
|
|
global_j = conn[local_j]
|
|||
|
|
|
|||
|
|
# Compute single 3×3 block K[i,j]
|
|||
|
|
# This is THE KEY OPERATION: block-based integration
|
|||
|
|
K_ij = compute_block!(
|
|||
|
|
prepared,
|
|||
|
|
kernel.material,
|
|||
|
|
local_i,
|
|||
|
|
local_j
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
# Scatter 3×3 block to triplets (adds 9 entries)
|
|||
|
|
scatter_block_to_triplets!(
|
|||
|
|
cache,
|
|||
|
|
K_ij,
|
|||
|
|
node_i,
|
|||
|
|
global_j,
|
|||
|
|
ndofs_per_node
|
|||
|
|
)
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
return nothing
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
scatter_block_to_triplets!(
|
|||
|
|
cache::NodeBasedCOOCache,
|
|||
|
|
K_block::Tensor{2,3},
|
|||
|
|
node_i::Int,
|
|||
|
|
node_j::Int,
|
|||
|
|
ndofs_per_node::Int
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
Scatter single 3×3 block to COO triplets **in-place**.
|
|||
|
|
|
|||
|
|
Maps block[α,β] → triplet at DOF indices:
|
|||
|
|
- Row: 3*(node_i-1) + α
|
|||
|
|
- Col: 3*(node_j-1) + β
|
|||
|
|
- Val: K_block[α,β]
|
|||
|
|
|
|||
|
|
# Arguments
|
|||
|
|
- `cache`: Node-based COO cache
|
|||
|
|
- `K_block`: 3×3 stiffness block (Tensor{2,3})
|
|||
|
|
- `node_i`: Global row node index
|
|||
|
|
- `node_j`: Global column node index
|
|||
|
|
- `ndofs_per_node`: DOFs per node (typically 3)
|
|||
|
|
|
|||
|
|
# Zero-Allocation
|
|||
|
|
|
|||
|
|
Writes to pre-allocated triplet arrays, updates counter.
|
|||
|
|
"""
|
|||
|
|
function scatter_block_to_triplets!(
|
|||
|
|
cache::NodeBasedCOOCache,
|
|||
|
|
K_block::Tensor{2,3,Float64},
|
|||
|
|
node_i::Int,
|
|||
|
|
node_j::Int,
|
|||
|
|
ndofs_per_node::Int
|
|||
|
|
)
|
|||
|
|
counter = cache.counter[]
|
|||
|
|
|
|||
|
|
# Check capacity
|
|||
|
|
new_triplets = ndofs_per_node * ndofs_per_node # 3×3 = 9
|
|||
|
|
if counter + new_triplets > cache.capacity
|
|||
|
|
error("Node-based COO cache overflow: need $(counter + new_triplets) triplets, " *
|
|||
|
|
"capacity is $(cache.capacity). Increase cache size.")
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
# DOF offsets for nodes i and j
|
|||
|
|
row_offset = ndofs_per_node * (node_i - 1)
|
|||
|
|
col_offset = ndofs_per_node * (node_j - 1)
|
|||
|
|
|
|||
|
|
# Scatter 3×3 block to triplets
|
|||
|
|
for β in 1:ndofs_per_node # Column (node j DOF)
|
|||
|
|
j_global = col_offset + β
|
|||
|
|
for α in 1:ndofs_per_node # Row (node i DOF)
|
|||
|
|
i_global = row_offset + α
|
|||
|
|
counter += 1
|
|||
|
|
cache.I[counter] = i_global
|
|||
|
|
cache.J[counter] = j_global
|
|||
|
|
cache.V[counter] = K_block[α, β]
|
|||
|
|
end
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
cache.counter[] = counter
|
|||
|
|
return nothing
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
extract_system(cache::NodeBasedCOOCache) -> (K, f)
|
|||
|
|
|
|||
|
|
Build sparse matrix from triplets and return system.
|
|||
|
|
|
|||
|
|
Calls `sparse(I, J, V)` to build CSC matrix. Duplicates are summed automatically.
|
|||
|
|
|
|||
|
|
# Arguments
|
|||
|
|
- `cache`: Assembled node-based COO cache
|
|||
|
|
|
|||
|
|
# Returns
|
|||
|
|
- `K::SparseMatrixCSC`: Global stiffness matrix
|
|||
|
|
- `f::Vector`: Global force vector
|
|||
|
|
|
|||
|
|
# Allocation
|
|||
|
|
|
|||
|
|
Allocates sparse matrix structure (CSC format). This is the only allocation
|
|||
|
|
outside cache construction.
|
|||
|
|
"""
|
|||
|
|
function extract_system(cache::NodeBasedCOOCache)
|
|||
|
|
ntriplets = cache.counter[]
|
|||
|
|
|
|||
|
|
# Build sparse matrix (duplicates are summed automatically)
|
|||
|
|
I_used = @view cache.I[1:ntriplets]
|
|||
|
|
J_used = @view cache.J[1:ntriplets]
|
|||
|
|
V_used = @view cache.V[1:ntriplets]
|
|||
|
|
|
|||
|
|
K = sparse(I_used, J_used, V_used, cache.ndofs, cache.ndofs)
|
|||
|
|
|
|||
|
|
return K, cache.f
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
# ============================================================================
|
|||
|
|
# HELPER FUNCTIONS
|
|||
|
|
# ============================================================================
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
create_cache(
|
|||
|
|
assembler::NodeBasedCOOAssembler,
|
|||
|
|
mesh::AbstractMesh,
|
|||
|
|
kernel::ContinuumKernel
|
|||
|
|
) -> NodeBasedCOOCache
|
|||
|
|
|
|||
|
|
Create pre-allocated cache for node-based COO assembly.
|
|||
|
|
|
|||
|
|
Convenience function that wraps `NodeBasedCOOCache(mesh, kernel)`.
|
|||
|
|
|
|||
|
|
# Example
|
|||
|
|
|
|||
|
|
```julia
|
|||
|
|
assembler = NodeBasedCOOAssembler()
|
|||
|
|
cache = create_cache(assembler, mesh, kernel)
|
|||
|
|
assemble!(cache, assembler, kernel, mesh)
|
|||
|
|
K, f = extract_system(cache)
|
|||
|
|
```
|
|||
|
|
"""
|
|||
|
|
function create_cache(
|
|||
|
|
assembler::NodeBasedCOOAssembler,
|
|||
|
|
mesh::AbstractMesh,
|
|||
|
|
kernel::ContinuumKernel
|
|||
|
|
)
|
|||
|
|
return NodeBasedCOOCache(mesh, kernel)
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
"""
|
|||
|
|
dofs_per_node(kernel::ContinuumKernel) -> Int
|
|||
|
|
|
|||
|
|
Return DOFs per node for continuum kernel (always 3 for displacement).
|
|||
|
|
|
|||
|
|
Dispatches on kernel field dimension.
|
|||
|
|
"""
|
|||
|
|
function dofs_per_node(kernel::ContinuumKernel{Theory,Mat}) where {Theory,Mat}
|
|||
|
|
field = kernel.field
|
|||
|
|
return field.dim # Displacement{3} → 3
|
|||
|
|
end
|
|||
|
|
|
|||
|
|
# ============================================================================
|
|||
|
|
# PERFORMANCE NOTES
|
|||
|
|
# ============================================================================
|
|||
|
|
|
|||
|
|
#=
|
|||
|
|
# CPU Performance Comparison (Estimated)
|
|||
|
|
|
|||
|
|
**Element-Based Assembly:**
|
|||
|
|
- Elements: 1000 Tet4
|
|||
|
|
- Operations: 1000 elements × 4×4 blocks × 3×3 entries = 48,000 block computations
|
|||
|
|
- Time: ~5ms (baseline)
|
|||
|
|
|
|||
|
|
**Node-Based Assembly:**
|
|||
|
|
- Nodes: 500 nodes
|
|||
|
|
- Operations: 500 nodes × 8 elements/node × 4 blocks/element = 16,000 block computations
|
|||
|
|
- But: 3× more kernel calls due to overlaps
|
|||
|
|
- Time: ~7-10ms (1.5-2× slower single-threaded)
|
|||
|
|
|
|||
|
|
**Why slower on CPU?**
|
|||
|
|
- Each block computed once in element assembly
|
|||
|
|
- Each block computed 2× on average in nodal assembly (shared between 2 elements)
|
|||
|
|
- More function call overhead
|
|||
|
|
|
|||
|
|
**Why faster on GPU?**
|
|||
|
|
- Element assembly: Sequential (can't parallelize over elements efficiently)
|
|||
|
|
- Nodal assembly: Massive parallelism (one thread per node)
|
|||
|
|
- GPU speedup: ~10-50× depending on problem size
|
|||
|
|
|
|||
|
|
**Multi-threaded CPU (Threads.@threads):**
|
|||
|
|
- Can parallelize outer node loop
|
|||
|
|
- Expected speedup: 1.5-2× over element-based
|
|||
|
|
- No race conditions (each node writes different triplets)
|
|||
|
|
|
|||
|
|
# Memory Comparison
|
|||
|
|
|
|||
|
|
**Element-Based:**
|
|||
|
|
- Triplet storage: ~50 KB per 1000 elements
|
|||
|
|
- Element cache: ~2 KB per thread
|
|||
|
|
|
|||
|
|
**Node-Based:**
|
|||
|
|
- Triplet storage: Same (~50 KB)
|
|||
|
|
- Element cache: ~2 KB per thread
|
|||
|
|
- Inverse connectivity: ~10-20 KB (one-time)
|
|||
|
|
|
|||
|
|
→ Nearly identical memory usage!
|
|||
|
|
|
|||
|
|
# When to Use Node-Based Assembly
|
|||
|
|
|
|||
|
|
**Use when:**
|
|||
|
|
- ✅ GPU acceleration needed
|
|||
|
|
- ✅ Contact mechanics (naturally nodal)
|
|||
|
|
- ✅ Matrix-free methods (K*v without forming K)
|
|||
|
|
- ✅ Adaptive refinement (local node operations)
|
|||
|
|
- ✅ Multi-threading on CPU
|
|||
|
|
|
|||
|
|
**Don't use when:**
|
|||
|
|
- ❌ Single-threaded CPU only
|
|||
|
|
- ❌ Simple problems (< 1000 nodes)
|
|||
|
|
- ❌ Prototyping/debugging (element-based is clearer)
|
|||
|
|
=#
|