First working GPU assembly implementation (proof-of-concept stage)
demonstrating complete FEM solve staying on GPU for 2D elasticity.
Implementation:
- Element-parallel GPU kernel for Quad4 elements
- 2×2 Gauss quadrature on GPU
- Plain vector approach (before Tensors.jl integration)
- Matrix-free Jacobian-vector product
- Complete Newton-Krylov loop on GPU
- BC enforcement via masking
Test case: 10×10 Quad4 mesh (100 elements, 242 DOFs)
- Material: Steel (E=200 GPa, ν=0.3)
- BC: Fixed left edge, displacement on right edge
Architecture validation:
- GPU assembly matches CPU (error < 1e-15)
- Entire solve stays on GPU (no ping-pong)
- Only transfers: mesh (once) + u0/u_final (boundary)
Note: This is the initial version using plain vectors and manual
indexing. See gpu_assembly_poc_tensors.jl for corrected version
using proper Tensors.jl material API (606 lines).
Complete GPU-accelerated FEM solve for 3D cantilever beam using
nodal assembly strategy and matrix-free Newton-Krylov solver.
GPU implementation:
- Gmsh mesh generation (same as CPU version)
- Data transfer to GPU (nodes, connectivity, BC)
- GPU kernels for nodal assembly (element contributions)
- Matrix-free Jacobian-vector product on GPU
- GMRES solver on GPU (Krylov.jl with CuArrays)
- CPU fallback for Anderson acceleration
Problem characteristics:
- Geometry: 10m × 1m × 1m cantilever beam
- Elements: Tet4 from Gmsh
- Material: Steel (E=210 GPa, ν=0.3)
- BC: Fixed left end, tip force on right end
Architecture:
- Single GPU transfer: mesh + BC → GPU at start
- Entire Newton-Krylov loop stays on GPU
- Single result transfer: u_final ← GPU at end
- No ping-pong between CPU and GPU during solve
Demonstrates complete GPU FEM pipeline from meshing to solution
with realistic geometry and material properties.
Compares traditional element assembly vs nodal assembly on CPU for
cantilever beam example, validating assembly equivalence and measuring
performance characteristics.
Comparison:
- Element assembly: Traditional FEM (loop over elements, atomic scatter)
- Nodal assembly: Modern approach (loop over nodes, block operations)
Validation:
- Residual equivalence (element vs nodal assembly)
- Stiffness operator equivalence (matvec comparison)
- Assembly time comparison
- Memory allocation tracking
Problem: Same cantilever beam as cantilever_beam_simple.jl
- Tet4 mesh from Gmsh
- Steel properties
- Fixed left, force on right
Demonstrates CPU assembly strategies before GPU implementation,
establishing baseline for GPU performance comparison.
Demonstrates modern Physics API for 3D elasticity on realistic geometry
using Gmsh mesh generation and both direct/iterative solvers.
Features:
- Gmsh mesh generation (10m × 1m × 1m cantilever beam)
- Tet4 elements with controlled mesh size (lc=1.5)
- Physics API setup (Elasticity, continuum formulation)
- Steel material properties (E=210 GPa, ν=0.3)
- Boundary conditions: Fixed left end, force on right end
Problem setup:
- Geometry: Cantilever beam (aspect ratio 10:1:1)
- Discretization: Tet4 elements from Gmsh
- Loading: Tip force applied via Neumann BC
- Constraints: Fixed end via Dirichlet BC
Workflow demonstration:
1. Mesh generation with Gmsh API
2. Physics problem creation
3. Solver setup (direct or iterative)
4. Post-processing and visualization
Educational example showing complete FEM workflow from meshing
to solution with modern JuliaFEM API (183 lines).
Demonstrates modern Physics API for solving elasticity problems using
CPU backend with element assembly.
Features:
- Simple 2-element beam mesh (Hex8 elements, 12 nodes, 36 DOFs)
- Immutable Element API with field-based material properties
- Physics problem setup (Elasticity, continuum formulation)
- Material properties: Steel (E=210 GPa, ν=0.3)
Demonstrates workflow:
1. Create mesh (nodes dictionary + connectivity tuples)
2. Create Physics problem (Elasticity with continuum formulation)
3. Build elements with immutable API (fields tuple)
4. Add elements to physics
Educational example showing modern API usage for elasticity
problems with clean separation between geometry and physics (131 lines).
Documents architectural correction from manual Voigt indexing to proper
Tensors.jl material modeling in GPU assembly proof-of-concept.
Problem identified:
- Initial POC used plain vectors instead of SymmetricTensor
- Hardcoded constitutive matrix instead of material API
- Manual index arithmetic for stress components
- Didn't match established material_modeling.md architecture
Solution implemented:
- SymmetricTensor{2,2} for 2D strain and stress
- Material API: compute_stress(material, ε)
- LinearElastic struct with Lamé parameters
- Clean tensor operations matching theory
- GPU compatible (Tensors.jl works on CUDA)
Key architectural changes:
1. Material model struct (LinearElastic with E, ν)
2. Material API with Hooke's law (σ = λ·tr(ε)·I + 2μ·ε)
3. SymmetricTensor strain computation (εxx, εyy, γxy/2)
4. Stress-to-force conversion (Bᵀ·σ operator)
Reference: demos/gpu_assembly_poc_tensors.jl (264 lines)
Implementation roadmap for GPU-accelerated nonlinear solver pipeline derived
from CPU reference implementation (newton_krylov_anderson_cpu.jl).
Breakdown of solver pipeline:
- Outer loop: Newton iterations (residual assembly + line search)
- Middle loop: GMRES iterations (matrix-free matvec + Arnoldi)
- Inner operation: Element residual assembly with plasticity
Key GPU kernels identified:
1. Element residual assembly (workhorse kernel, nodal scatter with atomics)
2. Vector operations (standard cuBLAS: axpy, dot, norm)
Four-phase implementation strategy:
1. Single kernel test (residual assembly CPU vs GPU)
2. Matrix-free matvec test (Jacobian-vector product validation)
3. GMRES on GPU (Krylov.jl with CuArrays)
4. Complete pipeline (GPU main loop, CPU Anderson acceleration)
Includes plastic state GPU representation (NTuple vs SymmetricTensor),
kernel launch configuration, and atomic scatter pattern (296 lines).
New 518-line demo showing ElementSet + immutable fields architecture:
- ElementSet struct groups elements with type-stable fields
- Elements contain only connectivity (NTuple, zero-cost)
- Fields live in ElementSet as NamedTuple (immutable, type-stable)
- Mock GPU execution showing kernel compatibility
Three execution modes demonstrated:
1. CPU assembly with zero allocations
2. Mock GPU assembly (simulates CUDA pattern)
3. Time stepping with field container recreation
Key validations:
- Zero allocations in assembly loop (verified with @benchmark)
- GPU kernel accesses element.connectivity directly
- Creating new NamedTuple ~1000× faster than deepcopy (10ns vs 10μs)
- CPU/GPU results match within 1e-10 relative error
Features:
- Mock GPU module (simulates CUDA.jl without dependency)
- AssemblyCache for pre-allocated buffers
- Time stepping simulation (5 steps with displacement updates)
- Memory usage comparison (immutable vs mutable patterns)
- Complete validation suite with performance metrics
Run with: julia --project=. demos/gpu_elementset_demo.jl
New 203-line GPU-only demonstration (simplified without MPI complexity):
Setup and validation (lines 1-52):
- Checks for CUDA availability, exits if not found
- Reports GPU model and memory
- Sets up problem: 10000 nodes, 1000 elements
- Type-stable data: nodes (Float64), connectivity (Int), E, ν
GPU kernel (lines 54-120):
- assemble_element_kernel! for CUDA
- Type-stable: Float64, Int32, CuDeviceMatrix/Vector
- No allocations in kernel
- Computes simplified assembly: K_local = E * (1-ν²)
Execution (lines 122-170):
- Transfers data to GPU (nodes, connectivity)
- Reports bytes transferred
- Launches kernel with thread blocks
- Transfers results back from GPU
Verification (lines 172-203):
- Compares computed vs expected values
- Reports success/failure
- Key achievements summary:
* Type-stable kernel compiled
* Fast GPU memory transfer (typed arrays)
* Zero allocations in kernel
- Why it matters: Dict-based storage CANNOT compile for GPU
- Conclusion: type stability required for modern HPC
Purpose: Simpler test than full MPI version, focuses on GPU capability
Run: julia demos/gpu_only_demo.jl (requires CUDA GPU)
New 327-line mock demonstration (prototype before real hardware version):
MockCUDA module (lines 20-64):
- Mock CuArray type wrapping CPU arrays
- Mock cu() transfer (simulates CPU→GPU)
- Mock @cuda macro (simulates kernel launch)
- Mock thread/block indexing functions
- Demonstrates API without requiring CUDA.jl dependency
GPU kernel example (lines 66-180):
- Type-stable element assembly kernel
- Shows concrete types required (Float64, Matrix{Float64})
- Demonstrates zero-allocation pattern
- Mock execution showing what real CUDA would do
MPI communication examples (lines 182-280):
- Mock MPI module with Send/Recv
- Type-stable data transfer patterns
- Demonstrates fast vs slow paths
Summary (lines 282-327):
- Why type stability matters for GPU/MPI
- GPU: type-unstable code FAILS to compile
- MPI: typed arrays 100× faster than serialization
- Zero allocations required in GPU kernels
- Pattern: typed structures → pre-allocated buffers → type-stable code
- Critical insight: type stability is REQUIREMENT not optimization
Purpose: Educational prototype demonstrating concepts before real hardware.
Superseded by: gpu_mpi_demo.jl (real CUDA and MPI)
New 400-line distributed FEM solver demonstration with 6 parts:
Part 1: Generate test problem (lines 61-100)
- 10×10 SPD system, condition number ~3.45
- Distributed nodal assembly: each rank owns nodes
- Exact solution x=[1,2,...,10], RHS b=A*x
Part 2: Nodal assembly pattern (lines 101-140)
- get_row(i) and get_rhs(i) abstractions
- Row-by-row matrix construction
- Each rank assembles its local rows
Part 3: GPU transfer (lines 141-167)
- Transfer local data to GPU if CUDA available
- Falls back to CPU arrays if no GPU
- Reports bytes transferred per rank
Part 4: Distributed matrix-vector product (lines 168-203)
- matvec_distributed! function
- Each rank computes y_local = A_local * x_global
- GPU acceleration if available, CPU fallback
Part 5: Conjugate Gradient solver (lines 204-318)
- cg_distributed() with MPI collectives
- Allreduce for global dot products
- Allgatherv for vector assembly
- Reports convergence progress per iteration
Part 6: Verification (lines 319-345)
- Compare computed vs exact solution
- Report relative error
- Pass/fail verification (threshold 1e-6)
Summary (lines 346-400):
- Reports what was demonstrated on real hardware
- Nodal assembly, distributed computing, multi-GPU, Krylov CG
- Key insight: type-stable + nodal → scalable
- Relevance to JuliaFEM contact mechanics
Results: Converges in 9 iterations, 7.73×10⁻¹⁴ relative error
Hardware: 2 MPI ranks, NVIDIA RTX A2000 12GB per rank
Run: mpiexec -np 2 julia --project=. demos/krylov_mpi_gpu_demo.jl
New 322-line demonstration script proving type-stable data flows to GPU and MPI:
Part 1: Type-stable data structures (lines 54-77)
- Creates nodes, connectivity, displacement as typed arrays
- Material properties E, ν as Float64
- All structures explicitly typed (Matrix{Float64}, not Dict)
Part 2: MPI communication (lines 79-123)
- Rank 0 sends 24KB displacement data to rank 1
- Transfers material properties
- Validates data integrity with checksum
- Uses MPI.Send/Recv with typed buffers
Part 3: GPU kernel execution (lines 125-192)
- Defines assemble_element_kernel! for CUDA
- Type-stable kernel: Float64, Int32, no allocations
- Transfers data to GPU (CuArray)
- Launches kernel with thread blocks
- Validates results against expected values
Part 4: Combined GPU+MPI workflow (lines 194-271)
- Rank 0 computes on GPU
- Transfers results via MPI to rank 1
- End-to-end validation
Summary section (lines 273-322):
- Reports hardware used (GPU model, MPI ranks)
- Key insights: type stability required for GPU, enables fast MPI
- Conclusion: type-stable fields are foundation for modern FEM
Requirements: MPI (required), CUDA (optional, detects and uses if available)
Run: mpiexec -np 2 julia --project=. demos/gpu_mpi_demo.jl
New 77-line guide documenting:
- Prerequisites (MPI and CUDA globally installed)
- Running commands for MPI communication test
- Running commands for combined GPU+MPI test
- Single-process GPU test instructions
- What gets demonstrated (type stability requirement, MPI fast transfer, real hardware)
- Success indicators and result interpretation
- Key insight: same patterns enable CPU speedup, GPU execution, and MPI efficiency
New 125-line README documenting:
- Two main demonstrations (GPU+MPI and Krylov solver)
- Requirements (Julia 1.9+, MPI, optional CUDA)
- Key insights: type stability required for GPU/MPI/Krylov
- Nodal assembly pattern explanation
- Architecture validation (v0.5.1 vs v1.0 comparison)
- References to benchmarks and design docs
- Contributing guidelines for new demos
Jukka Aho