docs: Add comprehensive testing philosophy and roadmap

New testing strategy: Educational tests using Literate.jl Core principles: - Tests are primary teaching material (not just validation) - Literate.jl generates docs from test files (always synchronized) - Structured progression: fundamentals → linear → nonlinear → advanced - Fast tests (< 5 min unit, < 30 min full suite) - Target: 99% code coverage Test hierarchy: - tutorials/ - Literate.jl files (test + documentation) - unit/ - Fast isolated function tests - verification/ - Known analytical solutions 8-week implementation roadmap: Week 1: Infrastructure (Literate.jl setup) Week 2-3: Core tutorials (10-15 fundamental topics) Week 4-5: Advanced tutorials (contact, mortar) Week 6: Unit tests (fill coverage gaps → 99%) Week 7: Verification tests (validate correctness) Week 8: Polish and publish documentation Philosophy: 'Tests are not a chore - they teach users how to use JuliaFEM.' Ready to start Phase 1 implementation.
2026-08-06 04:21:33 +00:00 · 2025-11-09 01:47:30 +02:00
parent df40f631f6
commit 269b9ef0cc
1 changed files with 634 additions and 0 deletions
@@ -0,0 +1,634 @@
+# JuliaFEM Testing Philosophy
+
+**Date:** November 9, 2025  
+**Goal:** 99% code coverage with educational, fast, well-structured tests  
+**Tool:** Literate.jl for test-as-documentation
+
+---
+
+## Core Principles
+
+### 1. Tests as Teaching Material
+
+**Every test should teach something.**
+
+Tests are not just validation - they're the **primary way users learn JuliaFEM**. When someone asks "How do I solve an elasticity problem?", the answer should be: "Look at `test/tutorials/elasticity_basics.jl`"
+
+**Benefits:**
+
+- Users learn by example (better than API docs)
+- Tests stay current (if API changes, tests must update)
+- Documentation never lies (it's tested code!)
+- Newcomers can contribute tests (learning exercise)
+
+### 2. Literate.jl for Test-Driven Documentation
+
+Use Literate.jl to write tests as narrative documents:
+
+```julia
+# # Solving Your First Elasticity Problem
+#
+# This tutorial shows how to solve a simple 2D elasticity problem.
+# We'll create a square block, apply boundary conditions, and solve.
+
+using JuliaFEM
+using Test
+
+# ## Step 1: Create the Geometry
+#
+# First, define the nodes of a unit square:
+
+X = Dict(
+    1 => [0.0, 0.0],
+    2 => [1.0, 0.0],
+    3 => [1.0, 1.0],
+    4 => [0.0, 1.0]
+)
+
+# ## Step 2: Create Elements
+#
+# Create a single Quad4 element:
+
+element = Element(Quad4, (1, 2, 3, 4))
+update!(element, "geometry", X)
+
+# ... and so on
+```
+
+**Output:** Same file generates both test (runs in CI) and documentation (builds HTML).
+
+### 3. Structured Test Hierarchy
+
+Organize tests to match learning progression:
+
+```text
+test/
+├── tutorials/          # Literate.jl files (test + docs)
+│   ├── 01_fundamentals/
+│   │   ├── creating_elements.jl
+│   │   ├── fields_and_updates.jl
+│   │   ├── reading_meshes.jl
+│   │   └── basis_functions.jl
+│   ├── 02_linear_problems/
+│   │   ├── elasticity_1d.jl
+│   │   ├── elasticity_2d.jl
+│   │   ├── heat_transfer.jl
+│   │   └── boundary_conditions.jl
+│   ├── 03_nonlinear_problems/
+│   │   ├── large_deformation.jl
+│   │   ├── plasticity.jl
+│   │   └── contact_basics.jl
+│   ├── 04_advanced/
+│   │   ├── contact_2d.jl
+│   │   ├── contact_3d.jl
+│   │   ├── mortar_methods.jl
+│   │   └── friction.jl
+│   └── 05_parallel/
+│       ├── threading.jl
+│       ├── gpu_assembly.jl
+│       └── distributed.jl
+├── unit/               # Fast unit tests (not Literate)
+│   ├── basis/
+│   ├── assembly/
+│   └── solvers/
+├── verification/       # Known analytical solutions
+│   ├── timoshenko_beam.jl
+│   ├── hertz_contact.jl
+│   └── cook_membrane.jl
+└── runtests.jl        # Test runner
+```
+
+### 4. Fast Tests First
+
+**Test pyramid:**
+
+```text
+        /\
+       /  \  Integration tests (slow, few)
+      /____\
+     /      \  Tutorial tests (medium, some)
+    /________\
+   /          \  Unit tests (fast, many)
+  /__________\
+```
+
+**Timing targets:**
+- Unit tests: < 5 minutes (run during development)
+- Tutorial tests: < 15 minutes (run before commits)
+- Full suite: < 30 minutes (run in CI)
+
+**Strategy:**
+
+- Use small meshes in tutorials (10-100 elements, not 10,000)
+- Test correctness with analytical solutions, not big problems
+- Profile and optimize slow tests
+- Mark slow tests with `@testset "slow: hertz_contact"` (skip during dev)
+
+### 5. Coverage-Driven Development
+
+**Target: 99% code coverage**
+
+Every function should have:
+
+1. **Happy path test** - normal usage
+2. **Edge case tests** - empty input, single element, etc.
+3. **Error tests** - what happens with bad input?
+
+**Process:**
+
+1. Write tutorial (covers main API)
+2. Check coverage report
+3. Add unit tests for uncovered lines
+4. Repeat until 99%+
+
+**Tools:**
+
+- Coverage.jl (built into Julia)
+- LocalCoverage.jl (for local checks)
+- Codecov (in CI, we set this up yesterday)
+
+---
+
+## Test Structure Details
+
+### Tutorial Tests (Literate.jl)
+
+**Template:**
+
+```julia
+# # Tutorial Title
+#
+# Brief description of what this tutorial teaches.
+# Prerequisites: what the reader should know first.
+
+using JuliaFEM
+using Test
+
+# ## Section 1: Concept Explanation
+#
+# Explain the concept in prose, with equations if needed:
+# 
+# The strain-displacement relationship is:
+# ```math
+# ε = \frac{1}{2}(∇u + ∇u^T)
+# ```
+
+# Code demonstrating the concept
+element = Element(Quad4, (1,2,3,4))
+
+# ## Section 2: Building Up
+#
+# Step-by-step construction of a working example
+
+# ... code ...
+
+# ## Section 3: Validation
+#
+# Test that the result is correct (this is still a test!)
+
+@testset "Elasticity 2D" begin
+    @test isapprox(u_computed, u_analytical, rtol=1e-6)
+end
+
+# ## Discussion
+#
+# What did we learn? What can we do next?
+# Links to related tutorials.
+```
+
+**Generation:**
+
+```julia
+using Literate
+Literate.markdown("test/tutorials/01_fundamentals/creating_elements.jl", 
+                  "docs/src/tutorials/")
+Literate.notebook("test/tutorials/01_fundamentals/creating_elements.jl",
+                  "docs/notebooks/")
+```
+
+### Unit Tests (Fast Validation)
+
+**Purpose:** Test individual functions in isolation  
+**Style:** Concise, no narrative  
+**Location:** `test/unit/`
+
+```julia
+@testset "Basis Functions - Quad4" begin
+    @testset "Evaluation at ξ=0, η=0" begin
+        N = eval_basis(Quad4, (0.0, 0.0))
+        @test N ≈ [0.25, 0.25, 0.25, 0.25]
+    end
+    
+    @testset "Derivatives" begin
+        dN = eval_dbasis(Quad4, (0.0, 0.0))
+        @test size(dN) == (2, 4)
+    end
+    
+    @testset "Edge case: extreme ξ" begin
+        N = eval_basis(Quad4, (1.0, 1.0))
+        @test N[3] ≈ 1.0
+        @test sum(N) ≈ 1.0
+    end
+end
+```
+
+### Verification Tests (Known Solutions)
+
+**Purpose:** Validate against analytical solutions or published results  
+**Style:** Brief explanation + reference  
+**Location:** `test/verification/`
+
+```julia
+# Timoshenko Beam - Verification Test
+#
+# Reference: Timoshenko & Goodier, "Theory of Elasticity", 3rd Ed.
+# Problem: Cantilever beam with end load
+# Analytical solution available for tip displacement
+
+using JuliaFEM, Test
+
+# Problem parameters from reference
+L = 10.0  # Length
+h = 1.0   # Height
+E = 200e3 # Young's modulus
+ν = 0.3   # Poisson's ratio
+P = 100.0 # End load
+
+# ... setup and solve ...
+
+# Analytical solution
+u_tip_analytical = P*L^3 / (3*E*I)
+
+@testset "Timoshenko Beam Verification" begin
+    @test isapprox(u_tip_computed, u_tip_analytical, rtol=0.01)
+end
+```
+
+---
+
+## Implementation Roadmap
+
+### Phase 1: Infrastructure (Week 1)
+
+**Goal:** Set up Literate.jl integration and test structure
+
+**Tasks:**
+
+1. ✅ Create `test/tutorials/` directory structure
+2. ✅ Add Literate.jl to `docs/Project.toml`
+3. ✅ Update `docs/make.jl` to process tutorials
+4. ✅ Create tutorial template file
+5. ✅ Update `test/runtests.jl` to run tutorials
+6. ✅ Add coverage tools (LocalCoverage.jl)
+
+**Deliverable:** Running CI that executes tutorials and reports coverage
+
+### Phase 2: Core Tutorials (Week 2-3)
+
+**Goal:** Write 10-15 fundamental tutorials covering main API
+
+**Priority order:**
+
+1. Creating elements and updating fields ⭐
+2. Reading meshes (ABAQUS .inp format) ⭐
+3. Basis functions and integration ⭐
+4. Simple 1D elasticity ⭐
+5. 2D plane stress/strain ⭐
+6. Boundary conditions (Dirichlet)
+7. Surface loads and tractions
+8. Heat transfer basics
+9. Assembly process
+10. Solving linear systems
+
+**Success metric:** 60%+ code coverage from tutorials alone
+
+### Phase 3: Advanced Tutorials (Week 4-5)
+
+**Goal:** Cover advanced features
+
+**Topics:**
+
+1. Nonlinear elasticity (large deformation)
+2. Contact mechanics basics (2D)
+3. Mortar methods
+4. 3D contact
+5. Material models
+
+**Success metric:** 80%+ coverage
+
+### Phase 4: Unit Tests (Week 6)
+
+**Goal:** Fill coverage gaps with fast unit tests
+
+**Process:**
+
+1. Generate coverage report: `julia --code-coverage=user test/runtests.jl`
+2. Analyze: `using Coverage; LCOV.writefile("coverage.info", process_folder())`
+3. Find uncovered lines
+4. Write unit tests for each uncovered function
+5. Repeat until 99%+
+
+**Success metric:** 99% coverage, < 5 min unit test runtime
+
+### Phase 5: Verification (Week 7)
+
+**Goal:** Validate against known solutions
+
+**Tests:**
+
+1. Timoshenko beam (bending)
+2. Hertz contact (2D)
+3. Cook's membrane (stress concentration)
+4. Patch tests (element validation)
+5. Manufactured solutions
+
+**Success metric:** All verification tests pass with < 1% error
+
+### Phase 6: Documentation (Week 8)
+
+**Goal:** Polish and publish
+
+**Tasks:**
+
+1. Review all tutorial narratives
+2. Add figures and visualizations
+3. Cross-link tutorials
+4. Build documentation locally
+5. Deploy to GitHub Pages
+6. Write README.md guide to tutorials
+
+**Success metric:** Beautiful, usable documentation website
+
+---
+
+## Test Runner Design
+
+### `test/runtests.jl`
+
+```julia
+using Test, JuliaFEM
+
+# Determine what to run based on environment
+const RUN_UNIT = get(ENV, "JULIAFEM_TEST_UNIT", "true") == "true"
+const RUN_TUTORIALS = get(ENV, "JULIAFEM_TEST_TUTORIALS", "true") == "true"
+const RUN_SLOW = get(ENV, "JULIAFEM_TEST_SLOW", "false") == "true"
+const RUN_VERIFICATION = get(ENV, "JULIAFEM_TEST_VERIFICATION", "true") == "true"
+
+# Fast unit tests (always run)
+if RUN_UNIT
+    @testset "Unit Tests" begin
+        include("unit/basis/test_quad4.jl")
+        include("unit/basis/test_seg2.jl")
+        # ... more unit tests
+    end
+end
+
+# Tutorial tests (run in CI, optional locally)
+if RUN_TUTORIALS
+    @testset "Tutorials" begin
+        # These are also documentation!
+        include("tutorials/01_fundamentals/creating_elements.jl")
+        include("tutorials/01_fundamentals/reading_meshes.jl")
+        include("tutorials/02_linear_problems/elasticity_2d.jl")
+        # ... more tutorials
+    end
+end
+
+# Slow integration tests (CI only by default)
+if RUN_SLOW
+    @testset "Slow Tests" begin
+        include("verification/hertz_contact.jl")
+        # Large mesh tests
+    end
+end
+
+# Verification tests
+if RUN_VERIFICATION
+    @testset "Verification" begin
+        include("verification/timoshenko_beam.jl")
+        include("verification/cook_membrane.jl")
+    end
+end
+```
+
+**Usage:**
+
+```bash
+# During development (fast, < 5 min)
+julia --project=. -e 'using Pkg; Pkg.test()'
+
+# Before commit (< 15 min)
+JULIAFEM_TEST_TUTORIALS=true julia --project=. test/runtests.jl
+
+# Full CI run (< 30 min)
+JULIAFEM_TEST_SLOW=true julia --project=. test/runtests.jl
+
+# Only unit tests (< 2 min)
+JULIAFEM_TEST_TUTORIALS=false julia --project=. test/runtests.jl
+```
+
+---
+
+## Coverage Workflow
+
+### Local Development
+
+```bash
+# 1. Run tests with coverage
+julia --project=. --code-coverage=user test/runtests.jl
+
+# 2. Generate coverage report
+julia --project=. -e '
+using Coverage
+coverage = process_folder()
+covered = length(filter(c -> c.coverage > 0, coverage))
+total = length(coverage)
+println("Coverage: $(round(100*covered/total, digits=2))%")
+LCOV.writefile("coverage.info", coverage)
+'
+
+# 3. View in browser (requires genhtml from lcov package)
+genhtml coverage.info -o coverage/
+firefox coverage/index.html
+```
+
+### GitHub Actions CI
+
+Already set up yesterday! Codecov will:
+
+- Track coverage over time
+- Comment on PRs with coverage changes
+- Show which lines are uncovered
+- Badge in README.md
+
+---
+
+## Writing Style Guide
+
+### For Tutorials (Literate.jl)
+
+**Do:**
+
+- ✅ Explain **why**, not just **what**
+- ✅ Use simple examples (1-4 elements)
+- ✅ Include mathematical notation where helpful
+- ✅ Show output/results
+- ✅ Link to related tutorials
+- ✅ Test the actual result (still a test!)
+
+**Don't:**
+
+- ❌ Assume prior knowledge (explain or link)
+- ❌ Use large meshes (slow tests)
+- ❌ Skip explanation (this is documentation!)
+- ❌ Test implementation details (test behavior)
+
+### For Unit Tests
+
+**Do:**
+
+- ✅ Test one thing per `@testset`
+- ✅ Use descriptive test names
+- ✅ Test edge cases (empty, single, many)
+- ✅ Test error conditions (`@test_throws`)
+- ✅ Be concise (no prose needed)
+
+**Don't:**
+
+- ❌ Mix multiple concepts in one test
+- ❌ Use real-world complex examples
+- ❌ Duplicate tutorial content
+
+---
+
+## Migration Plan for Existing Tests
+
+### Current State (56 test files)
+
+Many old tests are:
+
+- Poorly documented
+- Using outdated API
+- Slow (large meshes)
+- Not structured for learning
+
+### Migration Strategy
+
+**Don't delete old tests immediately!** Instead:
+
+1. **Categorize:** Is it tutorial material, unit test, or verification?
+2. **Rewrite:** Create new version following philosophy
+3. **Verify:** Ensure new test covers same functionality
+4. **Archive:** Move old test to `test/archive/` with note
+5. **Delete:** After new tests run successfully in CI
+
+**Example:**
+
+```text
+test/test_elasticity_2d_linear_with_surface_load.jl  (old)
+  → test/tutorials/02_linear_problems/elasticity_2d.jl  (new, Literate)
+  → test/archive/test_elasticity_2d_linear_with_surface_load.jl.old  (keep for reference)
+  → delete after 1 month if no issues
+```
+
+---
+
+## Success Metrics
+
+### Quantitative
+
+- **Coverage:** 99%+ by end of Phase 4
+- **Speed:** < 5 min unit tests, < 30 min full suite
+- **Count:** 15+ tutorials, 100+ unit tests
+
+### Qualitative
+
+- **Can a new user learn JuliaFEM from tutorials alone?** (Ask someone!)
+- **Are tutorials referenced in issue discussions?** ("See tutorial X")
+- **Do contributors write tests first?** (TDD culture)
+- **Is documentation always up-to-date?** (Literate.jl ensures it)
+
+---
+
+## Questions to Resolve
+
+### 1. Literate.jl Execution Strategy
+
+**Option A:** Tutorials are in `test/tutorials/`, executed by `Pkg.test()`
+- Pro: Ensures tutorials always work
+- Con: Slows down test suite
+
+**Option B:** Tutorials are in `docs/tutorials/`, executed by docs build
+- Pro: Fast test suite
+- Con: Tutorials might break without noticing
+
+**Recommendation:** Option A (tutorials in test/), with `RUN_TUTORIALS` env var
+
+### 2. Visualization in Tutorials
+
+Should tutorials include plots/visualizations?
+
+**Option A:** Yes, using Makie.jl or similar
+
+- Pro: Very educational, shows results
+- Con: Heavy dependency, slow to precompile
+
+**Option B:** No plots in tests, but show code in docs
+
+- Pro: Fast tests
+- Con: Less visual learning
+
+**Recommendation:** Option B initially, add plots later as optional
+
+### 3. Mesh Files
+
+Many old tests use `.med` files (require HDF5). Options:
+
+**Option A:** Convert all to `.inp` (ABAQUS format, no HDF5 needed)
+**Option B:** Generate meshes programmatically in tests
+**Option C:** Add HDF5 as package extension (Julia 1.9+)
+
+**Recommendation:** Option B for tutorials (simple meshes), Option A for verification
+
+---
+
+## Next Steps (This Week)
+
+### Immediate (Today/Tomorrow)
+
+1. ✅ Create this document (done!)
+2. ⏳ Add Literate.jl to docs dependencies
+3. ⏳ Create tutorial directory structure
+4. ⏳ Write first tutorial template
+5. ⏳ Update `test/runtests.jl` for new structure
+
+### This Week
+
+1. Write 3 fundamental tutorials:
+   - Creating elements and fields
+   - Reading ABAQUS mesh
+   - Simple 1D elasticity
+2. Set up coverage workflow locally
+3. Test Literate.jl → HTML generation
+4. Update CI to run tutorials
+
+### Next Week
+
+Continue writing tutorials, aim for 10 total by end of week.
+
+---
+
+## Conclusion
+
+**This is a complete overhaul of testing strategy** - from "fix 49 failing tests" to "build educational test suite that reaches 99% coverage."
+
+**Timeline:** ~8 weeks for full implementation  
+**Effort:** Significant, but creates lasting value  
+**Benefit:** Tests become documentation, documentation is always tested
+
+**Philosophy:** Tests are not a chore - they're the best way to teach users how to use JuliaFEM.
+
+---
+
+**Status:** 📋 Planning complete, ready to implement  
+**Next:** Get your feedback, then start Phase 1