Skip to content

DS-MSP[rig] — config-driven evaluation on MC-Calib Blender datasets

Every row below is produced by writing a real MC-Calib-compatible calib_param.yml (under Blender_Images/configs/) and running it through ds-msp-calibrate-rig --config <file> — no in-process shortcuts.

Note

Blender_Images is MC-Calib's Blender ground-truth dataset (about 1 GB) and isn't bundled with this repo, so reproducing these numbers needs it fetched separately. For the counterpart that drives the same scenarios in-process instead of through configs, see Rig evaluation table.

Each scenario runs in three modes:

  • given — the default given intrinsics, held fixed with no optimization (camera_models: radtan, cam_params_path set, fix_intrinsic: true). Extrinsics-only.
  • dsplus — calibrated from scratch with DS+ (camera_models: dsplus, cam_params_path: None, fix_intrinsic: false). Intrinsics and extrinsics estimated jointly, with no prior intrinsics at all.
  • dsplus_seeded — DS+ again, but seeded from the same given intrinsics (cam_params_path set, fix_intrinsic: false). convert() seeds DS+ from the original lens, then intrinsics and extrinsics are refined jointly from that seed.

Both dsplus and dsplus_seeded reuse the pre-detected 2D keypoints (keypoints_path), so only the rig reconstruction and bundle-adjustment math is exercised — not the corner detector.

Two error metrics recur through every table:

  • base%GT — worst inter-camera baseline error vs ground-truth extrinsics.
  • foc%GT — paraxial-focal error vs ground-truth intrinsics (model-independent).

Example: one config, one command

Every scenario has a real config checked into Blender_Images/configs/. Running the Scenario_1 given config directly reproduces that row of the summary table below:

$ ds-msp-calibrate-rig --config Blender_Images/configs/Scenario_1.given.yml --quiet --no-webviewer
=== Scenario_1.given.yml: 2 cameras, 3 board(s), 2 calibrated ===
per-camera model: {0: 'radtan', 1: 'radtan'}
worst baseline error vs GroundTruth : 0.012%

 cam  model              n    mean  median     p95     max     rms  inlier%
---------------------------------------------------------------------------
   0  radtan          3228   0.060   0.046   0.146   0.516   0.085  100.00%
   1  radtan          3061   0.064   0.050   0.153   0.459   0.085  100.00%
---------------------------------------------------------------------------
 all                  6289   0.062   0.048   0.149   0.516   0.085  100.00%

verdict: PASS  median 0.048px, p95 0.149px <= 1.00/3.00px

wrote MC-Calib-format output to: Blender_Images/Scenario_1/Results_dsmsp_given

That matches the Scenario_1 given row below: 0.085 px per-camera RMS, 0.012% worst baseline error vs ground truth.

Summary (one row per scenario × mode)

dataset ncam mode model fix max rms px mean rms px worst base%GT max foc%GT median foc%GT
Scenario_1 2 given radtan true 0.085 0.085 0.012 4.102 2.340
Scenario_1 2 dsplus dsplus false 0.085 0.085 0.013 4.154 3.068
Scenario_1 2 dsplus_seeded dsplus false 0.085 0.085 0.014 4.099 2.362
Scenario_2 5 given radtan true 0.090 0.072 0.029 0.639 0.575
Scenario_2 5 dsplus dsplus false 0.090 0.072 0.024 0.707 0.618
Scenario_2 5 dsplus_seeded dsplus false 0.090 0.072 0.021 1.038 0.689
Scenario_3 4 given radtan true 0.066 0.046 0.131 4.106 2.345
Scenario_3 4 dsplus dsplus false 0.067 0.046 0.156 4.108 2.381
Scenario_3 4 dsplus_seeded dsplus false 0.067 0.046 0.156 4.120 2.395
Scenario_4 4 given radtan true 0.097 0.058 0.102 4.093 2.333
Scenario_4 4 dsplus dsplus false 0.097 0.058 0.072 4.177 2.338
Scenario_4 4 dsplus_seeded dsplus false 0.097 0.058 0.080 4.096 2.489
Scenario_5 4 given radtan true 0.704 0.413 0.010 0.596 0.592
Scenario_5 4 dsplus dsplus false 0.706 0.414 0.005 0.614 0.589
Scenario_5 4 dsplus_seeded dsplus false 0.706 0.413 0.004 0.595 0.592

Per-camera detail

dataset mode cam model rms px base%GT foc%GT
Scenario_1 given 0 radtan 0.085 — 4.102
Scenario_1 given 1 radtan 0.085 0.012 0.577
Scenario_1 dsplus 0 dsplus 0.085 — 4.154
Scenario_1 dsplus 1 dsplus 0.085 0.013 1.981
Scenario_1 dsplus_seeded 0 dsplus 0.085 — 4.099
Scenario_1 dsplus_seeded 1 dsplus 0.085 0.014 0.624
Scenario_2 given 0 radtan 0.077 — 0.575
Scenario_2 given 1 radtan 0.073 0.016 0.607
Scenario_2 given 2 radtan 0.054 0.003 0.639
Scenario_2 given 3 radtan 0.066 0.029 0.525
Scenario_2 given 4 radtan 0.090 0.011 0.521
Scenario_2 dsplus 0 dsplus 0.078 — 0.580
Scenario_2 dsplus 1 dsplus 0.073 0.024 0.592
Scenario_2 dsplus 2 dsplus 0.054 0.015 0.633
Scenario_2 dsplus 3 dsplus 0.066 0.006 0.707
Scenario_2 dsplus 4 dsplus 0.090 0.006 0.618
Scenario_2 dsplus_seeded 0 dsplus 0.078 — 0.689
Scenario_2 dsplus_seeded 1 dsplus 0.073 0.021 0.834
Scenario_2 dsplus_seeded 2 dsplus 0.054 0.013 0.668
Scenario_2 dsplus_seeded 3 dsplus 0.066 0.007 0.618
Scenario_2 dsplus_seeded 4 dsplus 0.090 0.001 1.038
Scenario_3 given 0 radtan 0.038 — 4.101
Scenario_3 given 1 radtan 0.043 0.033 4.106
Scenario_3 given 2 radtan 0.037 0.131 0.590
Scenario_3 given 3 radtan 0.066 0.022 0.587
Scenario_3 dsplus 0 dsplus 0.038 — 4.102
Scenario_3 dsplus 1 dsplus 0.043 0.033 4.108
Scenario_3 dsplus 2 dsplus 0.037 0.156 0.641
Scenario_3 dsplus 3 dsplus 0.067 0.036 0.660
Scenario_3 dsplus_seeded 0 dsplus 0.038 — 4.102
Scenario_3 dsplus_seeded 1 dsplus 0.043 0.035 4.120
Scenario_3 dsplus_seeded 2 dsplus 0.037 0.156 0.592
Scenario_3 dsplus_seeded 3 dsplus 0.067 0.031 0.688
Scenario_4 given 0 radtan 0.045 — 4.086
Scenario_4 given 1 radtan 0.036 0.102 4.093
Scenario_4 given 2 radtan 0.097 0.031 0.579
Scenario_4 given 3 radtan 0.053 0.003 0.580
Scenario_4 dsplus 0 dsplus 0.045 — 4.177
Scenario_4 dsplus 1 dsplus 0.036 0.072 4.090
Scenario_4 dsplus 2 dsplus 0.097 0.027 0.575
Scenario_4 dsplus 3 dsplus 0.053 0.003 0.587
Scenario_4 dsplus_seeded 0 dsplus 0.045 — 4.089
Scenario_4 dsplus_seeded 1 dsplus 0.036 0.080 4.096
Scenario_4 dsplus_seeded 2 dsplus 0.097 0.026 0.890
Scenario_4 dsplus_seeded 3 dsplus 0.053 0.003 0.581
Scenario_5 given 0 radtan 0.704 — 0.583
Scenario_5 given 1 radtan 0.703 0.006 0.596
Scenario_5 given 2 radtan 0.094 0.010 0.590
Scenario_5 given 3 radtan 0.150 0.001 0.594
Scenario_5 dsplus 0 dsplus 0.706 — 0.586
Scenario_5 dsplus 1 dsplus 0.704 0.003 0.592
Scenario_5 dsplus 2 dsplus 0.094 0.005 0.614
Scenario_5 dsplus 3 dsplus 0.150 0.002 0.585
Scenario_5 dsplus_seeded 0 dsplus 0.706 — 0.593
Scenario_5 dsplus_seeded 1 dsplus 0.704 0.003 0.595
Scenario_5 dsplus_seeded 2 dsplus 0.094 0.004 0.590
Scenario_5 dsplus_seeded 3 dsplus 0.150 0.002 0.587

Reference parity — DS-MSP[dsplus] vs MC-Calib's own published reprojection

Same 2D corners for both. MC rms/MC med come from each scenario's shipped Results/reprojection_error_data.yml.

Our per-camera RMS matches MC-Calib's — including the cameras whose RMS is inflated by a few outlier corners (low median, high RMS) present in the shared detections.

dataset cam ours rms px MC rms px ours/MC MC median px
Scenario_1 0 0.085 0.084 1.00× 0.047
Scenario_1 1 0.085 0.086 0.99× 0.051
Scenario_2 0 0.078 0.077 1.01× 0.053
Scenario_2 1 0.073 0.073 1.01× 0.045
Scenario_2 2 0.054 0.054 1.00× 0.038
Scenario_2 3 0.066 0.066 1.00× 0.040
Scenario_2 4 0.090 0.090 1.00× 0.057
Scenario_3 0 0.038 0.039 0.98× 0.028
Scenario_3 1 0.043 0.045 0.97× 0.031
Scenario_3 2 0.037 0.038 0.97× 0.029
Scenario_3 3 0.067 0.066 1.01× 0.038
Scenario_4 0 0.045 0.049 0.92× 0.031
Scenario_4 1 0.036 0.036 0.98× 0.028
Scenario_4 2 0.097 0.090 1.07× 0.039
Scenario_4 3 0.053 0.056 0.95× 0.036
Scenario_5 0 0.706 0.690 1.02× 0.057
Scenario_5 1 0.704 0.693 1.02× 0.054
Scenario_5 2 0.094 0.097 0.97× 0.036
Scenario_5 3 0.150 0.150 1.00× 0.035

Extrinsics: original intrinsics held fixed vs DS+ seeded and optimized

Both given and dsplus_seeded start from the same authors' original intrinsics.

  • given holds them fixed (radtan, fix_intrinsic: true) and solves extrinsics only.
  • dsplus_seeded selects DS+ with fix_intrinsic: false. convert() seeds DS+ from that original lens, then the bundle adjustment refines intrinsics and extrinsics jointly.

base%GT is the worst inter-camera baseline error vs ground-truth extrinsics — lower is better.

dataset given base%GT (fixed) dsplus_seeded base%GT (optimized) Δ base%GT given max rms dsplus_seeded max rms
Scenario_1 0.012 0.014 +0.001 0.085 0.085
Scenario_2 0.029 0.021 -0.009 0.090 0.090
Scenario_3 0.131 0.156 +0.025 0.066 0.067
Scenario_4 0.102 0.080 -0.022 0.097 0.097
Scenario_5 0.010 0.004 -0.006 0.704 0.706

Worst extrinsic baseline error vs ground truth: 0.131% with the original intrinsics held fixed, 0.156% with DS+ seeded and optimized.

Tip

Both stay far below the 1% bar. Optimizing intrinsics in DS+ — seeded from the original lens — keeps extrinsics at essentially the same ground-truth accuracy as trusting the original intrinsics outright.

On the foc%GT ≈ 4% rows (an observability limit, not an error)

A few cameras show a focal error vs ground truth of about 4% in both modes.

That is not a DS-MSP deficiency. On exactly those cameras, the scenario's own prior MC-Calib calibration — the intrinsics given mode holds fixed — already deviates from ground truth by up to 4.11%.

Those camera views simply don't constrain the focal, a known property of these Blender camera placements. The focal is inherently unrecoverable from that view, regardless of solver.

The decisive check: DS+ calibrated from scratch lands at the same ~4% gap as the given MC-Calib intrinsics. Per-camera foc%GT agrees to within 0.1% between modes — DS+ is exactly as close to ground truth as the established MC-Calib reference.

Tip

Where the focal is observable, every camera recovers it to under 0.7% of ground truth.

Overall: PASS

Worst extrinsic baseline error vs ground truth: 0.156% (pass, threshold <1%). Worst reprojection RMS: 0.706 px (pass, threshold <1 px).

Across every Blender scenario, with the given intrinsics held fixed, with DS+ estimated from scratch, and with DS+ seeded from the given intrinsics, DS-MSP[rig] recovers extrinsics within 1% of ground truth at sub-pixel reprojection.

All three modes agree to within 0.001 px mean RMS.