{"as_of":"2026-07-13","reproduce":"python scripts/backtest_comps_avm.py","bar":{"median":0.1,"coverage":0.85,"within20":0.75},"method":"Temporal holdout: targets are qualifying improved sales 2023+ (sampled 1500/county, seed 1729); comps are qualifying sales in the 2yr window before each sale, excluding the parcel itself. Maricopa targets are recorded SALES AFFIDAVITS (assessor arm's-length code, single-parcel, > $100,000, recorded deeds only) joined to parcel attributes by APN; its comp pool stays the parcels table (freshness thins after 2025-Q1 — disclosed, not hidden) with strict guards: no pool sale on the target's APN or deed number, and every pool sale predates the target's sale month. See maricopa_target_basis for the side-by-side.","enabled":["Douglas","Hillsborough","Lancaster","Maricopa","Pinal","Sarpy","Wake"],"markets":[{"market":"Wake","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1422,"coverage":0.9813,"median":0.0493,"within10":0.7314,"within20":0.9114,"mae":0.1351,"rmse":0.7525,"p10":0.0086,"p25":0.0197,"p75":0.1061,"p90":0.186,"bias":0.0692,"band_hit_rate":0.8084,"band_n":736,"band_by_conf":{"medium":0.8151,"high":0.8138,"low":0.619},"by_sale_band":{"1-2M":{"n":91,"median":0.0923},"<1M":{"n":1369,"median":0.0484},"2-3M":{"n":8,"median":0.211},"3M+":{"n":4,"median":0.3261}},"verdict":"PASS"},{"market":"Maricopa","passes":true,"target_source":"affidavit_arms_length","pool_source":"parcels_last_sale","n":1404,"coverage":0.9833,"median":0.066,"within10":0.6531,"within20":0.8925,"mae":0.0978,"rmse":0.1512,"p10":0.0118,"p25":0.031,"p75":0.1275,"p90":0.2068,"bias":0.0137,"band_hit_rate":0.794,"band_n":738,"band_by_conf":{"medium":0.7633,"high":0.8175,"low":0.5625},"by_sale_band":{"<1M":{"n":1365,"median":0.0643},"1-2M":{"n":81,"median":0.1406},"3M+":{"n":12,"median":0.3286},"2-3M":{"n":17,"median":0.1808}},"verdict":"PASS"},{"market":"Hillsborough","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1358,"coverage":0.9633,"median":0.0769,"within10":0.609,"within20":0.8527,"mae":0.1238,"rmse":0.2662,"p10":0.012,"p25":0.031,"p75":0.1478,"p90":0.2457,"bias":0.0404,"band_hit_rate":0.8091,"band_n":723,"band_by_conf":{"high":0.8291,"medium":0.8466,"low":0.4444},"by_sale_band":{"1-2M":{"n":82,"median":0.1367},"<1M":{"n":1341,"median":0.078},"2-3M":{"n":15,"median":0.256},"3M+":{"n":7,"median":0.2155}},"verdict":"PASS"},{"market":"Pinal","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1376,"coverage":0.9567,"median":0.0588,"within10":0.6773,"within20":0.8757,"mae":0.1136,"rmse":0.2877,"p10":0.0094,"p25":0.0243,"p75":0.1237,"p90":0.2271,"bias":0.0246,"band_hit_rate":0.7953,"band_n":718,"band_by_conf":{"high":0.8033,"medium":0.8289,"low":0.4444},"by_sale_band":{"<1M":{"n":1383,"median":0.0583},"2-3M":{"n":16,"median":0.8125},"1-2M":{"n":24,"median":0.3016},"3M+":{"n":12,"median":0.8899}},"verdict":"PASS"},{"market":"Douglas","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1176,"coverage":0.938,"median":0.0921,"within10":0.5357,"within20":0.8053,"mae":0.2195,"rmse":0.6951,"p10":0.0144,"p25":0.0389,"p75":0.1673,"p90":0.3271,"bias":0.1087,"band_hit_rate":0.7997,"band_n":704,"band_by_conf":{"high":0.8555,"low":0.7414,"medium":0.7479},"by_sale_band":{"<1M":{"n":1377,"median":0.0999},"1-2M":{"n":21,"median":0.2023},"2-3M":{"n":5,"median":0.2739},"3M+":{"n":4,"median":0.991}},"verdict":"PASS"},{"market":"Lancaster","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1259,"coverage":0.936,"median":0.0874,"within10":0.5488,"within20":0.8205,"mae":0.1658,"rmse":0.5051,"p10":0.0168,"p25":0.0411,"p75":0.1632,"p90":0.2709,"bias":0.0757,"band_hit_rate":0.7991,"band_n":702,"band_by_conf":{"high":0.819,"low":0.5,"medium":0.8571},"by_sale_band":{"<1M":{"n":1390,"median":0.0924},"1-2M":{"n":14,"median":0.4354}},"verdict":"PASS"},{"market":"Sarpy","passes":true,"target_source":"parcels_last_sale","pool_source":"parcels_last_sale","n":1298,"coverage":0.958,"median":0.0787,"within10":0.5971,"within20":0.8513,"mae":0.1459,"rmse":0.4353,"p10":0.013,"p25":0.0364,"p75":0.1485,"p90":0.2476,"bias":0.0566,"band_hit_rate":0.8067,"band_n":719,"band_by_conf":{"high":0.8352,"medium":0.7843,"low":0.6818},"by_sale_band":{"<1M":{"n":1431,"median":0.0819},"1-2M":{"n":4,"median":0.7147},"2-3M":{"n":2,"median":0.9024}},"verdict":"PASS"}],"maricopa_target_basis":{"market":"Maricopa","served":"affidavit_arms_length","refused":false,"note":"Methodology change: Maricopa backtest targets moved from the parcels-table last-sale feed (collapses after 2025-Q1; 2026 = 2 priced rows) to recorded sales affidavits (assessor arm's-length code — a disclosed heuristic composite — single-parcel, > $100,000, recorded deeds only). Both bases are recorded here at the production seed; the affidavit basis clears the pass bar and serves the headline, the parcels basis stays visible as methodology history.","results":{"parcels_last_sale":{"n":1394,"attempted":1500,"coverage":0.976,"median":0.0653,"within10":0.6435,"within20":0.8694,"bias":0.0005,"passes":true},"affidavit_arms_length":{"n":1404,"attempted":1500,"coverage":0.9833,"median":0.066,"within10":0.6531,"within20":0.8925,"bias":0.0137,"passes":true}}},"content_hash":"c175c3ea0d16befb","rent":{"as_of":"2026-07-14","mf":{"n":1561,"median":0.123,"within10":0.4061,"within20":0.6893,"mae":0.2069,"bias":0.0704,"p25":0.0592,"p75":0.237,"p90":0.4801,"method":"leave-one-out CV over 1561 assessor-rent-recorded Phoenix complexes: predict each complex's GPR as (median $/sqft/yr of ALL OTHER complexes) x its complex-aggregated building sqft; err=(gpr_hat-gpr)/gpr. Rent-roll gate priced/counted>=0.75; $/sqft/yr band 8-40; sqft summed across sub-parcels by complex_name (calibrate_mf_caps conventions).","as_of":"2026-07-14"},"unit_type":{"studio":{"n":535,"median":0.1991,"within10":0.2935,"within20":0.5084,"mae":0.2771,"bias":0.0701,"p25":0.0834,"p75":0.3052,"p90":0.4331,"median_rent":1076},"one_br":{"n":1602,"median":0.1579,"within10":0.3246,"within20":0.6217,"mae":0.2912,"bias":0.1476,"p25":0.0763,"p75":0.2739,"p90":0.5277,"median_rent":1296},"two_br":{"n":1665,"median":0.1558,"within10":0.3255,"within20":0.615,"mae":0.3034,"bias":0.1321,"p25":0.0757,"p75":0.2795,"p90":0.6119,"median_rent":1520},"three_br":{"n":786,"median":0.1355,"within10":0.3817,"within20":0.6692,"mae":0.2439,"bias":0.0803,"p25":0.0625,"p75":0.2492,"p90":0.5295,"median_rent":2025}},"sfr_twin":{"n":100,"median":0.1041,"within10":0.47,"within20":0.74,"mae":0.1679,"bias":-0.099,"p25":0.0496,"p75":0.2092,"p90":0.4067,"live_calls":0,"sampled":100,"city":"Phoenix, AZ","seed":1729,"as_of":"2026-07-14","method":"AVM rent_estimate vs ASKING rent of active long-term SFR rental listings (Phoenix AZ, <= 2 pages of 500, deterministic sample seed 1729, n <= 100); err = (rent_hat - asking) / asking.","caveat":"ASKING-rent CONSISTENCY twin: the estimator can see active listings (in-sample) and asking rents are not signed leases — this is NOT out-of-sample signed-lease accuracy."},"sfr_twin_status":"measured","method":"leave-one-out vs n=1,561 assessor-recorded complexes","reproduce":"PYTHONPATH=. python3 scripts/backtest_rent_accuracy.py"},"teardown_recall":{"as_of":"2026-07-13","baseline":{"auc":0.787,"scottsdale_recall_250":0.12},"n_candidates":476457,"n_positives":800,"base_rate":0.00168,"auc_static":0.8018,"auc_spatial":0.783,"overall_static":{"250":{"recall":0.0187,"hits":15,"precision":0.06},"500":{"recall":0.0262,"hits":21,"precision":0.042},"1000":{"recall":0.0387,"hits":31,"precision":0.031},"2500":{"recall":0.0587,"hits":47,"precision":0.0188}},"scottsdale":{"n_candidates":40394,"n_positives":135,"static":{"100":{"recall":0.0963,"hits":13,"precision":0.13},"250":{"recall":0.1852,"hits":25,"precision":0.1},"500":{"recall":0.2148,"hits":29,"precision":0.058}},"spatial":{"100":{"recall":0.0815,"hits":11,"precision":0.11},"250":{"recall":0.1333,"hits":18,"precision":0.072},"500":{"recall":0.1556,"hits":21,"precision":0.042}}},"method":"5-fold stratified OOF (seed 42) HistGradientBoosting over Maricopa old-SFR (built 1900-1990) candidates; labels = permit-confirmed demolitions (az__city_permits demo class + az__shovels_demolitions); spatial-density leg recomputed per fold from training-fold positives only (leave-one-out) so no label leaks into its own fold's feature.","reproduce":"PYTHONPATH=. python3 scripts/teardown_classifier.py emit"},"bid_calibration":{"as_of":"2026-07-10","market":"Maricopa","strategy":"value_add","target_roi":0.15,"method":"leakage-safe comps-AVM (accuracy-backtest machinery, temporal precedence + same-APN/deed exclusion) -> the served deal shape (screening cost model) -> breakeven_offer inversion; ratio = recommended max offer / recorded arm's-length sale price","interpretation":"CALIBRATION, not realized profit: P(offer>=price) is the share of real transactions a buyer following the recommendation could have won while the model held its 15% return-on-cost target. Realized rehab scope / exit / hold outcomes are unobserved.","conventions":"attributes from the current assessor snapshot (accuracy-backtest convention); screening costs, not quotes; high/medium confidence only (the serving trust policy)","seed":"1729","seed_agreement":{"1729":0.9256,"42":0.925},"overall":{"n":18852,"median":0.9256,"p25":0.8598,"p75":0.9927,"p_offer_at_or_above_price":0.2325},"by_year":{"2023":{"n":5917,"median":0.9111,"p25":0.8503,"p75":0.97,"p_offer_at_or_above_price":0.1746},"2024":{"n":5896,"median":0.9254,"p25":0.8638,"p75":0.9811,"p_offer_at_or_above_price":0.1981},"2025":{"n":5205,"median":0.9407,"p25":0.8677,"p75":1.0196,"p_offer_at_or_above_price":0.2999},"2026":{"n":1834,"median":0.948,"p25":0.8606,"p75":1.0369,"p_offer_at_or_above_price":0.3391}},"by_confidence":{"high":{"n":14309,"median":0.9288,"p25":0.8704,"p75":0.99,"p_offer_at_or_above_price":0.2242},"medium":{"n":4543,"median":0.9064,"p25":0.8212,"p75":1.0038,"p_offer_at_or_above_price":0.2589}},"reproduce":"PYTHONPATH=. python3 scripts/measure_bid_validation.py"},"cost_outcomes":{"n_quotes":22,"n_realized_pairs":0,"method":"latest production quote joined to the latest customer-reported outcome per parcel (/feedback/outcome); errors only when both sides of a pair exist","status":"staged — quotes are being captured in production; no customer-reported outcome has paired with one yet, so no error is claimable"}}