Star history, last 90 days
Date	Stars
2026-07-31	251
2026-08-05	251
2026-08-06	251

AutoQuant Reference Manual

Companion Packages:

Table of Contents

Background
Installation

Documentation + Code Examples

Supervised Learning
Model Scoring
Model Evaluation
Panel Data Forecasting
Time Series Forecasting

Architecture + Redesign Notes

AutoQuant vNext Archaeology And Design describes proposed next-generation contracts for supervised learning, scoring, forecasting, panel forecasting, typed artifacts, Rodeo ownership boundaries, AnalyticsShinyApp integration, and H2O retirement strategy. This is design documentation, not an implemented API manual.
AutoQuant vNext CatBoost Supervised Learning documents the first implemented vNext vertical slices: CatBoost regression and binary classification specs, deterministic fit, prediction artifacts, canonical scoring artifacts, delayed outcome attachment, realized assessment, bounded monitoring evidence, threshold policy, and installed-package QA.
AutoQuant vNext Forecasting Foundation documents naive, seasonal naive, ETS, ARIMA, CatBoost forecasting, panel forecasting, hierarchy reconciliation, panel strategy comparison, intermittent-demand forecasting, and rolling-origin evidence.
AutoQuant vNext Intermittent Demand Forecasting documents Croston, SBA, TSB, supervised Hurdle, diagnostics, method comparison, and advisory selection evidence.
AutoQuant vNext Funnel Forecasting Foundation documents funnel specification, explicit transitions, maturity evidence, stage forecasting, transition forecasting, assessment, and strategy comparison.
AutoQuant vNext Multi-Target Forecasting Foundation documents multi-target specification, shared temporal preparation evidence, target-level forecasts, cross-target evidence, assessment, and strategy comparison.
AutoQuant vNext Cross-Target Forecasting documents supervised CatBoost cross-target feature learning, Rodeo-owned leakage-safe feature preparation, negative-transfer evidence, and advisory strategy comparison.
AutoQuant vNext Forecasting Capability Planning documents deterministic forecasting capability discovery, evidence-guided strategy planning, planning artifacts, and the historical CatBoost CARMA mechanism inventory for future feature-tuning experiments.
Causal Intelligence Philosophy explains why causal planning is separate from prediction, association, and effect estimation.
Causal Intelligence Framework documents Phase 1 causal questions, estimands, question-relative roles, graph diagnostics, adjustment guidance, identification planning, design eligibility, and causal planning artifacts.
Causal Experiment Design Framework documents Phase 2 governed experiment questions, design specs, deterministic assignment proposals, balance diagnostics, power/timing/measurement plans, validity threats, approval gates, information value, and experiment plan artifacts.
Causal Completed-Experiment Evidence Framework documents Phase 3 completed-experiment evidence ingestion, assignment preservation, execution reconciliation, integrity/fidelity diagnostics, estimand preservation, and analysis-readiness classification without estimating effects.
Causal Randomized ITT Estimation Framework documents Phase 4 governed randomized intent-to-treat estimation, readiness gating, uncertainty, guardrails, materiality, sensitivity, effect artifacts, lifecycle review, and prohibited claims.
Causal Randomized Design Analysis Framework documents Phase 5 randomized-analysis depth, governed variance reduction, design-aware method eligibility, outcome windows, carryover, multiplicity, guardrail decisions, robustness, and causal report contracts.
Observational Causal Planning Framework documents observational study design, target-trial thinking, assignment-mechanism evidence, temporal eligibility, overlap, balance, selection, unmeasured-confounding risk, estimator-family readiness, and planning artifacts.
Governed Observational Estimation Framework implements the first conservative observational estimator slice: approved binary-treatment ATE/ATT analysis with logistic propensity models, nearest-neighbor matching diagnostics, deterministic weights, independent balance validation, AIPW estimation, sensitivity reminders, and claim governance.
Governed Difference-in-Differences Framework implements the first time-based observational estimator slice: approved classic two-group DiD with frozen intervention timing, pre-period diagnostics, parallel-trends assessment, composition stability, sensitivity reminders, and claim governance.

AutoQuant vNext Cookbook

The vNext API is the artifact-first redesign path. It is additive and does not remove the legacy AutoQuant wrappers. The examples below are copy-paste oriented and correspond to validated scripts in inst/examples/.

vNext Quick Start

Details

Click to expand

library(data.table)
library(AutoQuant)

dt <- data.table(
  id = 1:80,
  spend = runif(80, 10, 100),
  clicks = rpois(80, 40),
  channel = sample(c("Search", "Social", "Email"), 80, TRUE)
)
dt[, revenue := 25 + 1.8 * spend + 0.7 * clicks + rnorm(.N, 0, 5)]

spec <- aq_model_spec(
  task = "regression",
  target = "revenue",
  features = c("spend", "clicks", "channel"),
  engine_params = list(iterations = 10, depth = 2),
  dataset_id = "example_training_data"
)

fit <- aq_fit_model(spec, dt)
score <- aq_score_model(fit, dt[1:10], row_id_cols = "id", outcome_col = "revenue")

Validated script: inst/examples/vnext_supervised_learning.R

Causal Intelligence Planning

Causal Intelligence Phase 1 is planning-only. It helps define the causal question, estimand, roles, graph assumptions, adjustment guidance, identification concerns, candidate designs, and a canonical planning artifact. It does not estimate treatment effects.

question <- aq_causal_question(
  question = list(
    causal_question_id = "cq_tv_revenue",
    decision_context_id = "decision_tv_budget",
    exposure = "tv_spend",
    outcome = "revenue",
    population = "eligible markets",
    unit_of_analysis = "market-week",
    time_zero = "campaign start",
    treatment_window = "next quarter",
    outcome_window = "same quarter and four-week lag",
    comparison_condition = "current weekly TV spend range",
    intervention_definition = "increase weekly TV spend from current market range to proposed eligible-market range",
    estimand = "ATE",
    effect_scale = "incremental revenue difference",
    target_population = "eligible markets"
  )
)

Validated script: inst/examples/causal_intelligence_framework.R

Causal Randomized ITT Estimation

Causal Intelligence Phase 4 is the first randomized estimator slice. It runs only randomized intent-to-treat analyses after completed-experiment evidence is classified as ITT-compatible. The first observational adjustment estimator slice is deliberately narrow: it runs only after observational planning/readiness evidence is approved, supports binary-treatment ATE/ATT, and preserves matching, weighting, balance, AIPW, assumptions, sensitivity reminders, and prohibited claims. The first time-based observational slice supports only classic two-group Difference-in-Differences after both Phase 1 readiness and DiD-specific readiness pass. These slices do not estimate treatment-on-treated effects, IV, mediation, synthetic controls, causal forests, heterogeneity, survival/count outcomes, staggered DiD, event studies, generalized TWFE, or optimization.

spec <- aq_randomized_itt_spec(
  completed_experiment_id = "ce_itt",
  experiment_plan_artifact_id = "plan_itt",
  causal_question_id = "cq_itt",
  estimand_id = "estimand_itt",
  treatment_arm = "treatment",
  comparison_arm = "control",
  outcome = "revenue",
  outcome_type = "continuous",
  baseline_covariates = "baseline_y",
  minimum_meaningful_effect = 1
)

gate <- aq_validate_randomized_itt_readiness(
  spec,
  completed_evidence,
  planned_analysis = planned_analysis_record,
  baseline_data = baseline_data
)

result <- aq_estimate_randomized_itt(
  spec,
  completed_evidence,
  baseline_data = baseline_data,
  planned_analysis = planned_analysis_record
)
artifact <- aq_randomized_itt_effect_artifact(result)

Validated script: inst/examples/causal_randomized_itt_framework.R

Causal Randomized Design Analysis

Causal Intelligence Phase 5 deepens randomized evidence after ITT estimation. It preserves the unadjusted assignment effect while adding design-aware analysis, governed ANCOVA, CUPED-style variance reduction, blocked/stratified evidence, cluster diagnostics, randomization inference, multiplicity, guardrail decisions, robustness, and a causal-effect report contract.

design_spec <- aq_randomized_design_analysis_spec(
  itt_analysis_id = result$spec$analysis_id,
  design_type = "blocked_randomized",
  analysis_modes = c("unadjusted", "ancova", "cuped", "blocked", "randomization_inference"),
  block_fields = "block",
  cluster_unit = "market",
  pre_period_fields = "pre_revenue",
  material_benefit = 1,
  material_harm = -1,
  multiplicity_policy = "holm"
)

depth <- aq_analyze_randomized_design_depth(result, design_spec)
report <- aq_randomized_causal_effect_report(result, depth)

Validated script: inst/examples/causal_randomized_design_analysis_framework.R

Causal Experiment Design Planning

Causal Experiment Design Phase 2 extends causal planning into governed experiment-design artifacts. It does not execute treatment and does not estimate causal effects.

experiment_question <- aq_experiment_question(list(
  experiment_question_id = "eq_budget_test",
  causal_question_id = "cq_budget_revenue",
  decision_context_id = "decision_budget",
  hypothesis = "Increasing budget raises revenue.",
  null_claim = "No revenue lift.",
  alternative_claim = "Revenue lift is positive.",
  treatment = "approved budget increase",
  comparison = "current budget",
  estimand = "ATE",
  assignment_population = "eligible markets",
  expected_mechanism = "More qualified reach creates more revenue.",
  primary_outcome = "revenue",
  guardrails = "cpa,customer_quality",
  decision_rule = "Approve rollout if lift exceeds cost and guardrails hold.",
  authority = "marketing_approval",
  coverage = "eligible_markets"
), causal_context)

design <- aq_experiment_design_spec(
  experiment_question,
  design_type = "stratified_randomized",
  assignment_unit = "market",
  analysis_unit = "market-week",
  stratification_variables = "region"
)

Validated script: inst/examples/causal_experiment_design_framework.R

Causal Completed-Experiment Evidence

Causal Intelligence Phase 3 ingests completed or in-progress experiment evidence and classifies analysis readiness. It preserves original assignment as the default ITT anchor, records treatment delivery, exposure, compliance, outcomes, guardrails, exclusions, and plan-versus-execution deviations, then creates a completed-experiment evidence artifact. It does not estimate treatment effects.

completed <- aq_completed_experiment(list(
  completed_experiment_id = "ce_budget_test",
  experiment_plan_artifact_id = "aq_experiment_plan_budget_test",
  decision_context_id = "decision_budget",
  causal_question_id = "cq_budget_revenue",
  estimand_id = "estimand_budget_itt",
  design_version = "v1",
  assignment_version = "v1",
  experiment_status = "completed",
  actual_start_date = "2026-01-01",
  actual_end_date = "2026-02-01",
  data_cutoff_date = "2026-02-15",
  execution_owner = "analytics"
))

assignment <- aq_assignment_evidence(data.table::data.table(
  unit_id = paste0("market_", 1:12),
  planned_arm = rep(c("control", "treatment"), each = 6),
  realized_assigned_arm = rep(c("control", "treatment"), each = 6)
))

outcomes <- aq_outcome_evidence(data.table::data.table(
  unit_id = paste0("market_", 1:12),
  outcome_id = "revenue",
  value = seq(100, 155, by = 5),
  outcome_role = "primary"
))

readiness <- aq_assess_experiment_analysis_readiness(
  completed,
  assignment_evidence = assignment,
  outcome_evidence = outcomes
)

Validated script: inst/examples/causal_completed_experiment_framework.R

Supervised Learning

Use aq_model_spec() to create an explicit supervised model contract, then aq_fit_model() to train. Current vNext supervised operators support CatBoost regression and binary classification. The returned fit includes the model, fit metrics, partition evidence, model metadata, and canonical artifacts.

binary_spec <- aq_model_spec(
  task = "binary",
  target = "converted",
  features = c("spend", "clicks", "channel"),
  positive_class = "yes",
  engine_params = list(iterations = 10, depth = 2)
)

binary_fit <- aq_fit_model(binary_spec, training_data)

Scoring and Realized Outcomes

Use aq_score_model() for canonical scoring artifacts. If outcomes are available at scoring time, pass outcome_col. Delayed outcomes can be attached later through the vNext scoring lifecycle helpers described in docs/vnext_catboost_regression.md.

scored <- aq_score_model(
  fit,
  new_data,
  row_id_cols = "customer_id",
  outcome_col = "actual_outcome",
  dataset_id = "campaign_scoring_population"
)

Model Bundles

Use model bundles when the same fitted model must be reloaded for reproducible inference.

bundle <- aq_save_model_bundle(fit, "exports/model_bundle", overwrite = TRUE)
loaded <- aq_load_model_bundle("exports/model_bundle")
rescored <- aq_score_model(loaded, new_data, row_id_cols = "customer_id")

Canonical Analytical Artifacts

vNext results carry canonical artifacts so downstream apps, reports, campaigns, and agents can consume evidence without reverse-engineering return objects. See docs/canonical_analytical_artifacts.md and inst/examples/artifact_schema_example.R.

Canonical Variable Semantics

Use aq_variable_semantics() to describe what variables mean across business role, operational eligibility, analytical role, causal role, operational, temporal, forecasting, measurement, and decision dimensions. Variable semantics are shared analytical knowledge for future feature engineering, modeling, forecasting, reporting, campaign reasoning, optimization planning, risk review, and GenAI guidance. The semantic artifact records confidence, evidence source, business-context references for future objectives, strategies, tactics, levers, KPIs, risks, and decisions, validation diagnostics, and supported downstream actions.

semantics <- aq_variable_semantics(
  variables = c("revenue", "spend", "region", "promo", "inventory_cap"),
  business_role = list(
    revenue = "objective_metric",
    spend = c("tactic_lever", "business_lever"),
    inventory_cap = "risk_indicator"
  ),
  operational_eligibility = list(spend = c("controllable", "optimization_eligible")),
  analytical_role = list(revenue = "target", promo = "known_future_regressor"),
  causal_role = list(spend = "exposure", region = "confounder_candidate"),
  operational = list(revenue = "outcome", spend = "driver", region = "control"),
  temporal = list(revenue = "future_unknown", promo = "future_known"),
  causal = list(spend = "exposure", region = "possible_confounder"),
  forecasting = list(revenue = "target", promo = "known_future_regressor"),
  decision = list(revenue = "primary_kpi", spend = "optimization_eligible"),
  confidence = list(revenue = 0.95, spend = 0.8, region = 0.7, promo = 0.9),
  business_context_refs = list(
    mission_id = "mission_growth",
    business_objective_id = "objective_revenue_growth",
    strategy_id = "strategy_media_efficiency",
    tactic_id = "tactic_budget_allocation",
    lever_id = "lever_paid_media_spend",
    risk_id = "risk_inventory_capacity"
  )
)

validation <- aq_validate_variable_semantics(semantics)
artifact <- aq_variable_semantics_artifact(semantics)

Validated script: inst/examples/variable_semantics_framework.R

Business Intent and Lever Management

Use aq_business_intent() to describe the organizational layer above variables: mission, objective, strategy, tactic, lever, KPI, guardrail, constraint, risk, assumption, recommendation, decision, authority, and coverage. This contract keeps organizational intent separate from variable semantics while linking them through deterministic relationships and variable-to-lever mappings.

The framework can assess structural strategy/tactic/measurement alignment, classify levers and assumptions as explore/exploit candidates, distinguish capability from authority, and create a canonical business_intent_artifact. It does not implement optimization, causal estimation, MMM, reinforcement learning, or automatic strategy changes.

semantics <- aq_variable_semantics(
  variables = c("revenue", "paid_search_spend", "planned_paid_search_spend", "inventory_cap"),
  business_role = list(
    revenue = c("objective_metric", "measurement_kpi"),
    paid_search_spend = c("tactic_lever", "business_lever", "cost"),
    planned_paid_search_spend = c("tactic_lever", "business_lever"),
    inventory_cap = c("constraint", "guardrail", "risk_indicator")
  ),
  operational_eligibility = list(
    paid_search_spend = c("controllable", "optimization_eligible", "experiment_eligible"),
    planned_paid_search_spend = c("controllable", "scenario_only"),
    inventory_cap = "non_controllable"
  ),
  analytical_role = list(
    revenue = "target",
    paid_search_spend = "predictor",
    planned_paid_search_spend = "known_future_regressor",
    inventory_cap = "predictor"
  ),
  causal_role = list(
    paid_search_spend = "exposure",
    revenue = "outcome",
    inventory_cap = "moderator"
  ),
  decision = list(
    revenue = "primary_kpi",
    paid_search_spend = "optimization_eligible",
    inventory_cap = "guardrail"
  )
)

intent <- aq_business_intent(
  missions = list(
    mission_id = "mission_growth",
    title = "Sustainable Growth"
  ),
  objectives = list(
    objective_id = "objective_revenue_growth",
    mission_id = "mission_growth",
    title = "Increase Revenue",
    primary_kpis = list("kpi_revenue")
  ),
  strategies = list(
    strategy_id = "strategy_qualified_demand",
    objective_id = "objective_revenue_growth",
    title = "Increase Qualified Demand",
    intended_mechanism = "Increase awareness and intent among eligible prospects."
  ),
  tactics = list(
    tactic_id = "tactic_paid_search",
    strategy_id = "strategy_qualified_demand",
    title = "Expand Paid Search Acquisition"
  ),
  levers = list(
    lever_id = "lever_paid_search_budget",
    tactic_id = "tactic_paid_search",
    related_variables = list(c("paid_search_spend", "planned_paid_search_spend")),
    evidence_strength = 0.82,
    uncertainty = 0.25,
    economic_importance = 0.9,
    measurement_quality = 0.8,
    reversibility = TRUE,
    experiment_eligible = TRUE,
    optimization_eligible = TRUE,
    execution_eligible = FALSE,
    approval_required = TRUE,
    validated_range = "80000-120000"
  ),
  kpis = list(
    kpi_id = "kpi_revenue",
    objective_id = "objective_revenue_growth",
    source_variable = "revenue",
    role = "primary"
  ),
  guardrails = list(
    guardrail_id = "guardrail_capacity",
    tactic_id = "tactic_paid_search",
    source_variable = "inventory_cap",
    measured = TRUE
  ),
  constraints = list(
    constraint_id = "constraint_budget",
    lever_id = "lever_paid_search_budget",
    lower_bound = 50000,
    upper_bound = 150000
  ),
  assumptions = list(
    assumption_id = "assumption_search_capacity",
    strategy_id = "strategy_qualified_demand",
    tactic_id = "tactic_paid_search",
    lever_id = "lever_paid_search_budget",
    statement = "Search capacity can absorb incremental qualified demand.",
    evidence_status = "partial",
    experiment_eligible = TRUE,
    evidence_strength = 0.45,
    unresolved_uncertainty = 0.65,
    economic_importance = 0.85,
    measurement_quality = 0.75
  ),
  authority = list(
    authority_id = "authority_marketing_advisory",
    authority_level = "recommend",
    approval_required = TRUE,
    required_approver = "CMO"
  ),
  coverage = list(
    coverage_id = "coverage_marketing_only",
    represented_domains = list("marketing"),
    missing_domains = list(c("finance", "operations")),
    limitation = "Marketing scope only; not an enterprise-wide allocation recommendation."
  ),
  variable_semantics = semantics
)

validation <- aq_validate_business_intent(intent)
alignment <- aq_assess_business_alignment(intent)
explore_exploit <- aq_assess_explore_exploit(intent)
artifact <- aq_business_intent_artifact(intent)

Validated script: inst/examples/business_intent_framework.R

Focused QA:

qa_business_intent_framework()

Decision Management

Use aq_decision_context() when analysis needs to support a governed decision rather than merely describe model output. Decision contexts preserve the decision question, baseline/current-policy alternative, candidate alternatives, criteria, financial impact, uncertainty, optionality, recommendation, selected decision, and outcome follow-up.

Use aq_decision_alternative() and aq_decision_optionality() directly when building those record tables before assembling the full context.

The framework is deterministic and human-governed. It can assess alternatives, flag authority escalation, identify reducible uncertainty, and produce a canonical decision_context_artifact; it does not execute actions or optimize budgets automatically.

decision <- aq_decision_context(
  context = list(
    decision_context_id = "decision_paid_search_budget",
    decision_question = "Should paid-search budget remain flat, increase within the validated range, or be piloted above the validated range?",
    related_objectives = list("objective_revenue_growth"),
    related_tactics = list("tactic_paid_search"),
    related_levers = list("lever_paid_search_budget"),
    authority = "authority_marketing_advisory",
    coverage = "coverage_marketing_only"
  ),
  alternatives = list(
    list(alternative_id = "alt_baseline", name = "Keep current budget", alternative_type = "do_nothing", baseline = TRUE),
    list(alternative_id = "alt_validated_increase", name = "Increase within validated range", alternative_type = "partial_implementation", authority_compatible = TRUE),
    list(alternative_id = "alt_pilot", name = "Pilot above validated range", alternative_type = "pilot", authority_compatible = FALSE)
  ),
  financial_impacts = list(
    list(financial_id = "fin_baseline", alternative_id = "alt_baseline", recurring_cost = 100000, expected_benefit = 110000),
    list(financial_id = "fin_validated", alternative_id = "alt_validated_increase", recurring_cost = 115000, expected_benefit = 140000),
    list(financial_id = "fin_pilot", alternative_id = "alt_pilot", recurring_cost = 140000, expected_benefit = 160000, downside_estimate = -35000)
  ),
  uncertainties = list(
    list(uncertainty_id = "unc_pilot_transfer", alternative_id = "alt_pilot", reducibility = "reducible", decision_sensitivity = "high")
  ),
  optionality = list(
    list(optionality_id = "opt_pilot_learning", alternative_id = "alt_pilot", option_type = "learn", future_decisions_enabled = list(c("expand", "abandon")), reversibility = TRUE)
  ),
  business_intent = intent,
  variable_semantics = semantics
)

validation <- aq_validate_decision_context(decision)
alternative_assessment <- aq_assess_decision_alternatives(decision)
optionality_assessment <- aq_assess_decision_optionality(decision)
artifact <- aq_decision_context_artifact(decision)

Validated script: inst/examples/decision_management_framework.R

Focused QA:

qa_decision_management_framework()

Decision Valuation Intelligence

Use aq_decision_valuation_context() after alternatives have been authored and evidence exists that may change the economic interpretation of those alternatives. The valuation layer compares alternatives against a baseline, preserves whether inputs are observed, causal, predictive, forecast, assumed, imported, missing, or unsupported, and computes transparent economics such as net benefit, ROI, payback, NPV, incremental NPV, thresholds, and governed recommendation states.

Decision valuation does not approve, optimize, allocate budget, or execute. It creates auditable valuation evidence.

valuation <- aq_decision_valuation_context(
  decision_context_id = "decision_paid_search_budget",
  alternatives_included = c("alt_baseline", "alt_pilot"),
  baseline_alternative = "alt_baseline",
  objective_refs = "objective_revenue_growth",
  strategy_refs = "strategy_qualified_demand",
  tactic_refs = "tactic_paid_search",
  lever_refs = "lever_paid_search_budget",
  authority = "authority_marketing_advisory",
  coverage = "coverage_marketing_only",
  time_horizon_periods = 1,
  discount_rate = 0.01
)

impact <- aq_evidence_impact_mapping(list(
  list(
    mapping_id = "itt_to_margin",
    source_artifact_id = "randomized_itt_effect_artifact",
    alternative_id = "alt_pilot",
    evidence_type = "randomized_itt",
    estimand_or_prediction = "ATE",
    effect_value = 0.04,
    affected_population = 50000,
    duration_periods = 1,
    unit_value = 8,
    source_type = "causally_estimated"
  )
))

economics <- aq_assess_alternative_economics(valuation, impact_mappings = impact)
recommendation <- aq_governed_decision_valuation_recommendation(valuation, economics)
artifact <- aq_decision_valuation_artifact(valuation, economics, recommendation)

Validated script: inst/examples/decision_valuation_framework.R

Focused QA:

qa_decision_valuation_framework()

Decision Workflow Intelligence

Use aq_decision_workflow() after valuation has produced a governed recommendation and the organization needs durable human review, approval, implementation tracking, monitoring, and realized-value follow-through.

The workflow layer preserves that recommendation, decision, implementation, and outcome are separate facts. It can assess review readiness, freeze an evidence package, record reviews and approvals, attach conditions, reconcile realized implementation against an approved plan, compare expected and realized value, and generate follow-up decision candidates. It does not approve, execute, or replace organizational authority.

workflow <- aq_decision_workflow(
  workflow_id = "workflow_paid_search_pilot",
  decision_context_id = "decision_paid_search_budget",
  valuation_artifact_id = artifact$id,
  recommendation_id = "recommendation_paid_search_pilot",
  selected_alternative = "alt_pilot",
  workflow_type = "pilot_approval",
  authority_tier = "manager"
)

readiness <- aq_assess_decision_review_readiness(workflow)
evidence_package <- aq_decision_evidence_package(
  workflow,
  evidence_refs = c(artifact$id, "randomized_itt_effect_artifact")
)

review <- aq_decision_review(list(
  workflow_id = workflow$workflow_id,
  reviewer = "Finance Owner",
  role = "financial",
  status = "endorse_with_conditions",
  conditions = "Monitor weekly spend and guardrails."
))

approval <- aq_decision_approval(list(
  workflow_id = workflow$workflow_id,
  approver = "Budget Manager",
  authority_basis = "marketing_budget_authority",
  approved_alternative = "alt_pilot",
  approved_budget = 100,
  authority_magnitude = 200,
  status = "conditionally_approved"
))

plan <- aq_decision_implementation_plan(list(
  workflow_id = workflow$workflow_id,
  implementation_id = "impl_paid_search_pilot",
  selected_alternative = "alt_pilot",
  approved_target_values = "paid_search_budget=110",
  budget = 100
))

actual <- aq_record_decision_implementation(list(
  workflow_id = workflow$workflow_id,
  implementation_id = "impl_paid_search_pilot",
  actual_cost = 105,
  actual_lever_settings = "paid_search_budget=110"
))

reconciliation <- aq_reconcile_decision_implementation(plan, actual)
quality <- aq_assess_decision_quality(workflow, readiness, review, approval, reconciliation)

Validated script: inst/examples/decision_workflow_framework.R

Focused QA:

qa_decision_workflow_framework()

Decision Lifecycle and Organizational Memory

Use aq_review_decision() after a governed decision has an observable outcome. The review records expected outcome, realized outcome, actual value, variance, execution state, assumption status, lessons learned, strategy implications, lever implications, and future recommendations. This turns decisions into durable organizational memory instead of one-time recommendations.

Use aq_decision_timeline() to reconstruct the deterministic lifecycle: context, alternatives, recommendations, selected decision, outcome follow-up, and learning review.

Use aq_decision_learning_summary() to summarize whether outcome evidence validated the decision pattern, produced partial learning, or created negative evidence. Use aq_decision_memory_artifact() to package the timeline and learning summary as a canonical artifact for collectors, reports, GenAI bounded context, and future knowledge promotion.

review <- aq_review_decision(
  decision,
  decision_id = "decision_budget_pending",
  realized_outcome = "Validated response inside delegated range.",
  actual_value = 36000,
  lessons_learned = "Validated range remains reusable.",
  assumption_status = "held"
)

timeline <- aq_decision_timeline(decision, review)
learning <- aq_decision_learning_summary(decision, review)
memory_artifact <- aq_decision_memory_artifact(decision, review)

Validated script: inst/examples/decision_lifecycle_and_memory.R

Focused QA:

qa_decision_lifecycle_framework()

Time-Series Forecasting

Use aq_forecast_spec() and aq_fit_forecast() for deterministic single-series forecasting. Current engines include naive, seasonal_naive, ets, arima, and catboost.

forecast_spec <- aq_forecast_spec(
  target = "demand",
  date = "ds",
  frequency = "day",
  horizon = 14,
  engine = "seasonal_naive",
  season_length = 7,
  prediction_intervals = TRUE
)

forecast <- aq_fit_forecast(forecast_spec, demand_history)
assessment <- aq_assess_forecast(forecast)

Validated script: inst/examples/vnext_forecasting.R

CatBoost Forecasting

CatBoost forecasting uses the same aq_forecast_spec() contract with engine = "catboost". Rodeo owns temporal transformation replay; AutoQuant owns model fitting, forecast artifacts, and assessment evidence.

catboost_forecast_spec <- aq_forecast_spec(
  target = "sales",
  date = "ds",
  frequency = "day",
  horizon = 7,
  engine = "catboost",
  forecast_strategy = "direct",
  future_known_variables = c("promotion", "holiday"),
  engine_parameters = list(iterations = 25, depth = 3)
)

Panel Forecasting

Use aq_panel_forecast_spec() when the same forecast target appears across multiple entities. The first vNext panel operator is a global CatBoost forecast with entity-aware evidence.

panel_spec <- aq_panel_forecast_spec(
  entity = "store",
  target = "sales",
  date = "ds",
  frequency = "day",
  horizon = 7,
  engine_parameters = list(iterations = 25, depth = 3)
)

Validated contract script: inst/examples/vnext_panel_hierarchy_strategy.R

Hierarchical Forecasting

Use aq_hierarchy_spec() and aq_reconcile_hierarchical_forecast() when entity forecasts must reconcile to parent totals. Current vNext reconciliation is deterministic bottom-up.

hierarchy_spec <- aq_hierarchy_spec(hierarchy_table)
reconciled <- aq_reconcile_hierarchical_forecast(panel_forecast, hierarchy_spec)

Panel Strategy Comparison

Use aq_panel_strategy_spec() and aq_evaluate_panel_strategies() to compare independent, grouped, and global panel approaches. The recommendation is advisory evidence, not automatic production selection.

strategy_spec <- aq_panel_strategy_spec(
  entity = "store",
  group = "region",
  target = "sales",
  date = "ds",
  candidate_strategies = c("independent", "grouped", "global"),
  horizon = 7
)

Intermittent-Demand Forecasting

Intermittent demand is a problem family with classical and supervised operators:

Classical: aq_croston_forecast_spec(), aq_sba_forecast_spec(), aq_tsb_forecast_spec()
Supervised: aq_hurdle_forecast_spec()
Comparison: aq_compare_intermittent_demand_methods()

diagnostics <- aq_intermittent_demand_diagnostics(history, target = "demand", date = "ds")

croston <- aq_fit_croston_forecast(
  aq_croston_forecast_spec(target = "demand", date = "ds", horizon = 7),
  history
)

sba <- aq_fit_sba_forecast(
  aq_sba_forecast_spec(target = "demand", date = "ds", horizon = 7),
  history
)

tsb <- aq_fit_tsb_forecast(
  aq_tsb_forecast_spec(target = "demand", date = "ds", horizon = 7, beta = 0.2),
  history
)

comparison <- aq_compare_intermittent_demand_methods(
  data = history,
  target = "demand",
  date = "ds",
  horizon = 3,
  include_hurdle = FALSE
)

Validated script: inst/examples/vnext_intermittent_demand.R

Funnel Forecasting

Funnel forecasting treats a process as ordered stages and explicit adjacent transitions. It is different from forecasting unrelated columns.

funnel_spec <- aq_funnel_forecast_spec(
  stages = c("prospects", "applications", "qualified", "enrollments"),
  stage = "stage",
  value = "volume",
  date = "date",
  cohort = "cohort",
  stage_date = "stage_date",
  maturity = "maturity",
  frequency = "week",
  horizon = 2,
  forecast_origin = as.Date("2026-02-12"),
  strategy = "stage"
)

validation <- aq_validate_funnel_forecast_spec(funnel_spec, funnel_data)
stage_forecast <- aq_fit_funnel_forecast(funnel_spec, funnel_data)
stage_assessment <- aq_assess_funnel_forecast(stage_forecast)

transition_spec <- funnel_spec
transition_spec$strategy <- "transition"
transition_forecast <- aq_fit_funnel_forecast(transition_spec, funnel_data)

comparison <- aq_compare_funnel_strategies(funnel_spec, funnel_data)

Validated script: inst/examples/vnext_funnel_forecasting.R

Multi-Target Forecasting

Multi-target forecasting treats several related targets as a shared analytical problem while preserving target-level evidence. Phase 19 supports deterministic independent and shared-workflow strategies; it does not implement VAR, VARMAX, state-space multivariate models, deep learning, or target-causality modeling.

multitarget_spec <- aq_multitarget_forecast_spec(
  targets = c("leads", "applications", "enrollments"),
  date = "date",
  frequency = "week",
  horizon = 3,
  forecast_origin = max(multitarget_data$date) - 21,
  known_future_variables = "promo",
  shared_predictors = "spend",
  target_specific_predictors = list(enrollments = "promo"),
  strategy = "independent",
  engine = "seasonal_naive",
  season_length = 4
)

validation <- aq_validate_multitarget_forecast_spec(multitarget_spec, multitarget_data)
forecast <- aq_fit_multitarget_forecast(multitarget_spec, multitarget_data)
assessment <- aq_assess_multitarget_forecast(forecast)

shared_spec <- multitarget_spec
shared_spec$strategy <- "shared_workflow"
shared_forecast <- aq_fit_multitarget_forecast(shared_spec, multitarget_data)

comparison <- aq_compare_multitarget_strategies(multitarget_spec, multitarget_data)

Validated script: inst/examples/vnext_multitarget_forecasting.R

Cross-Target Feature Forecasting

Cross-target feature forecasting asks whether prior information from one target helps forecast another target. AutoQuant treats this as an empirical hypothesis, not a default assumption. Rodeo owns deterministic cross-target lag, rolling, calendar, and known-future feature preparation. AutoQuant fits CatBoost direct horizon models and compares independent, shared-workflow, and cross-target strategies using realized forecast evidence.

cross_target_spec <- aq_multitarget_forecast_spec(
  targets = c("leads", "applications", "enrollments"),
  date = "date",
  frequency = "week",
  horizon = 2,
  forecast_origin = max(multitarget_data$date) - 14,
  known_future_variables = "promo",
  shared_predictors = "spend",
  target_specific_predictors = list(enrollments = "promo"),
  strategy = "cross_target_features",
  engine = "catboost",
  cross_target_feature_policy = "lags_rolls",
  shared_target_lags = c(1, 2),
  shared_rolling_windows = 3,
  target_relationship_metadata = list(source = "analyst_hypothesis", causal_claim = FALSE),
  engine_parameters = list(iterations = 5, depth = 2, learning_rate = 0.1)
)

validation <- aq_validate_multitarget_forecast_spec(cross_target_spec, multitarget_data)
forecast <- aq_fit_multitarget_forecast(cross_target_spec, multitarget_data)
assessment <- aq_assess_multitarget_forecast(forecast)
comparison <- aq_compare_multitarget_strategies(cross_target_spec, multitarget_data)

Validated script: inst/examples/vnext_cross_target_forecasting.R

Forecasting Capability Planning

Use aq_discover_forecasting_capabilities() and aq_plan_forecasting_strategy() before running broad forecasting experiments. The planner inspects schema, targets, date/entity structure, hierarchy, funnel shape, zero inflation, known-future variables, supported operators, missing evidence, and historical CARMA feature-tuning mechanisms. It returns a canonical forecast_planning_artifact; it does not execute models.

discovery <- aq_discover_forecasting_capabilities(
  multitarget_data,
  target = c("leads", "applications", "enrollments"),
  date = "date",
  hierarchy = "region",
  known_future_variables = "promo",
  horizon = 3
)

plan <- aq_plan_forecasting_strategy(discovery)

plan$recommendations
plan$required_baselines
plan$experiment_set
plan$artifact$metadata$carma_mechanism_inventory

Validated script: inst/examples/vnext_forecasting_planning.R

Forecasting Experiment Campaigns

Use aq_forecast_experiment_spec() after forecasting planning to run one bounded, evidence-guided experiment. The experiment preserves a frozen baseline, executes one deterministic challenger, assesses out-of-sample evidence, and returns a canonical forecast_experiment_artifact. Failed or unsuccessful challengers are preserved as negative evidence. No challenger is adopted automatically.

experiment <- aq_forecast_experiment_spec(
  planning_result = plan,
  experiment_type = "model",
  baseline = "naive",
  challenger = "ets",
  target = "demand",
  date = "date",
  horizon = 3,
  hypothesis = "ETS may reduce forecast error relative to a frozen naive baseline."
)

result <- aq_run_forecast_experiment(experiment, history)

result$learning
result$artifact

campaign <- aq_run_forecast_experiment_campaign(list(experiment), history)
campaign$summary

Validated script: inst/examples/vnext_forecasting_experiment_campaigns.R

Package QA

Run the installed package QA entry point after installation or before integrating with AnalyticsShinyApp:

qa <- AutoQuant::qa_autoquant_package()
qa[, .N, by = status]
qa[status != "pass"]

cross_target_qa <- AutoQuant::qa_vnext_multitarget_supervised_forecasting()
planning_qa <- AutoQuant::qa_vnext_forecasting_planning()
experiment_qa <- AutoQuant::qa_vnext_forecasting_experiment_campaigns()
semantics_qa <- AutoQuant::qa_variable_semantics_framework()
business_intent_qa <- AutoQuant::qa_business_intent_framework()
decision_qa <- AutoQuant::qa_decision_management_framework()
decision_lifecycle_qa <- AutoQuant::qa_decision_lifecycle_framework()

The vNext QA includes supervised learning, scoring, model bundles, artifact contracts, canonical variable semantics, business intent, decision management, decision lifecycle memory, forecasting, panel forecasting, hierarchy reconciliation, panel strategy comparison, intermittent-demand operators, funnel forecasting, multi-target forecasting, cross-target feature forecasting, forecasting capability planning, governed forecasting experiment campaigns, and README/example coverage.

Legacy API Status

The legacy AutoQuant functions remain available for compatibility. vNext is the preferred path for new artifact-first work because it separates specifications, fitting, scoring, assessment, artifacts, and downstream evidence consumption. H2O remains historical/compatibility-oriented for vNext planning rather than the default redesign engine.

Detailed Architecture Documents

docs/autoquant_vnext_archaeology_and_design.md
docs/vnext_catboost_regression.md
docs/canonical_analytical_artifacts.md
docs/variable_semantics_framework.md
docs/business_intent_framework.md
docs/decision_management_philosophy.md
docs/decision_management_framework.md
docs/decision_lifecycle_and_memory.md
docs/vnext_forecasting_foundation.md
docs/vnext_forecasting_planning.md
docs/vnext_forecasting_experiment_campaigns.md
docs/vnext_panel_forecasting_foundation.md
docs/vnext_hierarchical_forecasting_foundation.md
docs/vnext_panel_strategy_selection.md
docs/vnext_intermittent_demand_forecasting.md
docs/vnext_funnel_forecasting_foundation.md
docs/vnext_multitarget_forecasting_foundation.md
docs/vnext_cross_target_forecasting.md

Background

Expand to view content

Automated Machine Learning - In my view, AutoML should consist of functions to help make professional model development and operationalization more efficient. The functions in this package are there to help no matter which part of the ML lifecycle you are working on. The functions in this package have been tested across a variety of industries and have consistently outperformed competing methods.

Package Details

Supervised Learning - Currently, I'm utilizing CatBoost, LightGBM, XGBoost, and H2O for all of the automated Machine Learning related functions. GPU's can be utilized with CatBoost, LightGBM, and XGBoost, while those and the H2O models can all utilize 100% of CPU. Multi-armed bandit grid tuning is available for CatBoost, LightGBM, and XGBoost models, which utilize the concept of randomized probability matching, which is detailed in the R pacakge "bandit". My choice of included ML algorithms in the package is based on previous success when compared against other algorithms on real world use cases, the additional utilities these packages offer aside from accurate predictions, their ability to work on big data, and the fact that they're available in both R and Python which makes managing multiple languages a little more seamless in a professional setting.

Documentation - Each exported function in the package has a help file and can be viewed in your RStudio session, e.g. ?Rodeo::ModelDataPrep. Many of them come with examples coded up in the help files (at the bottom) that you can run to get a feel for how to set the parameters. There's also a listing of exported functions by category with code examples at the bottom of this readme. You can also jump into the R folder here to dig into the source code.

Overall process: Typically, I go to the warehouse to get all of my base features and then I run through all the relevant feature engineering functions in this package. Personally, I set up templates for features engineering, model training optimization, and model scoring (including feature engineering for scoring). I collect all relevant metdata in a list that is shared across templates and as a result, I never have to touch the model scoring template, which makes operationalize and maintenace a breeze. I can simply list out the columns of interest, which feature engineering functions I want to utilize, and then I simply kick off some command line scripts and everything else is automatically managed.

Installation

The Description File is designed to require only the minimum number of packages to install AutoQuant. However, in order to utilize most of the functions in the package, you'll have to install additional libraries. I set it up this way on purpose. You don't need to install every single possible dependency if you are only interested in using a few of the functions. For example, if you only want to use CatBoost then install the catboost package and forget about the h2o, xgboost, and lightgbm packages. This is one of the primary benefits of not hosting an R package on cran, as they require dependencies to be part of the Imports section on the Description File, which subsequently requires users to have all dependencies installed in order to install the package.

The minimal set of packages that need to be installed are below. The full list can be found by expanding the section (Expand to view content).

bit64
data.table
doParallel
foreach
lubridate
timeDate

# Core pacakges
if(!("data.table" %in% rownames(installed.packages()))) install.packages("data.table"); print("data.table")
if(!("collapse" %in% rownames(installed.packages()))) install.packages("collapse"); print("collapse")
if(!("bit64" %in% rownames(installed.packages()))) install.packages("bit64"); print("bit64")
if(!("devtools" %in% rownames(installed.packages()))) install.packages("devtools"); print("devtools")
if(!("doParallel" %in% rownames(installed.packages()))) install.packages("doParallel"); print("doParallel")
if(!("foreach" %in% rownames(installed.packages()))) install.packages("foreach"); print("foreach")
if(!("lubridate" %in% rownames(installed.packages()))) install.packages("lubridate"); print("lubridate")
if(!("timeDate" %in% rownames(installed.packages()))) install.packages("timeDate"); print("timeDate")

# AutoQuant
devtools::install_github('AdrianAntico/AutoQuant', upgrade = FALSE, dependencies = FALSE, force = TRUE)

Additional Packages to Install

Install ALL R package dependencies for all functions:

XGBoost and LightGBM can be used with GPU. However, their installation is much more involved than CatBoost, which comes with GPU capabilities simply by installing their package. The installation instructions for them below is for the CPU version only. Refer to each's home page for instructions for installing for GPU.

# Install Dependencies----
if(!("devtools" %in% rownames(installed.packages()))) install.packages("devtools"); print("devtools")

# Core pacakges
if(!("data.table" %in% rownames(installed.packages()))) install.packages("data.table"); print("data.table")
if(!("collapse" %in% rownames(installed.packages()))) install.packages("collapse"); print("collapse")
if(!("bit64" %in% rownames(installed.packages()))) install.packages("bit64"); print("bit64")
if(!("devtools" %in% rownames(installed.packages()))) install.packages("devtools"); print("devtools")
if(!("doParallel" %in% rownames(installed.packages()))) install.packages("doParallel"); print("doParallel")
if(!("foreach" %in% rownames(installed.packages()))) install.packages("foreach"); print("foreach")
if(!("lubridate" %in% rownames(installed.packages()))) install.packages("lubridate"); print("lubridate")
if(!("timeDate" %in% rownames(installed.packages()))) install.packages("timeDate"); print("timeDate")

# Additional dependencies for specific use cases
if(!("combinat" %in% rownames(installed.packages()))) install.packages("combinat"); print("combinat")
if(!("DBI" %in% rownames(installed.packages()))) install.packages("DBI"); print("DBI")
if(!("e1071" %in% rownames(installed.packages()))) install.packages("e1071"); print("e1071")
if(!("fBasics" %in% rownames(installed.packages()))) install.packages("fBasics"); print("fBasics")
if(!("forecast" %in% rownames(installed.packages()))) install.packages("forecast"); print("forecast")
if(!("fpp" %in% rownames(installed.packages()))) install.packages("fpp"); print("fpp")
if(!("ggplot2" %in% rownames(installed.packages()))) install.packages("ggplot2"); print("ggplot2")
if(!("gridExtra" %in% rownames(installed.packages()))) install.packages("gridExtra"); print("gridExtra")
if(!("itertools" %in% rownames(installed.packages()))) install.packages("itertools"); print("itertools")
if(!("MLmetrics" %in% rownames(installed.packages()))) install.packages("MLmetrics"); print("MLmetrics")
if(!("nortest" %in% rownames(installed.packages()))) install.packages("nortest"); print("nortest")
if(!("pROC" %in% rownames(installed.packages()))) install.packages("pROC"); print("pROC")
if(!("RColorBrewer" %in% rownames(installed.packages()))) install.packages("RColorBrewer"); print("RColorBrewer")
if(!("recommenderlab" %in% rownames(installed.packages()))) install.packages("recommenderlab"); print("recommenderlab")
if(!("RPostgres" %in% rownames(installed.packages()))) install.packages("RPostgres"); print("RPostgres")
if(!("Rfast" %in% rownames(installed.packages()))) install.packages("Rfast"); print("Rfast")
if(!("scatterplot3d" %in% rownames(installed.packages()))) install.packages("scatterplot3d"); print("scatterplot3d")
if(!("stringr" %in% rownames(installed.packages()))) install.packages("stringr"); print("stringr")
if(!("tsoutliers" %in% rownames(installed.packages()))) install.packages("tsoutliers"); print("tsoutliers")
if(!("xgboost" %in% rownames(installed.packages()))) install.packages("xgboost"); print("xgboost")
if(!("lightgbm" %in% rownames(installed.packages()))) install.packages("lightgbm"); print("lightgbm")
if(!("regmedint" %in% rownames(installed.packages()))) install.packages("regmedint"); print("regmedint")
for(pkg in c("RCurl","jsonlite")) if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }
install.packages("h2o", type = "source", repos = (c("http://h2o-release.s3.amazonaws.com/h2o/latest_stable_R")))
devtools::install_github('catboost/catboost', subdir = 'catboost/R-package')

# Dependencies for ML Reports
if(!("reactable" %in% rownames(installed.packages()))) install.packages("reactable"); print("reactable")
devtools::install_github('AdrianAntico/prettydoc', upgrade = FALSE, dependencies = FALSE, force = TRUE)

# And lastly, AutoQuant
devtools::install_github('AdrianAntico/AutoQuant', upgrade = FALSE, dependencies = FALSE, force = TRUE)

Reactable-backed report tables support text exclusion filters. In a text column filter, use:

Impressions to keep rows containing Impressions
!Impressions to exclude rows containing Impressions
-Impressions to exclude rows containing Impressions

Text matching is case-insensitive. Numeric and date columns keep their standard reactable filtering behavior.

Installation Troubleshooting

The most common issue some users are having when trying to install AutoQuant is the installation of the catboost package dependency. Since catboost is not on CRAN it can only be installed through GitHub. To install catboost without error (and consequently install AutoQuant without error), try running this line of code first, then restart your R session, then re-run the 2-step installation process above. (Reference): If you're still having trouble submit an issue and I'll work with you to get it installed.

# Method for on premise servers
options(devtools.install.args = c("--no-multiarch", "--no-test-load"))
install.packages("https://github.com/catboost/catboost/releases/download/<version>/catboost-R-Windows-<version>.tgz", repos = NULL, type = "source", INSTALL_opts = c("--no-multiarch", "--no-test-load"))

# Method for azure machine learning Designer pipelines

## catboost
install.packages("https://github.com/catboost/catboost/releases/download/<version>/catboost-R-Windows-<version>.tgz", repos = NULL, type = "source", INSTALL_opts = c("--no-multiarch", "--no-test-load"))

## AutoQuant
install.packages("https://github.com/AdrianAntico/AutoQuant/archive/refs/tags/<version>.tar.gz", repos = NULL, type = "source", INSTALL_opts = c("--no-multiarch", "--no-test-load"))

Code Examples

EDA

Expand to view content

# ============================================================
# Fake EDA Report QA Data for AutoQuant::EDAReport()
# ============================================================

set.seed(8675309)

library(data.table)
library(AutoQuant)

# ----------------------------
# 1. Create fake data
# ----------------------------

n <- 5000

dates <- seq.Date(
  from = as.Date("2023-01-01"),
  to   = as.Date("2025-12-31"),
  by   = "day"
)

dt <- data.table(
  id = seq_len(n),

  # Date / trend fields
  event_date = sample(dates, n, replace = TRUE),

  # Grouping fields
  channel = sample(
    c("Search", "Social", "Email", "Direct", "Affiliate"),
    n,
    replace = TRUE,
    prob = c(0.34, 0.24, 0.16, 0.18, 0.08)
  ),

  region = sample(
    c("West", "South", "Midwest", "Northeast"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.28, 0.20, 0.17)
  ),

  customer_segment = sample(
    c("Budget", "Standard", "Premium", "Enterprise"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.40, 0.20, 0.05)
  )
)

# Date features for synthetic signal
dt[, month_num := as.integer(format(event_date, "%m"))]
dt[, year_num  := as.integer(format(event_date, "%Y"))]
dt[, day_index := as.integer(event_date - min(event_date))]

# Base effects
channel_effect <- c(
  Search    = 1.20,
  Social    = 0.85,
  Email     = 1.35,
  Direct    = 1.00,
  Affiliate = 0.65
)

segment_effect <- c(
  Budget     = 0.70,
  Standard   = 1.00,
  Premium    = 1.55,
  Enterprise = 2.40
)

region_effect <- c(
  West      = 1.10,
  South     = 0.95,
  Midwest   = 0.85,
  Northeast = 1.05
)

# Seasonality and trend
dt[, seasonality := 1 + 0.25 * sin(2 * pi * month_num / 12)]
dt[, trend       := 1 + day_index / max(day_index) * 0.35]

# Numeric variables with structure
dt[, impressions := round(
  rgamma(.N, shape = 8, scale = 1200) *
    channel_effect[channel] *
    seasonality *
    trend
)]

dt[, clicks := round(
  impressions * pmin(
    pmax(rnorm(.N, mean = 0.035, sd = 0.012), 0.002),
    0.12
  )
)]

dt[, spend := round(
  impressions * runif(.N, 0.008, 0.025) *
    channel_effect[channel] *
    region_effect[region],
  2
)]

dt[, conversions := rpois(
  .N,
  lambda = pmax(
    clicks *
      runif(.N, 0.025, 0.12) *
      segment_effect[customer_segment] *
      region_effect[region],
    0.01
  )
)]

dt[, revenue := round(
  conversions *
    rgamma(.N, shape = 4, scale = 90) *
    segment_effect[customer_segment] *
    runif(.N, 0.75, 1.35),
  2
)]

dt[, ctr := clicks / pmax(impressions, 1)]
dt[, cvr := conversions / pmax(clicks, 1)]
dt[, cpc := spend / pmax(clicks, 1)]
dt[, roas := revenue / pmax(spend, 1)]

# Some less-behaved variables for QA
dt[, noise_normal := rnorm(.N, mean = 100, sd = 15)]
dt[, skewed_score := rgamma(.N, shape = 2, scale = 20)]
dt[, binary_flag := rbinom(.N, size = 1, prob = 0.28)]

# Deliberate correlated variables
dt[, spend_lag_proxy := spend * runif(.N, 0.85, 1.15) + rnorm(.N, 0, 25)]
dt[, revenue_proxy   := revenue * runif(.N, 0.90, 1.10) + rnorm(.N, 0, 100)]

# Deliberate outliers
outlier_rows <- sample(seq_len(n), size = 35)
dt[outlier_rows, spend := spend * runif(.N, 4, 9)]
dt[outlier_rows, revenue := revenue * runif(.N, 3, 7)]
dt[outlier_rows, impressions := round(impressions * runif(.N, 2, 5))]

# Deliberate missingness
dt[sample(.N, 150), revenue := NA_real_]
dt[sample(.N, 120), cpc := NA_real_]
dt[sample(.N, 90), customer_segment := NA_character_]
dt[sample(.N, 80), ctr := NA_real_]

# Deliberate zero-heavy variable
dt[, zero_heavy_metric := fifelse(
  runif(.N) < 0.72,
  0,
  round(rgamma(.N, shape = 2, scale = 15), 2)
)]

# Clean helper columns if you do not want them in report
# Keeping them can also help QA univariate behavior.
# dt[, c("month_num", "year_num", "day_index", "seasonality", "trend") := NULL]


# ----------------------------
# 2. Define report inputs
# ----------------------------

UnivariateVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "binary_flag",
  "zero_heavy_metric",
  "channel",
  "region",
  "customer_segment"
)

CorrVars <- c(
  "impressions",
  "clicks",
  "spend",
  "spend_lag_proxy",
  "conversions",
  "revenue",
  "revenue_proxy",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "zero_heavy_metric"
)

TrendVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas"
)

OutputPath <- getwd()


# ----------------------------
# 3. Full report: all options
# ----------------------------

eda_artifacts <- generate_eda_artifacts(
  data = dt,
  DataName = "EDA Data",
  UnivariateVars = UnivariateVars,
  CorrVars = CorrVars,
  TrendVars = TrendVars,
  TrendDateVar = "event_date",
  TrendGroupVar = "channel",
  Theme = "dark",
  OutputPath = "eda_artifacts",
  ExportPNG = FALSE
)

# Generate Report
EDAReport(
  artifacts = eda_artifacts,
  DataName = "AutoQuant Fake Marketing QA Dataset",
  OutputPath = OutputPath,
  Theme = "dark"
)

Model Readiness

Expand to view content

# ============================================================
# Fake EDA Report QA Data for AutoQuant::EDAReport()
# ============================================================

set.seed(8675309)

library(data.table)
library(AutoQuant)

# ----------------------------
# 1. Create fake data
# ----------------------------

n <- 5000

dates <- seq.Date(
  from = as.Date("2023-01-01"),
  to   = as.Date("2025-12-31"),
  by   = "day"
)

dt <- data.table(
  id = seq_len(n),

  # Date / trend fields
  event_date = sample(dates, n, replace = TRUE),

  # Grouping fields
  channel = sample(
    c("Search", "Social", "Email", "Direct", "Affiliate"),
    n,
    replace = TRUE,
    prob = c(0.34, 0.24, 0.16, 0.18, 0.08)
  ),

  region = sample(
    c("West", "South", "Midwest", "Northeast"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.28, 0.20, 0.17)
  ),

  customer_segment = sample(
    c("Budget", "Standard", "Premium", "Enterprise"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.40, 0.20, 0.05)
  )
)

# Date features for synthetic signal
dt[, month_num := as.integer(format(event_date, "%m"))]
dt[, year_num  := as.integer(format(event_date, "%Y"))]
dt[, day_index := as.integer(event_date - min(event_date))]

# Base effects
channel_effect <- c(
  Search    = 1.20,
  Social    = 0.85,
  Email     = 1.35,
  Direct    = 1.00,
  Affiliate = 0.65
)

segment_effect <- c(
  Budget     = 0.70,
  Standard   = 1.00,
  Premium    = 1.55,
  Enterprise = 2.40
)

region_effect <- c(
  West      = 1.10,
  South     = 0.95,
  Midwest   = 0.85,
  Northeast = 1.05
)

# Seasonality and trend
dt[, seasonality := 1 + 0.25 * sin(2 * pi * month_num / 12)]
dt[, trend       := 1 + day_index / max(day_index) * 0.35]

# Numeric variables with structure
dt[, impressions := round(
  rgamma(.N, shape = 8, scale = 1200) *
    channel_effect[channel] *
    seasonality *
    trend
)]

dt[, clicks := round(
  impressions * pmin(
    pmax(rnorm(.N, mean = 0.035, sd = 0.012), 0.002),
    0.12
  )
)]

dt[, spend := round(
  impressions * runif(.N, 0.008, 0.025) *
    channel_effect[channel] *
    region_effect[region],
  2
)]

dt[, conversions := rpois(
  .N,
  lambda = pmax(
    clicks *
      runif(.N, 0.025, 0.12) *
      segment_effect[customer_segment] *
      region_effect[region],
    0.01
  )
)]

dt[, revenue := round(
  conversions *
    rgamma(.N, shape = 4, scale = 90) *
    segment_effect[customer_segment] *
    runif(.N, 0.75, 1.35),
  2
)]

dt[, ctr := clicks / pmax(impressions, 1)]
dt[, cvr := conversions / pmax(clicks, 1)]
dt[, cpc := spend / pmax(clicks, 1)]
dt[, roas := revenue / pmax(spend, 1)]

# Some less-behaved variables for QA
dt[, noise_normal := rnorm(.N, mean = 100, sd = 15)]
dt[, skewed_score := rgamma(.N, shape = 2, scale = 20)]
dt[, binary_flag := rbinom(.N, size = 1, prob = 0.28)]

# Deliberate correlated variables
dt[, spend_lag_proxy := spend * runif(.N, 0.85, 1.15) + rnorm(.N, 0, 25)]
dt[, revenue_proxy   := revenue * runif(.N, 0.90, 1.10) + rnorm(.N, 0, 100)]

# Deliberate outliers
outlier_rows <- sample(seq_len(n), size = 35)
dt[outlier_rows, spend := spend * runif(.N, 4, 9)]
dt[outlier_rows, revenue := revenue * runif(.N, 3, 7)]
dt[outlier_rows, impressions := round(impressions * runif(.N, 2, 5))]

# Deliberate missingness
dt[sample(.N, 150), revenue := NA_real_]
dt[sample(.N, 120), cpc := NA_real_]
dt[sample(.N, 90), customer_segment := NA_character_]
dt[sample(.N, 80), ctr := NA_real_]

# Deliberate zero-heavy variable
dt[, zero_heavy_metric := fifelse(
  runif(.N) < 0.72,
  0,
  round(rgamma(.N, shape = 2, scale = 15), 2)
)]

# Clean helper columns if you do not want them in report
# Keeping them can also help QA univariate behavior.
# dt[, c("month_num", "year_num", "day_index", "seasonality", "trend") := NULL]


# ----------------------------
# 2. Define report inputs
# ----------------------------

UnivariateVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "binary_flag",
  "zero_heavy_metric",
  "channel",
  "region",
  "customer_segment"
)

CorrVars <- c(
  "impressions",
  "clicks",
  "spend",
  "spend_lag_proxy",
  "conversions",
  "revenue",
  "revenue_proxy",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "zero_heavy_metric"
)

TrendVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas"
)

OutputPath <- getwd()


# ----------------------------
# 3. Full report: all options
# ----------------------------

model_readiness_artifacts <- generate_model_assessment_artifacts(
  data = dt,
  DataName = "Modeling Data",
  TargetVar = "conversions",
  TrendDateVar = "event_date",
  TrendGroupVar = c("channel", "market"),
  Theme = "dark",
  OutputPath = "model_readiness_artifacts",
  ExportPNG = FALSE
)

TargetAnalysisReport(
  artifacts = model_readiness_artifacts,
  OutputPath = OutputPath
)

Supervised Learning

Expand to view content

CatBoost Builder Artifact Generator

AutoCatBoostRegression() and AutoCatBoostClassifier() remain the core AutoQuant CatBoost training functions. generate_catboost_builder_artifacts() is the artifact-first wrapper for app/report workflows: it routes training through those existing functions, standardizes scored output with Predict and .split, and preserves Shap_ contribution columns when the wrapped AutoCatBoost output includes them. The scored output is designed to feed Model Assessment, Model Insights, and SHAP Analysis; it does not run those downstream modules automatically. AnalyticsShinyApp should call generate_catboost_builder_artifacts(), not the lower-level training functions directly.

library(data.table)

set.seed(42)
n <- 300L
dt <- data.table(
  id = seq_len(n),
  event_date = as.Date("2025-01-01") + seq_len(n) - 1L,
  channel = sample(c("Search", "Email", "Social", "Direct"), n, replace = TRUE),
  region = sample(c("West", "Midwest", "South"), n, replace = TRUE),
  spend = runif(n, 50, 500),
  clicks = rpois(n, 40),
  discount = runif(n, 0, 0.3)
)

dt[, revenue := 10 + 2.5 * spend + 1.8 * clicks - 120 * discount +
  fifelse(channel == "Search", 80, 0) +
  fifelse(region == "West", 40, 0) +
  rnorm(.N, 0, 35)]

builder <- AutoQuant::generate_catboost_builder_artifacts(
  data = dt,
  target_col = "revenue",
  feature_cols = c("channel", "region", "spend", "clicks", "discount"),
  problem_type = "regression",
  id_cols = "id",
  DateVar = "event_date",
  ByVars = c("channel", "region"),
  split_method = "time",
  iterations = 30L,
  depth = 4L,
  learning_rate = 0.08,
  compute_shap = TRUE
)

builder$metadata$artifact_index
builder$value$scored_data
builder$value$downstream_handoff

Modern Report API Contract

Modern AutoQuant reports are artifact renderers. Artifact generators own analytical parameters; report functions own rendering parameters.

Preferred generator-first workflow:

reg_artifacts <- AutoQuant::generate_regression_shap_analysis_artifacts(
  data = scored_data,
  target_col = "Target",
  prediction_col = "Predict",
  DateVar = "Date",
  ByVars = "Segment"
)

AutoQuant::RegressionShapAnalysisReport(
  artifact_result = reg_artifacts,
  OutputPath = "reports",
  OutputFile = "regression_shap.html"
)

Convenience wrapper workflow:

AutoQuant::RegressionShapAnalysisReport(
  data = scored_data,
  OutputPath = "reports",
  OutputFile = "regression_shap.html",
  target_col = "Target",
  prediction_col = "Predict",
  DateVar = "Date",
  ByVars = "Segment"
)

The same pattern applies to EDAReport(), TargetAnalysisReport(), RegressionModelInsightsReport(), BinaryClassificationModelInsightsReport(), RegressionShapAnalysisReport(), and BinaryClassificationShapAnalysisReport(). ModelInsightsReport() is legacy compatibility only.

See docs/report_api_contract.md.

Optional AutoNLS SHAP effect curves

SHAP generators can optionally use AutoNLS vNext as an internal curve-fitting backend for numeric dependence data:

reg_artifacts <- AutoQuant::generate_regression_shap_analysis_artifacts(
  data = scored_data,
  target_col = "Target",
  prediction_col = "Predict",
  effect_curve_backend = "autonls",
  effect_curve_models = "stable",
  effect_curve_sample_size = 50000,
  effect_curve_max_features = 20,
  effect_curve_validation_fraction = 0.20
)

AutoNLS is optional. The default effect_curve_backend = "none" keeps existing SHAP workflows unchanged and does not require AutoNLS to be installed.

AutoQuant accepts original-scale feature and SHAP contribution columns. AutoNLS may use internal scaling, log/log1p starts, or family-specific transformed initialization for optimizer stability, but returned curve values, predictions, derivatives, and elasticities remain on the original scale. If fitting is unavailable or unsuitable, AutoQuant returns effect-curve diagnostics and continues generating SHAP artifacts.

Typed Artifact Schema Framework

Future AutoQuant generators should return reusable typed artifacts that can power reports, dashboards, Shiny apps, APIs, and LLM agents without recomputing analytical work.

The first typed artifact layer supports:

table
plot
diagnostic
finding
warning
metadata
computation_graph
display_plan
quality_gate

Small example:

importance <- data.table::data.table(
  feature = c("Spend", "Clicks"),
  mean_abs_contribution = c(0.42, 0.18)
)

table_artifact <- AutoQuant::new_table_artifact(
  id = "tbl_global_importance",
  title = "Global Importance",
  data = importance,
  source_generator = "example_generator"
)

finding_artifact <- AutoQuant::new_finding_artifact(
  id = "finding_top_driver",
  title = "Top Driver",
  claim = "Spend is the strongest driver in this example result.",
  evidence_ids = "tbl_global_importance",
  confidence = 0.8,
  source_generator = "example_generator"
)

AutoQuant::validate_artifact_collection(list(table_artifact, finding_artifact))

See docs/contracts/artifact_schema_contract.md and inst/examples/artifact_schema_example.R.

Regression

click to expand

Regression Description

The Auto_Regression() models handle a multitude of tasks. In order:

Convert your data to data.table format for faster processing
Transform your target variable using the best normalization method based on the AutoTransformationCreate() function
Create train, validation, and test data, utilizing the AutoDataPartition() function, if you didn't supply those directly to the function
Consoldate columns that are used for modeling and what metadata you want returned in your test data with predictions
Dichotomize categorical variables (for AutoXGBoostRegression()) and save the factor levels for scoring in a way that guarentees consistency across training, validation, and test data sets, utilizing the DummifyDT() function
Save the final modeling column names for reference
Handles the data conversion to the appropriate modeling type, such as CatBoost, H2O, and XGBoost
Multi-armed bandit hyperparameter tuning using randomized probability matching, if you choose to grid tune
Loop through the grid-tuning process, building N models
Collect the evaluation metrics for each grid tune run
Identify the best model of the set of models built in the grid tuning search
Save the hyperparameters from the winning grid tuned model
Build the final model based on the best model from the grid tuning model search (I remove each model after evaluation metrics are generated in the grid tune to avoid memory overflow)
Back-transform your predictions based on the best transformation used earlier in the process
Collect evaluation metrics based on performance on test data (based on back-transformed data)
Store the final predictions with the associated test data and other columns you want included in that set
Save your transformation metadata for recreating them in a scoring process
Build out and save an Evaluation Calibration Line Plot and Evaluation Calibration Box-Plot, using the EvalPlot() function
Generate and save Variable Importance
Generate and save Partital Dependence Calibration Line Plots and Partital Dependence Calibration Box-Plots, using the ParDepPlots() function
Return all the objects generated in a named list for immediate use and evaluation

CatBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 10000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoCatBoostRegression(

  # GPU or CPU and the number of available GPUs
  TrainOnFull = FALSE,
  task_type = 'GPU',
  NumGPUs = 1,
  DebugMode = FALSE,

  # Metadata args
  OutputSelection = c('Importances', 'EvalPlots', 'EvalMetrics', 'Score_TrainData'),
  ModelID = 'Test_Model_1',
  model_path = normalizePath('./'),
  metadata_path = normalizePath('./'),
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  ReturnModelObjects = TRUE,

  # Data args
  data = data,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = names(data)[!names(data) %in% c('IDcol_1', 'IDcol_2','Adrian')],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c('IDcol_1','IDcol_2'),
  TransformNumericColumns = 'Adrian',
  Methods = c('BoxCox', 'Asinh', 'Asin', 'Log',
    'LogPlus1', 'Sqrt', 'Logit'),

  # Model evaluation
  eval_metric = 'RMSE',
  eval_metric_value = 1.5,
  loss_function = 'RMSE',
  loss_function_value = 1.5,
  MetricPeriods = 10L,
  NumOfParDepPlots = ncol(data)-1L-2L,

  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 60*60,
  BaselineComparison = 'default',

  # ML args
  langevin = FALSE,
  diffusion_temperature = 10000,
  Trees = 1000,
  Depth = 9,
  L2_Leaf_Reg = NULL,
  RandomStrength = 1,
  BorderCount = 128,
  LearningRate = NULL,
  RSM = 1,
  BootStrapType = NULL,
  GrowPolicy = 'SymmetricTree',
  model_size_reg = 0.5,
  feature_border_type = 'GreedyLogSum',
  sampling_unit = 'Object',
  subsample = NULL,
  score_function = 'Cosine',
  min_data_in_leaf = 1)
  
# Legacy compatibility training report.
# New regression/binary workflows should use the artifact-first report functions.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = names(data)[!names(data) %in% c('IDcol_1', 'IDcol_2','Adrian')],
  SampleSize = 1e+05,
  ModelObject = TestModel,
  ModelID = "Test_Model_1",
  OutputPath = getwd()
)

XGBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoXGBoostRegression(
  
  # GPU or CPU
  TreeMethod = 'hist',
  NThreads = parallel::detectCores(),
  LossFunction = 'reg:squarederror',
  
  # Metadata args
  OutputSelection = c('Importances', 'EvalPlots', 'EvalMetrics', 'Score_TrainData'),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  EncodingMethod = "binary",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  DebugMode = FALSE,
  
  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c('IDcol_1','IDcol_2','Adrian')],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c('IDcol_1','IDcol_2'),
  TransformNumericColumns = 'Adrian',
  Methods = c('Asinh','Asin','Log','LogPlus1','Sqrt','Logit'),
  
  # Model evaluation args
  eval_metric = 'rmse',
  NumOfParDepPlots = 3L,
  
  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  grid_eval_metric = 'r2',
  BaselineComparison = 'default',
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,
  
  # ML args
  Trees = 50L,
  eta = 0.05,
  max_depth = 4L,
  min_child_weight = 1.0,
  subsample = 0.55,
  colsample_bytree = 0.55)

LightGBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoLightGBMRegression(

  # Metadata args
  OutputSelection = c('Importances','EvalPlots','EvalMetrics','Score_TrainData'),
  model_path = normalizePath('./'),
  metadata_path = NULL,
  ModelID = 'Test_Model_1',
  NumOfParDepPlots = 3L,
  EncodingMethod = 'credibility',
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  DebugMode = FALSE,

  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = names(data)[!names(data) %in% c('IDcol_1', 'IDcol_2','Adrian')],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c('IDcol_1','IDcol_2'),
  TransformNumericColumns = NULL,
  Methods = c('Asinh','Asin','Log','LogPlus1','Sqrt','Logit'),

  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = 'r2',
  BaselineComparison = 'default',
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,

  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = 'train',
  device_type = 'CPU',
  NThreads = parallel::detectCores() / 2,
  objective = 'regression',
  metric = 'rmse',
  boosting = 'gbdt',
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,

  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = NULL,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = 'advanced',
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,

  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,

  # Convert Parameters
  convert_model = NULL,
  convert_model_language = 'cpp',

  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  alpha = 0.90,
  fair_c = 1.0,
  poisson_max_delta_step = 0.70,
  tweedie_variance_power = 1.5,
  lambdarank_truncation_level = 30,

  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),

  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,

  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

H2O-GBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoH2oGBMRegression(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = normalizePath("./"),
  metadata_path = file.path(normalizePath("./")),
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = names(data)[!names(data) %in% c('IDcol_1','IDcol_2','Adrian')],
  WeightsColumn = NULL,
  TransformNumericColumns = NULL,
  Methods = c('Asinh','Asin','Log','LogPlus1','Sqrt','Logit'),
  
  # ML grid tuning args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,
  MaxModelsInGrid = 2,
  
  # Model args
  Trees = 50,
  LearnRate = 0.10,
  LearnRateAnnealing = 1,
  eval_metric = "RMSE",
  Alpha = NULL,
  Distribution = "poisson",
  MaxDepth = 20,
  SampleRate = 0.632,
  ColSampleRate = 1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel  = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO")

H2O-DRF Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoH2oDRFRegression(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1L, parallel::detectCores() - 2L),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation:
  eval_metric = "RMSE",
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data Args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumn = NULL,
  TransformNumericColumns = NULL,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  
  # Grid Tuning Args
  GridStrategy = "Cartesian",
  GridTune = FALSE,
  MaxModelsInGrid = 10,
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,
  
  # ML Args
  Trees = 50,
  MaxDepth = 20,
  SampleRate = 0.632,
  MTries = -1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO")

H2O-GLM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoH2oGLMRegression(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation:
  eval_metric = "RMSE",
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments:
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  RandomColNumbers = NULL,
  InteractionColNumbers = NULL,
  WeightsColumn = NULL,
  TransformNumericColumns = NULL,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  
  # Model args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "gaussian",
  Link = "identity",
  TweedieLinkPower = NULL,
  TweedieVariancePower = NULL,
  RandomDistribution = NULL,
  RandomLink = NULL,
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

H2O-AutoML Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoH2oMLRegression(

  # Compute management
  MaxMem = "32G",
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  IfSaveModel = "mojo",

  # Model evaluation
  eval_metric = "RMSE",
  NumOfParDepPlots = 3,

  # Metadata arguments
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,

  # Data arguments
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  TransformNumericColumns = NULL,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Logit"),

  # Model args
  GridTune = FALSE,
  ExcludeAlgos = NULL,
  Trees = 50,
  MaxModelsInGrid = 10)

H2O-GAM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Define GAM Columns to use - up to 9 are allowed
GamCols <- names(which(unlist(lapply(data, is.numeric))))
GamCols <- GamCols[!GamCols %in% c("Adrian","IDcol_1","IDcol_2")]
GamCols <- GamCols[1L:(min(9L,length(GamCols)))]

# Run function
TestModel <- AutoQuant::AutoH2oGAMRegression(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation:
  eval_metric = "RMSE",
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments:
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  InteractionColNumbers = NULL,
  WeightsColumn = NULL,
  GamColNames = GamCols,
  TransformNumericColumns = NULL,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  
  # Model args
  num_knots = NULL,
  keep_gam_cols = TRUE,
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "gaussian",
  Link = "Family_Default",
  TweedieLinkPower = NULL,
  TweedieVariancePower = NULL,
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

Binary Classification

click to expand

Classification Description

The Auto_Classifier() models handle a multitude of tasks. In order:

Convert your data to data.table format for faster processing
Create train, validation, and test data if you didn't supply those directly to the function
Consoldate columns that are used for modeling and what is to be kept for data returned
Dichotomize categorical variables (for AutoXGBoostRegression) and save the factor levels for scoring in a way that guarentees consistency across training, validation, and test data sets
Saves the final column names for modeling to a csv for later reference
Handles the data conversion to the appropriate type, based on model type (CatBoost, H2O, and XGBoost)
Multi-armed bandit hyperparameter tuning using randomized probability matching, if you choose to grid tune
Build the grid tuned models
Collect the evaluation metrics for each grid tune run
Identify the best model of the set of models built in the grid tuning setup
Save the hyperparameters from the winning grid tuned model
Build the final model based on the best model from the grid tuning model search
Collect evaluation metrics based on performance on test data
Store the final predictions with the associated test data and other columns you want included in that set
Build out and save an Evaluation Calibration Line Plot
Build out and save an ROC plot with the top 5 models used in grid-tuning (includes the winning model)
Generate and save Variable Importance data
Generate and save Partital Dependence Calibration Line Plots
Return all the objects generated in a named list for immediate use

CatBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 10000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoCatBoostClassifier(

  # GPU or CPU and the number of available GPUs
  task_type = 'GPU',
  NumGPUs = 1,
  TrainOnFull = FALSE,
  DebugMode = FALSE,

  # Metadata args
  OutputSelection = c('Score_TrainData', 'Importance', 'EvalPlots', 'Metrics', 'PDF'),
  ModelID = 'Test_Model_1',
  model_path = normalizePath('./'),
  metadata_path = normalizePath('./'),
  SaveModelObjects = FALSE,
  ReturnModelObjects = TRUE,
  SaveInfoToPDF = FALSE,

  # Data args
  data = data,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = names(data)[!names(data) %in% c('IDcol_1','IDcol_2','Adrian')],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c('IDcol_1','IDcol_2'),

  # Evaluation args
  ClassWeights = c(1L,1L),
  CostMatrixWeights = c(1,0,0,1),
  EvalMetric = 'AUC',
  grid_eval_metric = 'MCC',
  LossFunction = 'Logloss',
  MetricPeriods = 10L,
  NumOfParDepPlots = ncol(data)-1L-2L,

  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  BaselineComparison = 'default',

  # ML args
  Trees = 1000,
  Depth = 9,
  LearningRate = NULL,
  L2_Leaf_Reg = NULL,
  model_size_reg = 0.5,
  langevin = FALSE,
  diffusion_temperature = 10000,
  RandomStrength = 1,
  BorderCount = 128,
  RSM = 1,
  BootStrapType = 'Bayesian',
  GrowPolicy = 'SymmetricTree',
  feature_border_type = 'GreedyLogSum',
  sampling_unit = 'Object',
  subsample = NULL,
  score_function = 'Cosine',
  min_data_in_leaf = 1)

# Legacy compatibility training report.
# New regression/binary workflows should use the artifact-first report functions.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = names(data)[!names(data) %in% c('IDcol_1', 'IDcol_2','Adrian')],
  SampleSize = 1e+05,
  ModelObject = TestModel,
  ModelID = "Test_Model_1",
  OutputPath = getwd()
)

XGBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoXGBoostClassifier(

  # GPU or CPU
  TreeMethod = "hist",
  NThreads = parallel::detectCores(),

  # Metadata args
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "PDFs", "Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  EncodingMethod = "binary",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,

  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in%
    c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumnName = NULL,
  IDcols = c("IDcol_1","IDcol_2"),

  # Model evaluation
  LossFunction = 'reg:logistic',
  CostMatrixWeights = c(1,0,0,1),
  eval_metric = "auc",
  grid_eval_metric = "MCC",
  NumOfParDepPlots = 3L,

  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,

  # ML args
  Trees = 500L,
  eta = 0.30,
  max_depth = 9L,
  min_child_weight = 1.0,
  subsample = 1,
  colsample_bytree = 1,
  DebugMode = FALSE)

LightGBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoLightGBMClassifier(

  # Metadata args
  OutputSelection = c("Importances","EvalPlots","EvalMetrics","Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  NumOfParDepPlots = 3L,
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  DebugMode = FALSE,

  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c("IDcol_1","IDcol_2"),

  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = 'Utility',
  BaselineComparison = 'default',
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,

  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = "train",
  device_type = 'CPU',
  NThreads = parallel::detectCores() / 2,
  objective = 'binary',
  metric = 'binary_logloss',
  boosting = 'gbdt',
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,

  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = NULL,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = "advanced",
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,

  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,

  # Convert Parameters
  convert_model = NULL,
  convert_model_language = "cpp",

  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  is_unbalance = FALSE,
  scale_pos_weight = 1.0,

  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),

  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,

  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

H2O-GBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

TestModel <- AutoQuant::AutoH2oGBMClassifier(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = normalizePath("./"),
  metadata_path = file.path(normalizePath("./")),
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumn = NULL,
  
  # ML grid tuning args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,
  MaxModelsInGrid = 2,
  
  # Model args
  Trees = 50,
  LearnRate = 0.10,
  LearnRateAnnealing = 1,
  eval_metric = "auc",
  Distribution = "bernoulli",
  MaxDepth = 20,
  SampleRate = 0.632,
  ColSampleRate = 1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel  = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO")

H2O-DRF Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

TestModel <- AutoQuant::AutoH2oDRFClassifier(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1L, parallel::detectCores() - 2L),
  IfSaveModel = "mojo",
  H2OShutdown = FALSE,
  H2OStartUp = TRUE,
  
  # Metadata arguments:
  eval_metric = "auc",
  NumOfParDepPlots = 3L,
  
  # Data arguments:
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Model evaluation:
  data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2", "Adrian")],
  WeightsColumn = NULL,
  
  # Grid Tuning Args
  GridStrategy = "Cartesian",
  GridTune = FALSE,
  MaxModelsInGrid = 10,
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,
  
  # Model args
  Trees = 50L,
  MaxDepth = 20,
  SampleRate = 0.632,
  MTries = -1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO")

H2O-GLM Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoH2oGLMClassifier(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation args
  eval_metric = "auc",
  NumOfParDepPlots = 3,
  
  # Metadata args
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in%
                                  c("IDcol_1", "IDcol_2","Adrian")],
  RandomColNumbers = NULL,
  InteractionColNumbers = NULL,
  WeightsColumn = NULL,
  
  # ML args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "binomial",
  Link = "logit",
  RandomDistribution = NULL,
  RandomLink = NULL,
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

H2O-AutoML Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85, 
  N = 1000L, 
  ID = 2L, 
  ZIP = 0L, 
  AddDate = FALSE, 
  Classification = TRUE, 
  MultiClass = FALSE)

TestModel <- AutoQuant::AutoH2oMLClassifier(
   data,
   TrainOnFull = FALSE,
   ValidationData = NULL,
   TestData = NULL,
   TargetColumnName = "Adrian",
   FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
   ExcludeAlgos = NULL,
   eval_metric = "auc",
   Trees = 50,
   MaxMem = "32G",
   NThreads = max(1, parallel::detectCores()-2),
   MaxModelsInGrid = 10,
   model_path = normalizePath("./"),
   metadata_path = file.path(normalizePath("./"), "MetaData"),
   ModelID = "FirstModel",
   NumOfParDepPlots = 3,
   ReturnModelObjects = TRUE,
   SaveModelObjects = FALSE,
   IfSaveModel = "mojo",
   H2OShutdown = FALSE,
   HurdleModel = FALSE)

H2O-GAM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = TRUE,
  MultiClass = FALSE)

# Define GAM Columns to use - up to 9 are allowed
GamCols <- names(which(unlist(lapply(data, is.numeric))))
GamCols <- GamCols[!GamCols %in% c("Adrian","IDcol_1","IDcol_2")]
GamCols <- GamCols[1L:(min(9L,length(GamCols)))]

# Run function
TestModel <- AutoQuant::AutoH2oGAMClassifier(

  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",

  # Model evaluation:
  eval_metric = "auc",
  NumOfParDepPlots = 3,

  # Metadata arguments:
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,

  # Data arguments:
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumn = NULL,
  GamColNames = GamCols,

  # ML args
  num_knots = NULL,
  keep_gam_cols = TRUE,
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "binomial",
  Link = "logit",
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

MultiClass Classification

click to expand

MultiClass Description

The Auto_MultiClass() models handle a multitude of tasks. In order:

Convert your data to data.table format for faster processing
Create train, validation, and test data if you didn't supply those directly to the function
Consoldate columns that are used for modeling and what is to be kept for data returned
Dichotomize categorical variables (for AutoXGBoostRegression) and save the factor levels for scoring in a way that guarentees consistency across training, validation, and test data sets
Saves the final column names for modeling to a csv for later reference
Ensures the target levels are consistent across train, validate, and test sets and save the levels to file
Handles the data conversion to the appropriate type, based on model type (CatBoost, H2O, and XGBoost)
Multi-armed bandit hyperparameter tuning using randomized probability matching, if you choose to grid tune
Build the grid tuned models
Collect the evaluation metrics for each grid tune run
Identify the best model of the set of models built in the grid tuning setup
Save the hyperparameters from the winning grid tuned model
Build the final model based on the best model from the grid tuning model search
Collect evaluation metrics based on performance on test data
Store the final predictions with the associated test data and other columns you want included in that set
Generate and save Variable Importance data
Return all the objects generated in a named list for immediate use

CatBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 10000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoCatBoostMultiClass(
  
  # GPU or CPU and the number of available GPUs
  task_type = 'GPU',
  NumGPUs = 1,
  TrainOnFull = FALSE,
  DebugMode = FALSE,
  
  # Metadata args
  OutputSelection = c('Importances', 'EvalPlots', 'EvalMetrics', 'Score_TrainData'),
  ModelID = 'Test_Model_1',
  model_path = normalizePath('./'),
  metadata_path = normalizePath('./'),
  SaveModelObjects = FALSE,
  ReturnModelObjects = TRUE,
  
  # Data args
  data = data,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = names(data)[!names(data) %in%
                                  c('IDcol_1', 'IDcol_2','Adrian')],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  ClassWeights = c(1L,1L,1L,1L,1L),
  IDcols = c('IDcol_1','IDcol_2'),
  
  # Model evaluation
  eval_metric = 'MCC',
  loss_function = 'MultiClassOneVsAll',
  grid_eval_metric = 'Accuracy',
  MetricPeriods = 10L,
  NumOfParDepPlots = 3,
  
  # Grid tuning args
  PassInGrid = NULL,
  GridTune = TRUE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  BaselineComparison = 'default',
  
  # ML args
  langevin = FALSE,
  diffusion_temperature = 10000,
  Trees = seq(100L, 500L, 50L),
  Depth = seq(4L, 8L, 1L),
  LearningRate = seq(0.01,0.10,0.01),
  L2_Leaf_Reg = seq(1.0, 10.0, 1.0),
  RandomStrength = 1,
  BorderCount = 254,
  RSM = c(0.80, 0.85, 0.90, 0.95, 1.0),
  BootStrapType = c('Bayesian', 'Bernoulli', 'Poisson', 'MVS', 'No'),
  GrowPolicy = c('SymmetricTree', 'Depthwise', 'Lossguide'),
  model_size_reg = 0.5,
  feature_border_type = 'GreedyLogSum',
  sampling_unit = 'Object',
  subsample = NULL,
  score_function = 'Cosine',
  min_data_in_leaf = 1)

# Legacy compatibility training report.
# New regression/binary workflows should use the artifact-first report functions.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = names(data)[!names(data) %in% c('IDcol_1', 'IDcol_2','Adrian')],
  SampleSize = 1e+05,
  ModelObject = TestModel,
  ModelID = "Test_Model_1",
  OutputPath = getwd()
)

XGBoost Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoXGBoostMultiClass(
  
  # GPU or CPU
  TreeMethod = "hist",
  NThreads = parallel::detectCores(),
  
  # Metadata args
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "PDFs", "Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = normalizePath("./"),
  ModelID = "Test_Model_1",
  EncodingMethod = "binary",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  
  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in%
                                  c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumnName = NULL,
  IDcols = c("IDcol_1","IDcol_2"),
  
  # Model evaluation args
  eval_metric = "merror",
  LossFunction = 'multi:softprob',
  grid_eval_metric = "accuracy",
  NumOfParDepPlots = 3L,
  
  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,
  DebugMode = FALSE,
  
  # ML args
  Trees = 50L,
  eta = 0.05,
  max_depth = 4L,
  min_child_weight = 1.0,
  subsample = 0.55,
  colsample_bytree = 0.55)

LightGBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Run function
TestModel <- AutoQuant::AutoLightGBMMultiClass(

  # Metadata args
  OutputSelection = c("Importances","EvalPlots","EvalMetrics","Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  NumOfParDepPlots = 3L,
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  DebugMode = FALSE,

  # Data args
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c("IDcol_1","IDcol_2"),

  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = 'microauc',
  BaselineComparison = 'default',
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,

  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = "train",
  device_type = 'CPU',
  NThreads = parallel::detectCores() / 2,
  objective = 'multiclass',
  multi_error_top_k = 1,
  metric = 'multi_logloss',
  boosting = 'gbdt',
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,

  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = NULL,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = "advanced",
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,

  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,

  # Convert Parameters
  convert_model = NULL,
  convert_model_language = "cpp",

  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  is_unbalance = FALSE,
  scale_pos_weight = 1.0,

  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),

  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,

  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

H2O-GBM Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoH2oGBMMultiClass(
   data,
   TrainOnFull = FALSE,
   ValidationData = NULL,
   TestData = NULL,
   TargetColumnName = "Adrian",
   FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
   WeightsColumn = NULL,
   eval_metric = "logloss",
   MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
   NThreads = max(1, parallel::detectCores()-2),
   model_path = normalizePath("./"),
   metadata_path = file.path(normalizePath("./")),
   ModelID = "FirstModel",
   ReturnModelObjects = TRUE,
   SaveModelObjects = FALSE,
   IfSaveModel = "mojo",
   H2OShutdown = TRUE,
   H2OStartUp = TRUE,

   # Model args
   GridTune = FALSE,
   GridStrategy = "Cartesian",
   MaxRuntimeSecs = 60*60*24,
   StoppingRounds = 10,
   MaxModelsInGrid = 2,
   Trees = 50,
   LearnRate = 0.10,
   LearnRateAnnealing = 1,
   eval_metric = "RMSE",
   Distribution = "multinomial",
   MaxDepth = 20,
   SampleRate = 0.632,
   ColSampleRate = 1,
   ColSampleRatePerTree = 1,
   ColSampleRatePerTreeLevel  = 1,
   MinRows = 1,
   NBins = 20,
   NBinsCats = 1024,
   NBinsTopLevel = 1024,
   HistogramType = "AUTO",
   CategoricalEncoding = "AUTO")

H2O-DRF Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoH2oDRFMultiClass(
  data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumn = NULL,
  eval_metric = "logloss",
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  model_path = normalizePath("./"),
  metadata_path = file.path(normalizePath("./")),
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  IfSaveModel = "mojo",
  H2OShutdown = FALSE,
  H2OStartUp = TRUE,

  # Grid Tuning Args
  GridStrategy = "Cartesian",
  GridTune = FALSE,
  MaxModelsInGrid = 10,
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,

  # ML args
  Trees = 50,
  MaxDepth = 20,
  SampleRate = 0.632,
  MTries = -1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO")

H2O-GLM Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoH2oGLMMultiClass(
  
  # Compute management
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  H2OShutdown = TRUE,
  H2OStartUp = TRUE,
  IfSaveModel = "mojo",
  
  # Model evaluation:
  eval_metric = "logloss",
  NumOfParDepPlots = 3,
  
  # Metadata arguments:
  model_path = NULL,
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments:
  data = data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  RandomColNumbers = NULL,
  InteractionColNumbers = NULL,
  WeightsColumn = NULL,
  
  # Model args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "multinomial",
  Link = "family_default",
  RandomDistribution = NULL,
  RandomLink = NULL,
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

H2O-AutoML Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(Correlation = 0.85, N = 1000, ID = 2, ZIP = 0, AddDate = FALSE, Classification = FALSE, MultiClass = TRUE)

# Run function
TestModel <- AutoQuant::AutoH2oMLMultiClass(
   data,
   TrainOnFull = FALSE,
   ValidationData = NULL,
   TestData = NULL,
   TargetColumnName = "Adrian",
   FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
   ExcludeAlgos = NULL,
   eval_metric = "logloss",
   Trees = 50,
   MaxMem = "32G",
   NThreads = max(1, parallel::detectCores()-2),
   MaxModelsInGrid = 10,
   model_path = normalizePath("./"),
   metadata_path = file.path(normalizePath("./"), "MetaData"),
   ModelID = "FirstModel",
   ReturnModelObjects = TRUE,
   SaveModelObjects = FALSE,
   IfSaveModel = "mojo",
   H2OShutdown = FALSE,
   HurdleModel = FALSE)

H2O-GAM Example

# Create some dummy correlated data with numeric and categorical features
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 1000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Define GAM Columns to use - up to 9 are allowed
GamCols <- names(which(unlist(lapply(data, is.numeric))))
GamCols <- GamCols[!GamCols %in% c("Adrian","IDcol_1","IDcol_2")]
GamCols <- GamCols[1L:(min(9L,length(GamCols)))]

# Run function
TestModel <- AutoQuant::AutoH2oGAMMultiClass(
  data,
  TrainOnFull = FALSE,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = names(data)[!names(data) %in% c("IDcol_1", "IDcol_2","Adrian")],
  WeightsColumn = NULL,
  GamColNames = GamCols,
  eval_metric = "logloss",
  MaxMem = {gc();paste0(as.character(floor(as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) / 1000000)),"G")},
  NThreads = max(1, parallel::detectCores()-2),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "FirstModel",
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  IfSaveModel = "mojo",
  H2OShutdown = FALSE,
  H2OStartUp = TRUE,
  
  # ML args
  num_knots = NULL,
  keep_gam_cols = TRUE,
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  StoppingRounds = 10,
  MaxRunTimeSecs = 3600 * 24 * 7,
  MaxModelsInGrid = 10,
  Distribution = "multinomial",
  Link = "Family_Default",
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

Model Scoring

Expand to view content

Scoring Description

AutoCatBoostScoring() is an automated scoring function that compliments the AutoCatBoost__() model training functions. This function requires you to supply features for scoring. It will run ModelDataPrep() to prepare your features for catboost data conversion and scoring. It will also handle and transformations and back-transformations if you utilized that feature in the regression training case.

AutoXGBoostScoring() is an automated scoring function that compliments the AutoXGBoost__() model training functions. This function requires you to supply features for scoring. It will run ModelDataPrep() and the CategoricalEncoding() functions to prepare your features for xgboost data conversion and scoring. It will also handle and transformations and back-transformations if you utilized that feature in the regression training case.

AutoLightGBMScoring() is an automated scoring function that compliments the AutoLightGBM__() model training functions. This function requires you to supply features for scoring. It will run ModelDataPrep() and the CategoricalEncoding() functions to prepare your features for lightgbm data conversion and scoring. It will also handle and transformations and back-transformations if you utilized that feature in the regression training case.

AutoH2OMLScoring() is an automated scoring function that compliments the AutoH2oGBM__() and AutoH2oDRF__() model training functions. This function requires you to supply features for scoring. It will run ModelDataPrep()to prepare your features for H2O data conversion and scoring. It will also handle transformations and back-transformations if you utilized that feature in the regression training case and didn't do it yourself before hand.

AutoCatBoost__() Examples

AutoCatBoostRegression() Scoring Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 10000,
  ID = 2,
  ZIP = 0,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = FALSE)

# Copy data
data1 <- data.table::copy(data)

# Feature Colnames
Features <- names(data1)[!names(data1) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")]

# Run function
TestModel <- AutoQuant::AutoCatBoostRegression(
  
  # GPU or CPU and the number of available GPUs
  TrainOnFull = FALSE,
  task_type = 'CPU',
  NumGPUs = 1,
  DebugMode = FALSE,
  
  # Metadata args
  OutputSelection = c('Importances','EvalPlots','EvalMetrics','Score_TrainData'),
  ModelID = 'Test_Model_1',
  model_path = getwd(),
  metadata_path = getwd(),
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  ReturnModelObjects = TRUE,
  
  # Data args
  data = data1,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = 'Adrian',
  FeatureColNames = Features,
  PrimaryDateColumn = NULL,
  WeightsColumnName = NULL,
  IDcols = c('IDcol_1','IDcol_2'),
  TransformNumericColumns = 'Adrian',
  Methods = c('Asinh','Asin','Log','LogPlus1','Sqrt','Logit'),
  
  # Model evaluation
  eval_metric = 'RMSE',
  eval_metric_value = 1.5,
  loss_function = 'RMSE',
  loss_function_value = 1.5,
  MetricPeriods = 10L,
  NumOfParDepPlots = ncol(data1)-1L-2L,
  
  # Grid tuning args
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 60*60,
  BaselineComparison = 'default',
  
  # ML args
  langevin = FALSE,
  diffusion_temperature = 10000,
  Trees = 1000,
  Depth = 9,
  L2_Leaf_Reg = NULL,
  RandomStrength = 1,
  BorderCount = 128,
  LearningRate = NULL,
  RSM = 1,
  BootStrapType = NULL,
  GrowPolicy = 'SymmetricTree',
  model_size_reg = 0.5,
  feature_border_type = 'GreedyLogSum',
  sampling_unit = 'Object',
  subsample = NULL,
  score_function = 'Cosine',
  min_data_in_leaf = 1)


# Legacy compatibility insights report.
# New regression/binary workflows should use the artifact-first report functions.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = Features,
  SampleSize = 100000,
  ModelObject = ModelObject,
  ModelID = 'Test_Model_1',
  OutputPath = getwd())


# Score data
Preds <- AutoQuant::AutoCatBoostScoring(
  TargetType = 'regression',
  ScoringData = data,
  FeatureColumnNames = Features,
  FactorLevelsList = TestModel$FactorLevelsList,
  IDcols = c('IDcol_1','IDcol_2'),
  OneHot = FALSE,
  ReturnShapValues = TRUE,
  ModelObject = TestModel$Model,
  ModelPath = NULL,
  ModelID = 'Test_Model_1',
  ReturnFeatures = TRUE,
  MultiClassTargetLevels = NULL,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = FALSE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = '0',
  MDP_MissNum = -1,
  RemoveModel = FALSE)

AutoCatBoostClassifier() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = TRUE,
  MultiClass = FALSE)

# Copy data (used for scoring below``)
data1 <- data.table::copy(data)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Feature Colnames
Features <- names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")]

# AutoCatBoostClassifier
TestModel <- AutoQuant::AutoCatBoostClassifier(
  
  # GPU or CPU and the number of available GPUs
  task_type = "CPU",
  NumGPUs = 1,
  
  # Metadata arguments
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "Score_TrainData"),
  ModelID = "Test_Model_1",
  model_path = normalizePath("./"),
  metadata_path = normalizePath("./"),
  SaveModelObjects = FALSE,
  ReturnModelObjects = TRUE,
  SaveInfoToPDF = FALSE,
  
  # Data arguments
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = Features,
  PrimaryDateColumn = "DateTime",
  WeightsColumnName = "Weights",
  ClassWeights = c(1L,1L),
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  
  # Model evaluation
  CostMatrixWeights = c(2,0,0,1),
  EvalMetric = "MCC",
  LossFunction = "Logloss",
  grid_eval_metric = "Utility",
  MetricPeriods = 10L,
  NumOfParDepPlots = 3,
  
  # Grid tuning arguments
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  BaselineComparison = "default",
  
  # ML args
  Trees = 100L,
  Depth = 4L,
  LearningRate = NULL,
  L2_Leaf_Reg = NULL,
  RandomStrength = 1,
  BorderCount = 128,
  RSM = 0.80,
  BootStrapType = "Bayesian",
  GrowPolicy = "SymmetricTree",
  langevin = FALSE,
  diffusion_temperature = 10000,
  model_size_reg = 0.5,
  feature_border_type = "GreedyLogSum",
  sampling_unit = "Object",
  subsample = NULL,
  score_function = "Cosine",
  min_data_in_leaf = 1,
  DebugMode = TRUE)


# Legacy compatibility insights report.
# New binary workflows should use BinaryClassificationModelInsightsReport().
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = Features,
  SampleSize = 100000,
  ModelObject = ModelObject,
  ModelID = 'Test_Model_1',
  OutputPath = getwd())


# Score data
Preds <- AutoQuant::AutoCatBoostScoring(
  TargetType = 'classifier',
  ScoringData = data,
  FeatureColumnNames = Features,
  FactorLevelsList = TestModel$FactorLevelsList,
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  OneHot = FALSE,
  ReturnShapValues = TRUE,
  ModelObject = TestModel$Model,
  ModelPath = NULL,
  ModelID = 'Test_Model_1',
  ReturnFeatures = TRUE,
  MultiClassTargetLevels = NULL,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = '0',
  MDP_MissNum = -1,
  RemoveModel = FALSE)

AutoCatBoostMultiClasss() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = FALSE,
  MultiClass = TRUE)

# Copy data (used for scoring below``)
data1 <- data.table::copy(data)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Feature Colnames
Features <- names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","Adrian","DateTime")]

# Run function
TestModel <- AutoQuant::AutoCatBoostMultiClass(
  
  # GPU or CPU and the number of available GPUs
  task_type = "GPU",
  NumGPUs = 1,
  
  # Metadata arguments
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "Score_TrainData"),
  ModelID = "Test_Model_1",
  model_path = normalizePath("./"),
  metadata_path = normalizePath("./"),
  SaveModelObjects = FALSE,
  ReturnModelObjects = TRUE,
  
  # Data arguments
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = Features,
  PrimaryDateColumn = "DateTime",
  WeightsColumnName = "Weights",
  ClassWeights = c(1L,1L,1L,1L,1L),
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  
  # Model evaluation
  eval_metric = "MCC",
  loss_function = "MultiClassOneVsAll",
  grid_eval_metric = "Accuracy",
  MetricPeriods = 10L,
  
  # Grid tuning arguments
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  BaselineComparison = "default",
  
  # ML args
  Trees = 100L,
  Depth = 4L,
  LearningRate = 0.01,
  L2_Leaf_Reg = 1.0,
  RandomStrength = 1,
  BorderCount = 128,
  langevin = FALSE,
  diffusion_temperature = 10000,
  RSM = 0.80,
  BootStrapType = "Bayesian",
  GrowPolicy = "SymmetricTree",
  model_size_reg = 0.5,
  feature_border_type = "GreedyLogSum",
  sampling_unit = "Group",
  subsample = NULL,
  score_function = "Cosine",
  min_data_in_leaf = 1,
  DebugMode = TRUE)


# Legacy compatibility insights report.
# Multiclass remains here until a multiclass artifact-first replacement exists.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = Features,
  SampleSize = 100000,
  ModelObject = ModelObject,
  ModelID = 'Test_Model_1',
  OutputPath = getwd())


# Score data
Preds <- AutoQuant::AutoCatBoostScoring(
  TargetType = 'multiclass',
  ScoringData = data,
  FeatureColumnNames = Features,
  FactorLevelsList = TestModel$FactorLevelsList,
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  OneHot = FALSE,
  ReturnShapValues = FALSE,
  ModelObject = TestModel$Model,
  ModelPath = NULL,
  ModelID = 'Test_Model_1',
  ReturnFeatures = TRUE,
  MultiClassTargetLevels = TestModel$TargetLevels,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = '0',
  MDP_MissNum = -1,
  RemoveModel = FALSE)

AutoLightGBM__() Examples

AutoLightGBMRegression() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = FALSE,
  MultiClass = FALSE)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoLightGBMRegression(
  
  # GPU or CPU
  NThreads = parallel::detectCores(),
  
  # Metadata args
  OutputSelection = c("Importances","EvalPlots","EvalMetrics","Score_TrainData"),
  model_path = getwd(),
  metadata_path = getwd(),
  ModelID = "Test_Model_1",
  NumOfParDepPlots = 3L,
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = TRUE,
  SaveInfoToPDF = FALSE,
  DebugMode = TRUE,
  
  # Data args
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  PrimaryDateColumn = "DateTime",
  WeightsColumnName = "Weights",
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  TransformNumericColumns = NULL,
  Methods = c("Asinh","Asin","Log","LogPlus1","Sqrt","Logit"),
  
  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = "r2",
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,
  
  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = "train",
  device_type = "CPU",
  objective = 'regression',
  metric = "rmse",
  boosting = "gbdt",
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,
  
  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = 6,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 1.0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = "advanced",
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,
  
  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,
  
  # Convert Parameters
  convert_model = NULL,
  convert_model_language = "cpp",
  
  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  alpha = 0.90,
  fair_c = 1.0,
  poisson_max_delta_step = 0.70,
  tweedie_variance_power = 1.5,
  lambdarank_truncation_level = 30,
  
  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),
  
  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,
  
  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

# Outcome
ModelID = "Test_Model_1"
colnames <- data.table::fread(file = file.path(getwd(), paste0(ModelID, "_ColNames.csv")))
Preds <- AutoQuant::AutoLightGBMScoring(
  TargetType = "regression",
  ScoringData = TTestData,
  ReturnShapValues = FALSE,
  FeatureColumnNames = colnames[[1L]],
  IDcols = c("IDcol_1","IDcol_2"),
  EncodingMethod = "credibility",
  FactorLevelsList = NULL,
  TargetLevels = NULL,
  ModelObject = NULL,
  ModelPath = getwd(),
  ModelID = "Test_Model_1",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

AutoLightGBMClassifier() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = TRUE,
  MultiClass = FALSE)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoLightGBMClassifier(
  
  # Multithreading
  NThreads = parallel::detectCores(),
  
  # Metadata args
  OutputSelection = c("Importances","EvalPlots","EvalMetrics","Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  NumOfParDepPlots = 3L,
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = TRUE,
  SaveInfoToPDF = FALSE,
  DebugMode = TRUE,
  
  # Data args
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  PrimaryDateColumn = NULL,
  WeightsColumnName = "Weights",
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  CostMatrixWeights = c(1,0,0,1),
  
  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = "Utility",
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,
  
  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = "train",
  device_type = "CPU",
  objective = 'binary',
  metric = 'binary_logloss',
  boosting = "gbdt",
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,
  
  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = 6,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 1.0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = 'advanced',
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,
  
  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,
  
  # Convert Parameters
  convert_model = NULL,
  convert_model_language = "cpp",
  
  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  is_unbalance = FALSE,
  scale_pos_weight = 1.0,
  
  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),
  
  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,
  
  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

# Outcome
ModelID = "Test_Model_1"
colnames <- data.table::fread(file = file.path(getwd(), paste0(ModelID, "_ColNames.csv")))
Preds <- AutoQuant::AutoLightGBMScoring(
  TargetType = "classification",
  ScoringData = TTestData,
  ReturnShapValues = FALSE,
  FeatureColumnNames = colnames[[1L]],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  EncodingMethod = "credibility",
  FactorLevelsList = NULL,
  TargetLevels = NULL,
  ModelObject = NULL,
  ModelPath = getwd(),
  ModelID = "Test_Model_1",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

AutoLightGBMMultiClasss() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = FALSE,
  MultiClass = TRUE)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoLightGBMMultiClass(
  
  # GPU or CPU
  NThreads = parallel::detectCores(),
  
  # Metadata args
  OutputSelection = c("Importances","EvalPlots","EvalMetrics","Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  NumOfParDepPlots = 3L,
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = TRUE,
  SaveInfoToPDF = FALSE,
  DebugMode = TRUE,
  
  # Data args
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1","IDcol_2","DateTime","Adrian")],
  PrimaryDateColumn = NULL,
  WeightsColumnName = "Weights",
  IDcols = c("IDcol_1","IDcol_2",'DateTime'),
  CostMatrixWeights = c(1,0,0,1),
  
  # Grid parameters
  GridTune = FALSE,
  grid_eval_metric = "microauc",
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  PassInGrid = NULL,
  
  # Core parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#core-parameters
  input_model = NULL, # continue training a model that is stored to file
  task = "train",
  device_type = "CPU",
  objective = 'multiclass',
  multi_error_top_k = 1,
  metric = 'multiclass_logloss',
  boosting = "gbdt",
  LinearTree = FALSE,
  Trees = 50L,
  eta = NULL,
  num_leaves = 31,
  deterministic = TRUE,
  
  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  force_col_wise = FALSE,
  force_row_wise = FALSE,
  max_depth = 6,
  min_data_in_leaf = 20,
  min_sum_hessian_in_leaf = 0.001,
  bagging_freq = 1.0,
  bagging_fraction = 1.0,
  feature_fraction = 1.0,
  feature_fraction_bynode = 1.0,
  lambda_l1 = 0.0,
  lambda_l2 = 0.0,
  extra_trees = FALSE,
  early_stopping_round = 10,
  first_metric_only = TRUE,
  max_delta_step = 0.0,
  linear_lambda = 0.0,
  min_gain_to_split = 0,
  drop_rate_dart = 0.10,
  max_drop_dart = 50,
  skip_drop_dart = 0.50,
  uniform_drop_dart = FALSE,
  top_rate_goss = FALSE,
  other_rate_goss = FALSE,
  monotone_constraints = NULL,
  monotone_constraints_method = 'advanced',
  monotone_penalty = 0.0,
  forcedsplits_filename = NULL, # use for AutoStack option; .json file
  refit_decay_rate = 0.90,
  path_smooth = 0.0,
  
  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  max_bin = 255,
  min_data_in_bin = 3,
  data_random_seed = 1,
  is_enable_sparse = TRUE,
  enable_bundle = TRUE,
  use_missing = TRUE,
  zero_as_missing = FALSE,
  two_round = FALSE,
  
  # Convert Parameters
  convert_model = NULL,
  convert_model_language = "cpp",
  
  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  boost_from_average = TRUE,
  is_unbalance = FALSE,
  scale_pos_weight = 1.0,
  
  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  is_provide_training_metric = TRUE,
  eval_at = c(1,2,3,4,5),
  
  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  num_machines = 1,
  
  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  gpu_platform_id = -1,
  gpu_device_id = -1,
  gpu_use_dp = TRUE,
  num_gpu = 1)

# Outcome
ModelID = "Test_Model_1"
colnames <- data.table::fread(file = file.path(getwd(), paste0(ModelID, "_ColNames.csv")))
Preds <- AutoQuant::AutoLightGBMScoring(
  TargetType = "multiclass",
  ScoringData = TTestData,
  ReturnShapValues = FALSE,
  FeatureColumnNames = colnames[[1L]],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  EncodingMethod = "credibility",
  FactorLevelsList = NULL,
  TargetLevels = NULL,
  ModelObject = NULL,
  ModelPath = getwd(),
  ModelID = "Test_Model_1",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

AutoXGBoost__() Examples

AutoXGBoostRegression() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  FactorCount = 3,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = FALSE,
  MultiClass = FALSE)

# Copy data
data1 <- data.table::copy(data)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data1,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoXGBoostRegression(
  
  # GPU or CPU
  TreeMethod = "hist",
  NThreads = parallel::detectCores(),
  LossFunction = 'reg:squarederror',
  
  # Metadata arguments
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = TRUE,
  DebugMode = TRUE,
  
  # Data arguments
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  WeightsColumnName = "Weights",
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  TransformNumericColumns = NULL,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  
  # Model evaluation
  eval_metric = "rmse",
  NumOfParDepPlots = 3L,
  
  # Grid tuning arguments
  PassInGrid = NULL,
  GridTune = FALSE,
  grid_eval_metric = "r2",
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,
  SaveInfoToPDF = TRUE,
  
  # ML args
  Trees = 50L,
  eta = 0.05,
  max_depth = 4L,
  min_child_weight = 1.0,
  subsample = 0.55,
  colsample_bytree = 0.55)

# Score model
Preds <- AutoQuant::AutoXGBoostScoring(
  TargetType = "regression",
  ScoringData = data,
  ReturnShapValues = FALSE,
  FeatureColumnNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  EncodingMethod = "credibility",
  FactorLevelsList = TestModel$FactorLevelsList,
  TargetLevels = NULL,
  ModelObject = TestModel$Model,
  ModelPath = "home",
  ModelID = "ModelTest",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

AutoXGBoostClassifier() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = TRUE,
  MultiClass = FALSE)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoXGBoostClassifier(
  
  # GPU or CPU
  TreeMethod = "hist",
  NThreads = parallel::detectCores(),
  
  # Metadata arguments
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = NULL,
  ModelID = "Test_Model_1",
  EncodingMethod = "credibility",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = TRUE,
  SaveInfoToPDF = TRUE,
  DebugMode = TRUE,
  
  # Data arguments
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  WeightsColumnName = "Weights",
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  
  # Model evaluation
  LossFunction = 'reg:logistic',
  eval_metric = "auc",
  grid_eval_metric = "MCC",
  CostMatrixWeights = c(1,0,0,1),
  NumOfParDepPlots = 3L,
  
  # Grid tuning arguments
  PassInGrid = NULL,
  GridTune = FALSE,
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,
  
  # ML Args
  Trees = 50L,
  eta = 0.05,
  max_depth = 4L,
  min_child_weight = 1.0,
  subsample = 0.55,
  colsample_bytree = 0.55)

# Score model
Preds <- AutoQuant::AutoXGBoostScoring(
  TargetType = "classifier",
  ScoringData = data,
  ReturnShapValues = FALSE,
  FeatureColumnNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  EncodingMethod = "credibility",
  FactorLevelsList = TestModel$FactorLevelsList,
  TargetLevels = NULL,
  ModelObject = TestModel$Model,
  ModelPath = "home",
  ModelID = "ModelTest",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

AutoXGBoostMultiClasss() Scoring Example

# Refresh data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 25000L,
  ID = 2L,
  AddWeightsColumn = TRUE,
  ZIP = 0L,
  AddDate = TRUE,
  Classification = FALSE,
  MultiClass = TRUE)

# Partition Data
Sets <- Rodeo::AutoDataPartition(
  data = data,
  NumDataSets = 3,
  Ratios = c(0.7,0.2,0.1),
  PartitionType = "random",
  StratifyColumnNames = "Adrian",
  TimeColumnName = NULL)
TTrainData <- Sets$TrainData
VValidationData <- Sets$ValidationData
TTestData <- Sets$TestData
rm(Sets)

# Run function
TestModel <- AutoQuant::AutoXGBoostMultiClass(
  
  # GPU or CPU
  TreeMethod = "hist",
  NThreads = parallel::detectCores(),
  
  # Metadata arguments
  OutputSelection = c("Importances", "EvalPlots", "EvalMetrics", "Score_TrainData"),
  model_path = normalizePath("./"),
  metadata_path = normalizePath("./"),
  ModelID = "Test_Model_1",
  ReturnFactorLevels = TRUE,
  ReturnModelObjects = TRUE,
  SaveModelObjects = FALSE,
  EncodingMethod = "credibility",
  DebugMode = TRUE,
  
  # Data arguments
  data = TTrainData,
  TrainOnFull = FALSE,
  ValidationData = VValidationData,
  TestData = TTestData,
  TargetColumnName = "Adrian",
  FeatureColNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  
  # Model evaluation
  eval_metric = "merror",
  LossFunction = 'multi:softprob',
  grid_eval_metric = "accuracy",
  NumOfParDepPlots = 3L,
  
  # Grid tuning arguments
  PassInGrid = NULL,
  GridTune = FALSE,
  BaselineComparison = "default",
  MaxModelsInGrid = 10L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,
  Verbose = 1L,
  
  # ML Args
  Trees = 50L,
  eta = 0.05,
  max_depth = 4L,
  min_child_weight = 1.0,
  subsample = 0.55,
  colsample_bytree = 0.55)

# Score model
Preds <- AutoQuant::AutoXGBoostScoring(
  TargetType = "multiclass",
  ScoringData = data,
  ReturnShapValues = FALSE,
  FeatureColumnNames = names(TTrainData)[!names(TTrainData) %in% c("IDcol_1", "IDcol_2","DateTime","Adrian")],
  IDcols = c("IDcol_1","IDcol_2","DateTime"),
  EncodingMethod = "credibility",
  FactorLevelsList = TestModel$FactorLevelsList,
  TargetLevels = TestModel$TargetLevels,
  ModelObject = TestModel$Model,
  ModelPath = NULL,
  ModelID = "ModelTest",
  ReturnFeatures = TRUE,
  TransformNumeric = FALSE,
  BackTransNumeric = FALSE,
  TargetColumnName = NULL,
  TransformationObject = NULL,
  TransID = NULL,
  TransPath = NULL,
  MDP_Impute = TRUE,
  MDP_CharToFactor = TRUE,
  MDP_RemoveDates = TRUE,
  MDP_MissFactor = "0",
  MDP_MissNum = -1)

Model Evaluation

Expand to view content

ModelInsightsReport() is a legacy compatibility wrapper for older supervised-learning examples. New regression workflows should use generate_regression_model_insights_artifacts() plus RegressionModelInsightsReport(). New binary classification workflows should use generate_binary_classification_model_insights_artifacts() plus BinaryClassificationModelInsightsReport(). Multiclass remains on the legacy wrapper until a multiclass artifact generator/report pair exists.

RegressionModelInsightsReport() Example

set.seed(8675309)

library(data.table)
library(AutoQuant)

# ----------------------------
# 1. Create fake data
# ----------------------------

n <- 5000

dates <- seq.Date(
  from = as.Date("2023-01-01"),
  to   = as.Date("2025-12-31"),
  by   = "day"
)

dt <- data.table(
  id = seq_len(n),

  # Date / trend fields
  event_date = sample(dates, n, replace = TRUE),

  # Grouping fields
  channel = sample(
    c("Search", "Social", "Email", "Direct", "Affiliate"),
    n,
    replace = TRUE,
    prob = c(0.34, 0.24, 0.16, 0.18, 0.08)
  ),

  region = sample(
    c("West", "South", "Midwest", "Northeast"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.28, 0.20, 0.17)
  ),

  customer_segment = sample(
    c("Budget", "Standard", "Premium", "Enterprise"),
    n,
    replace = TRUE,
    prob = c(0.35, 0.40, 0.20, 0.05)
  )
)

# Date features for synthetic signal
dt[, month_num := as.integer(format(event_date, "%m"))]
dt[, year_num  := as.integer(format(event_date, "%Y"))]
dt[, day_index := as.integer(event_date - min(event_date))]

# Base effects
channel_effect <- c(
  Search    = 1.20,
  Social    = 0.85,
  Email     = 1.35,
  Direct    = 1.00,
  Affiliate = 0.65
)

segment_effect <- c(
  Budget     = 0.70,
  Standard   = 1.00,
  Premium    = 1.55,
  Enterprise = 2.40
)

region_effect <- c(
  West      = 1.10,
  South     = 0.95,
  Midwest   = 0.85,
  Northeast = 1.05
)

# Seasonality and trend
dt[, seasonality := 1 + 0.25 * sin(2 * pi * month_num / 12)]
dt[, trend       := 1 + day_index / max(day_index) * 0.35]

# Numeric variables with structure
dt[, impressions := round(
  rgamma(.N, shape = 8, scale = 1200) *
    channel_effect[channel] *
    seasonality *
    trend
)]

dt[, clicks := round(
  impressions * pmin(
    pmax(rnorm(.N, mean = 0.035, sd = 0.012), 0.002),
    0.12
  )
)]

dt[, spend := round(
  impressions * runif(.N, 0.008, 0.025) *
    channel_effect[channel] *
    region_effect[region],
  2
)]

dt[, conversions := rpois(
  .N,
  lambda = pmax(
    clicks *
      runif(.N, 0.025, 0.12) *
      segment_effect[customer_segment] *
      region_effect[region],
    0.01
  )
)]

dt[, revenue := round(
  conversions *
    rgamma(.N, shape = 4, scale = 90) *
    segment_effect[customer_segment] *
    runif(.N, 0.75, 1.35),
  2
)]

dt[, ctr := clicks / pmax(impressions, 1)]
dt[, cvr := conversions / pmax(clicks, 1)]
dt[, cpc := spend / pmax(clicks, 1)]
dt[, roas := revenue / pmax(spend, 1)]

# Some less-behaved variables for QA
dt[, noise_normal := rnorm(.N, mean = 100, sd = 15)]
dt[, skewed_score := rgamma(.N, shape = 2, scale = 20)]
dt[, binary_flag := rbinom(.N, size = 1, prob = 0.28)]

# Deliberate correlated variables
dt[, spend_lag_proxy := spend * runif(.N, 0.85, 1.15) + rnorm(.N, 0, 25)]
dt[, revenue_proxy   := revenue * runif(.N, 0.90, 1.10) + rnorm(.N, 0, 100)]

# Deliberate outliers
outlier_rows <- sample(seq_len(n), size = 35)
dt[outlier_rows, spend := spend * runif(.N, 4, 9)]
dt[outlier_rows, revenue := revenue * runif(.N, 3, 7)]
dt[outlier_rows, impressions := round(impressions * runif(.N, 2, 5))]

# Deliberate missingness
dt[sample(.N, 150), revenue := NA_real_]
dt[sample(.N, 120), cpc := NA_real_]
dt[sample(.N, 90), customer_segment := NA_character_]
dt[sample(.N, 80), ctr := NA_real_]

# Deliberate zero-heavy variable
dt[, zero_heavy_metric := fifelse(
  runif(.N) < 0.72,
  0,
  round(rgamma(.N, shape = 2, scale = 15), 2)
)]

# Clean helper columns if you do not want them in report
# Keeping them can also help QA univariate behavior.
# dt[, c("month_num", "year_num", "day_index", "seasonality", "trend") := NULL]


# ----------------------------
# 2. Define report inputs
# ----------------------------

UnivariateVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "binary_flag",
  "zero_heavy_metric",
  "channel",
  "region",
  "customer_segment"
)

CorrVars <- c(
  "impressions",
  "clicks",
  "spend",
  "spend_lag_proxy",
  "conversions",
  "revenue",
  "revenue_proxy",
  "ctr",
  "cvr",
  "cpc",
  "roas",
  "noise_normal",
  "skewed_score",
  "zero_heavy_metric"
)

TrendVars <- c(
  "impressions",
  "clicks",
  "spend",
  "conversions",
  "revenue",
  "ctr",
  "cvr",
  "cpc",
  "roas"
)

OutputPath <- getwd()


# ----------------------------
# 4. Build Catboost model
# ----------------------------

target <- "conversions"
features <- c(
  "impressions", "clicks", "spend",
  "seasonality", "trend",
  "channel", "region", "customer_segment"
)

id_cols <- setdiff(names(dt), c(target, features))

# Run function
ModelObject <- AutoQuant::AutoCatBoostRegression(

  # GPU or CPU and the number of available GPUs
  task_type = 'GPU',
  NumGPUs = 1,
  NumOfParDepPlots = 0,

  # Metadata args
  OutputSelection = c('Importances','EvalMetrics','Score_TrainData'),
  ModelID = 'Test_Model_1',
  model_path = getwd(),
  metadata_path = getwd(),
  ReturnModelObjects = TRUE,

  # Data args
  data = dt,
  TargetColumnName = "conversions",
  FeatureColNames = features,
  IDcols = id_cols,
  TransformNumericColumns = NULL,
  Methods = c('Asinh','Asin','Log','LogPlus1','Sqrt','Logit'),
  Trees = 200
)

# ----------------------------
# 5. Run artifact generator
# ----------------------------

ModelInsightsArtifacts <- generate_regression_model_insights_artifacts(

  # Model object from AutoCatBoostRegression()
  ModelObject = ModelObject,

  # Important for CatBoost-specific importance / interaction handling
  Algo = "catboost",

  # Explicitly pass these so the generator does not have to infer
  TargetColumnName = target,
  PredictionColumnName = "Predict",
  FeatureColumnNames = features,

  # Good candidates for segment and Simpson's Paradox / by-variable checks
  SegmentVars = c("channel", "region", "customer_segment"),
  ByVars = c("channel", "region", "customer_segment"),

  # Optional, only use this if one of your columns is truly a date/time var
  DateVar = NULL,

  # Optional ID columns retained in top-error tables
  IDColumn = if (length(id_cols) > 0L) id_cols[1L] else NULL,

  # Visual theme
  Theme = "dark",

  # First run: keep everything dynamic, no exports
  OutputPath = NULL
)

ReportPath <- RegressionModelInsightsReport(
  artifacts = ModelInsightsArtifacts,
  OutputPath = getwd(),
  OutputFile = "Regression_ModelInsights_Report.html",
  Quiet = FALSE
)

Regression SHAP Artifact Generator Example

generate_regression_shap_analysis_artifacts() consumes precomputed Shap_ columns from AutoQuant modeling/scoring outputs. It does not compute SHAP values, call predict(), require a model object, or use a SHAP backend package.

library(data.table)
library(AutoQuant)

set.seed(123)

# Purpose-built regression QA dataset
n <- 15000L

dt <- data.table(
  IDcol_1 = seq_len(n),
  IDcol_2 = sample(100000:999999, n, TRUE),
  Date = as.Date("2023-01-01") + sample(0:540, n, TRUE),
  
  Channel = sample(c("Search", "Social", "Email", "Direct", "Affiliate"), n, TRUE,
                   prob = c(0.30, 0.24, 0.18, 0.18, 0.10)),
  Region = sample(c("West", "Midwest", "South", "Northeast"), n, TRUE),
  CustomerTier = sample(c("Bronze", "Silver", "Gold", "Platinum"), n, TRUE,
                        prob = c(0.42, 0.30, 0.20, 0.08)),
  
  Spend = round(rgamma(n, shape = 4, scale = 60), 2),
  Clicks = rpois(n, lambda = 40),
  Impressions = rpois(n, lambda = 1800),
  TenureMonths = sample(1:72, n, TRUE),
  DiscountRate = pmin(pmax(rbeta(n, 2, 8), 0), 0.75),
  CompetitorIndex = rnorm(n, 100, 15),
  Seasonality = sin(2 * pi * as.integer(format(as.Date("2023-01-01") + sample(0:540, n, TRUE), "%j")) / 365)
)

# Add correlated / engineered features
dt[, CTR := Clicks / pmax(Impressions, 1)]
dt[, SpendPerClick := Spend / pmax(Clicks, 1)]
dt[, IsHolidaySeason := fifelse(format(Date, "%m") %in% c("11", "12"), "Yes", "No")]

# Segment effects
dt[, ChannelEffect := fifelse(Channel == "Search", 45,
                              fifelse(Channel == "Social", 25,
                                      fifelse(Channel == "Email", 18,
                                              fifelse(Channel == "Affiliate", -10, 5))))]

dt[, RegionEffect := fifelse(Region == "West", 20,
                             fifelse(Region == "South", -12,
                                     fifelse(Region == "Northeast", 8, 0)))]

dt[, TierEffect := fifelse(CustomerTier == "Platinum", 80,
                           fifelse(CustomerTier == "Gold", 45,
                                   fifelse(CustomerTier == "Silver", 20, 0)))]

# Explicit nonlinear and interaction-like signal for model to learn
dt[, Spend_Channel_Interaction := fifelse(Channel == "Search", Spend * 0.12,
                                          fifelse(Channel == "Social", Spend * 0.08,
                                                  fifelse(Channel == "Affiliate", -Spend * 0.04, Spend * 0.03)))]

dt[, Discount_Tier_Interaction := fifelse(CustomerTier %in% c("Gold", "Platinum"),
                                          DiscountRate * 130,
                                          -DiscountRate * 35)]

dt[, Time_Channel_Interaction := fifelse(Channel == "Email", Seasonality * 35,
                                         fifelse(Channel == "Search", Seasonality * 22, Seasonality * 8))]

# Target
dt[, Adrian :=
     120 +
     0.55 * Spend +
     1.80 * Clicks +
     0.015 * Impressions +
     18 * log1p(TenureMonths) -
     65 * DiscountRate -
     0.30 * CompetitorIndex +
     ChannelEffect +
     RegionEffect +
     TierEffect +
     Spend_Channel_Interaction +
     Discount_Tier_Interaction +
     Time_Channel_Interaction +
     rnorm(.N, 0, 35)
]

# Keep target positive-ish for transforms if desired
dt[, Adrian := pmax(Adrian, 1)]

FeatureCols <- setdiff(
  names(dt),
  c(
    "IDcol_1", "IDcol_2", "Adrian",
    "ChannelEffect", "RegionEffect", "TierEffect"
  )
)

# Run model with SHAP
TestModel <- AutoQuant::AutoCatBoostRegression(
  ReturnShap = TRUE,
  
  TrainOnFull = FALSE,
  task_type = "GPU",
  NumGPUs = 1,
  DebugMode = FALSE,
  
  OutputSelection = c(
    "Importances",
    "EvalPlots",
    "EvalMetrics",
    "Score_TrainData"
  ),
  ModelID = "Regression_SHAP_QA_Model",
  model_path = normalizePath("./"),
  metadata_path = normalizePath("./"),
  SaveModelObjects = FALSE,
  SaveInfoToPDF = FALSE,
  ReturnModelObjects = TRUE,
  
  data = dt,
  ValidationData = NULL,
  TestData = NULL,
  TargetColumnName = "Adrian",
  FeatureColNames = FeatureCols,
  PrimaryDateColumn = "Date",
  WeightsColumnName = NULL,
  IDcols = c("IDcol_1", "IDcol_2"),
  TransformNumericColumns = "Adrian",
  Methods = c("BoxCox", "Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  
  eval_metric = "RMSE",
  eval_metric_value = 1.5,
  loss_function = "RMSE",
  loss_function_value = 1.5,
  MetricPeriods = 10L,
  NumOfParDepPlots = min(length(FeatureCols), 20L),
  
  PassInGrid = NULL,
  GridTune = FALSE,
  MaxModelsInGrid = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 60 * 60,
  BaselineComparison = "default",
  
  langevin = FALSE,
  diffusion_temperature = 10000,
  Trees = 700,
  Depth = 8,
  L2_Leaf_Reg = NULL,
  RandomStrength = 1,
  BorderCount = 128,
  LearningRate = NULL,
  RSM = 1,
  BootStrapType = NULL,
  GrowPolicy = "SymmetricTree",
  model_size_reg = 0.5,
  feature_border_type = "GreedyLogSum",
  sampling_unit = "Object",
  subsample = NULL,
  score_function = "Cosine",
  min_data_in_leaf = 1
)

scored_dt <- data.table::rbindlist(
  list(TestModel$TrainData, TestModel$TestData),
  use.names = TRUE,
  fill = TRUE
)

# Generate all currently supported Regression SHAP artifacts
ShapArtifacts <- AutoQuant::generate_regression_shap_analysis_artifacts(
  data = scored_dt,
  target_col = "Adrian",
  prediction_col = "Predict",
  feature_cols = FeatureCols,
  shap_prefix = "Shap_",
  
  id_cols = c("IDcol_1", "IDcol_2"),
  model_name = "Regression_SHAP_QA_Model",
  data_name = "Purpose-built SHAP QA dataset",
  
  DateVar = "Date",
  date_aggregation = "month",
  
  ByVars = c("Channel", "Region", "CustomerTier", "IsHolidaySeason"),
  
  selected_features = c(
    "Spend",
    "Clicks",
    "Impressions",
    "CTR",
    "SpendPerClick",
    "TenureMonths",
    "DiscountRate",
    "CompetitorIndex",
    "Seasonality",
    "Channel",
    "Region",
    "CustomerTier",
    "IsHolidaySeason",
    "Spend_Channel_Interaction",
    "Discount_Tier_Interaction",
    "Time_Channel_Interaction"
  ),
  
  local_row_ids = c(1L, 25L, 100L, 500L, 1000L),
  
  top_n = 25L,
  max_dependence_rows = 7000L,
  max_segment_levels = 20L,
  max_byvars = 4L,
  
  include_dependence = TRUE,
  include_segments = TRUE,
  include_time = TRUE,
  include_local = TRUE,
  include_interactions = TRUE,
  
  include_plots = TRUE,
  max_feature_effect_plots = 12L,
  max_dependence_plots = 12L,
  max_segment_plots = 4L,
  max_time_plots = 4L,
  max_local_plots = 5L,
  plot_top_n = 15L
)

# Quick artifact inventory
artifact_inventory <- rbindlist(lapply(names(ShapArtifacts$artifacts), function(nm) {
  a <- ShapArtifacts$artifacts[[nm]]
  data.table(
    artifact = nm,
    label = a$label,
    type = a$type,
    section = a$section,
    lens = a$metadata$lens %||% NA_character_,
    plot_type = a$metadata$plot_type %||% NA_character_
  )
}), fill = TRUE)

print(artifact_inventory)

# Preferred: render the existing artifact result without recomputing artifacts
reg_artifacts <- ShapArtifacts
ReportPath <- AutoQuant::RegressionShapAnalysisReport(
  artifact_result = reg_artifacts,
  OutputPath = normalizePath("./"),
  OutputFile = "regression_shap_analysis_full_qa_report.html",
  Title = "Regression SHAP Analysis Full QA Report",
  Open = TRUE,
  Quiet = FALSE
)

# Convenience wrapper: pass analytical arguments through ...
ConvenienceReportPath <- AutoQuant::RegressionShapAnalysisReport(
  data = scored_dt,
  OutputPath = normalizePath("./"),
  OutputFile = "regression_shap_analysis_convenience_report.html",
  Title = "Regression SHAP Analysis Convenience Report",
  target_col = "Adrian",
  prediction_col = "Predict",
  feature_cols = FeatureCols,
  id_cols = c("IDcol_1", "IDcol_2"),
  model_name = "Regression_SHAP_QA_Model",
  data_name = "Purpose-built SHAP QA dataset",
  
  DateVar = "Date",
  date_aggregation = "month",
  ByVars = c("Channel", "Region", "CustomerTier", "IsHolidaySeason"),
  selected_features = FeatureCols,
  local_row_ids = c(1L, 25L, 100L, 500L, 1000L),
  top_n = 25L,
  
  include_dependence = TRUE,
  include_segments = TRUE,
  include_time = TRUE,
  include_local = TRUE,
  include_interactions = TRUE,

  Open = TRUE,
  Quiet = FALSE
)

The generator-first workflow is preferred: use generate_regression_shap_analysis_artifacts() for analytical options, then pass the result to RegressionShapAnalysisReport(artifact_result = reg_artifacts, ...). The report function is a renderer. It can still call the generator as a convenience wrapper by passing data plus analytical arguments through ..., but artifact_result mode avoids recomputing artifacts.

The Shap_ prefix maps each contribution column to the source model variable by stripping the prefix, such as Shap_Impressions -> Impressions. ID, segment, date, target, and prediction columns may be present, but they are not treated as SHAP features unless they have matching Shap_ columns. When include_plots = TRUE, plot artifacts are created with AutoPlots high-level functions. Segment heatmaps use signed mean SHAP. Interaction diagnostics are optional binned/leveled candidate interaction surfaces from ordinary Shap_ columns; axes are actual source feature value bins/levels and heatmap values are signed mean SHAP for the attributed feature. Pairwise interaction diagnostics are canonical unordered pairs, so A x B and B x A are treated as the same non-directional analytical object. They are not exact SHAP interaction value decompositions.

Missing or insufficient interaction inputs do not fail SHAP generation. When interaction analysis is requested but candidate pair columns, source feature columns, usable SHAP columns, rows, or unique value combinations are unavailable, AutoQuant emits an interaction_diagnostics table with status, reason_code, reason, severity, required/available columns, and a recommendation, then skips ranking, surface, and heatmap artifacts. Effect-curve generation is independent of interaction analysis; effect-curves-only runs do not attempt interaction generation.

Binary Classification Model Insights Artifact Generator Example

generate_binary_classification_model_insights_artifacts() consumes scored binary classification output data. It does not require a full model object. The generator creates structured plot/table artifacts for threshold diagnostics, ROC / PR, calibration, lift/gains, prediction distribution, feature effects, segment diagnostics, and time diagnostics. Native report rendering should use the artifact result through BinaryClassificationModelInsightsReport().

library(data.table)
library(AutoQuant)

set.seed(321)
n <- 1000L

dt <- data.table(
  Date = as.Date("2025-01-01") + sample(0:240, n, TRUE),
  Channel = sample(c("Direct", "Search", "Social", "Email"), n, TRUE),
  Region = sample(c("West", "Midwest", "South", "Northeast"), n, TRUE),
  Spend = rgamma(n, shape = 5, scale = 40),
  Clicks = rpois(n, lambda = 60),
  CustomerTier = sample(c("Bronze", "Silver", "Gold", "Platinum"), n, TRUE)
)

dt[, score_linear :=
  -1.2 +
  0.008 * Spend +
  0.015 * Clicks +
  fifelse(Channel == "Search", 0.55, 0) +
  fifelse(CustomerTier == "Platinum", 0.7, 0) +
  fifelse(Region == "West", 0.25, 0)
]
dt[, Predict := 1 / (1 + exp(-score_linear))]
dt[, Target := fifelse(runif(.N) <= Predict, "Yes", "No")]
dt[, PredictedClass := fifelse(Predict >= 0.5, "Yes", "No")]

BinaryModelInsightsArtifacts <- AutoQuant::generate_binary_classification_model_insights_artifacts(
  data = dt,
  target_col = "Target",
  prediction_col = "Predict",
  predicted_class_col = "PredictedClass",
  positive_class = "Yes",
  feature_cols = c("Spend", "Clicks", "Channel", "Region", "CustomerTier"),
  model_name = "Binary_Model_Insights_QA_Model",
  data_name = "Binary scored QA data",
  DateVar = "Date",
  date_aggregation = "month",
  ByVars = c("Channel", "Region", "CustomerTier"),
  Threshold = 0.5,
  OptimizeMetric = "F1"
)

print(BinaryModelInsightsArtifacts$metadata$artifact_index)

ReportPath <- AutoQuant::BinaryClassificationModelInsightsReport(
  artifacts = BinaryModelInsightsArtifacts,
  OutputPath = normalizePath("./"),
  OutputFile = "binary_classification_model_insights_report.html",
  Quiet = FALSE
)

Binary Classification SHAP Artifact Generator Example

generate_binary_classification_shap_analysis_artifacts() consumes precomputed Shap_ columns from binary classification modeling/scoring outputs. It does not compute SHAP values, call predict(), require a model object, or use a SHAP backend package. positive_class and prediction_scale are explicit so the report can describe whether SHAP values are contributions toward the positive class on probability, logit, margin, or unknown scale.

library(data.table)
library(AutoQuant)

set.seed(123)
n <- 300L
x1 <- runif(n, 0, 100)
x2 <- runif(n, 0, 50)
seg <- sample(c("A", "B", "C"), n, TRUE)
date <- as.Date("2024-01-01") + sample(0:180, n, TRUE)
eta <- -1 + 0.025 * x1 + 0.03 * x2 + ifelse(seg == "A", 0.7, ifelse(seg == "B", -0.2, 0))
p <- 1 / (1 + exp(-eta))

dt <- data.table(
  Target = ifelse(runif(n) < p, "Yes", "No"),
  Predict = p,
  PredictedClass = ifelse(p >= 0.5, "Yes", "No"),
  Independent_Variable1 = x1,
  Independent_Variable2 = x2,
  Factor_1 = seg,
  IDCol_1 = seq_len(n),
  IDCol_2 = sample(1000:9999, n, TRUE),
  Date = date,
  Shap_Independent_Variable1 = 0.01 * x1 + ifelse(x2 > 25, 0.15, -0.05) + ifelse(seg == "A", 0.08, 0) + rnorm(n, 0, 0.02),
  Shap_Independent_Variable2 = 0.01 * x2 + ifelse(x1 > 50, 0.1, -0.04) + rnorm(n, 0, 0.02),
  Shap_Factor_1 = ifelse(seg == "A", 0.15, ifelse(seg == "B", -0.05, 0.02)) + rnorm(n, 0, 0.02)
)

BinaryShapArtifacts <- AutoQuant::generate_binary_classification_shap_analysis_artifacts(
  data = dt,
  target_col = "Target",
  prediction_col = "Predict",
  predicted_class_col = "PredictedClass",
  positive_class = "Yes",
  prediction_scale = "probability",
  threshold = 0.5,
  DateVar = "Date",
  date_aggregation = "month",
  ByVars = "Factor_1",
  id_cols = c("IDCol_1", "IDCol_2"),
  selected_features = c("Independent_Variable1", "Independent_Variable2", "Factor_1"),
  local_row_ids = c(1L, 2L),
  top_n = 3L,
  include_dependence = TRUE,
  include_segments = TRUE,
  include_time = TRUE,
  include_local = TRUE,
  include_interactions = TRUE,
  include_threshold_context = TRUE,
  include_class_balance = TRUE,
  include_plots = TRUE
)

or_na <- function(x) if (is.null(x)) NA_character_ else x
artifact_inventory <- rbindlist(lapply(names(BinaryShapArtifacts$artifacts), function(nm) {
  a <- BinaryShapArtifacts$artifacts[[nm]]
  data.table(
    artifact = nm,
    label = a$label,
    type = a$type,
    section = a$section,
    lens = or_na(a$metadata$lens),
    plot_type = or_na(a$metadata$plot_type)
  )
}), fill = TRUE)

print(artifact_inventory)

# Preferred: render the existing artifact result without recomputing artifacts
binary_artifacts <- BinaryShapArtifacts
BinaryShapReportPath <- AutoQuant::BinaryClassificationShapAnalysisReport(
  artifact_result = binary_artifacts,
  OutputPath = normalizePath("./"),
  OutputFile = "binary_classification_shap_analysis_full_qa_report.html",
  Title = "Binary Classification SHAP Analysis Full QA Report",
  Open = TRUE,
  Quiet = FALSE
)

# Convenience wrapper: pass analytical arguments through ...
BinaryShapConvenienceReportPath <- AutoQuant::BinaryClassificationShapAnalysisReport(
  data = dt,
  OutputPath = normalizePath("./"),
  OutputFile = "binary_classification_shap_analysis_convenience_report.html",
  Title = "Binary Classification SHAP Analysis Convenience Report",
  target_col = "Target",
  prediction_col = "Predict",
  predicted_class_col = "PredictedClass",
  positive_class = "Yes",
  prediction_scale = "probability",
  threshold = 0.5,

  DateVar = "Date",
  date_aggregation = "month",
  ByVars = "Factor_1",
  id_cols = c("IDCol_1", "IDCol_2"),
  selected_features = c("Independent_Variable1", "Independent_Variable2", "Factor_1"),
  local_row_ids = c(1L, 2L),
  top_n = 3L,

  include_dependence = TRUE,
  include_segments = TRUE,
  include_time = TRUE,
  include_local = TRUE,
  include_interactions = TRUE,
  include_threshold_context = TRUE,
  include_class_balance = TRUE,

  Open = TRUE,
  Quiet = FALSE
)

The generator-first workflow is preferred: use generate_binary_classification_shap_analysis_artifacts() for analytical options, then pass the result to BinaryClassificationShapAnalysisReport(artifact_result = binary_artifacts, ...). The report function is a renderer. It can still call the generator as a convenience wrapper by passing data plus analytical arguments through ..., including positive_class, prediction_scale, threshold, and predicted_class_col, but artifact_result mode avoids recomputing artifacts.

Binary SHAP artifacts include positive-class overview text, diagnostics/config tables, global importance, categorical/binned numeric level importance, dependence, segment, time, threshold context, class balance/outcome context, local explanations, and binned/leveled interaction diagnostics. Interaction surfaces use actual source feature bins/levels on the axes and signed mean SHAP as the heatmap value. Pairwise interaction diagnostics are canonical unordered pairs, so A x B and B x A are treated as the same non-directional analytical object. They are not exact pairwise SHAP interaction decompositions. Multiclass SHAP is deferred.

BinaryClassificationModelInsightsReport() Example

library(data.table)
library(AutoQuant)

set.seed(321)
n <- 1000L

scored_dt <- data.table(
  Date = as.Date("2025-01-01") + sample(0:240, n, TRUE),
  Channel = sample(c("Direct", "Search", "Social", "Email"), n, TRUE),
  Region = sample(c("West", "Midwest", "South", "Northeast"), n, TRUE),
  Spend = rgamma(n, shape = 5, scale = 40),
  Clicks = rpois(n, lambda = 60),
  CustomerTier = sample(c("Bronze", "Silver", "Gold", "Platinum"), n, TRUE)
)

scored_dt[, score_linear :=
  -1.2 +
  0.008 * Spend +
  0.015 * Clicks +
  fifelse(Channel == "Search", 0.55, 0) +
  fifelse(CustomerTier == "Platinum", 0.7, 0) +
  fifelse(Region == "West", 0.25, 0)
]
scored_dt[, Predict := 1 / (1 + exp(-score_linear))]
scored_dt[, Target := fifelse(runif(.N) <= Predict, "Yes", "No")]
scored_dt[, PredictedClass := fifelse(Predict >= 0.5, "Yes", "No")]

BinaryArtifacts <- AutoQuant::generate_binary_classification_model_insights_artifacts(
  data = scored_dt,
  target_col = "Target",
  prediction_col = "Predict",
  predicted_class_col = "PredictedClass",
  positive_class = "Yes",
  feature_cols = c("Spend", "Clicks", "Channel", "Region", "CustomerTier"),
  model_name = "Binary_Model_Insights_QA_Model",
  data_name = "Binary scored QA data",
  DateVar = "Date",
  date_aggregation = "month",
  ByVars = c("Channel", "Region", "CustomerTier"),
  Threshold = 0.5,
  OptimizeMetric = "F1"
)

ReportPath <- AutoQuant::BinaryClassificationModelInsightsReport(
  artifacts = BinaryArtifacts,
  OutputPath = normalizePath("./"),
  OutputFile = "binary_classification_model_insights_report.html",
  Quiet = FALSE
)

Legacy MultiClass ModelInsightsReport() Example

# Create some dummy correlated data
data <- AutoQuant::FakeDataGenerator(
  Correlation = 0.85,
  N = 10000L,
  ID = 2L,
  ZIP = 0L,
  AddDate = FALSE,
  Classification = FALSE,
  MultiClass = TRUE)

# Copy data
data1 <- data.table::copy(data)

# Feature Colnames
Features <- c(names(data1)[!names(data1) %in% c('IDcol_1','IDcol_2','Adrian')])

# Run function
ModelObject <- AutoQuant::AutoCatBoostMultiClass(
  
  # GPU or CPU and the number of available GPUs
  task_type = 'GPU',
  NumGPUs = 1,
  NumOfParDepPlots = length(Features),
  
  # Metadata args
  OutputSelection = c('Importances', 'EvalPlots', 'EvalMetrics', 'Score_TrainData'),
  ModelID = 'Test_Model_1',
  model_path = getwd(),
  metadata_path = getwd(),
  ReturnModelObjects = TRUE,
  
  # Data args
  data = data,
  TargetColumnName = 'Adrian',
  FeatureColNames = Features,
  IDcols = c('IDcol_1','IDcol_2'))

# Legacy compatibility multiclass report.
# Replace once a multiclass artifact-first report exists.
AutoQuant::ModelInsightsReport(
  TrainDataInclude = TRUE,
  FeatureColumnNames = Features,
  SampleSize = 100000,
  ModelObject = ModelObject,
  ModelID = 'Test_Model_1',
  OutputPath = getwd())

Panel Data Forecasting

Expand to view content

Code Example: AutoCatBoostCARMA()

# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# Out-of-Sample Feature + Grid Tuning of AutoQuant::AutoCatBoostCARMA()
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

# Set up your output file path for saving results as a .csv
Path <- "C:/YourPathHere"

# Run on GPU or CPU (some options in the grid tuning force usage of CPU for some runs)
TaskType = "GPU"

# Define number of CPU threads to allow data.table to utilize
data.table::setDTthreads(percent = max(1L, parallel::detectCores()-2L))

# Load data
data <- data <- data.table::fread("https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1")

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- Rodeo::TimeSeriesFill(
  data,
  DateColumnName = "Date",
  GroupVariables = c("Store","Dept"),
  TimeUnit = "weeks",
  FillType = "maxmax",
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE)

# Set negative numbers to 0
data <- data[, Weekly_Sales := data.table::fifelse(Weekly_Sales < 0, 0, Weekly_Sales)]

# Remove IsHoliday column
data[, IsHoliday := NULL]

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c("Date", "Store", "Dept")]

# Change data types
data[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]
xregs[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]

# Subset data so we have an out of time sample
data1 <- data.table::copy(data[, ID := 1L:.N, by = c("Store","Dept")][ID <= 125L][, ID := NULL])
data[, ID := NULL]

# Define values for SplitRatios and FCWindow Args
N1 <- data1[, .N, by = c("Store","Dept")][1L, N]
N2 <- xregs[, .N, by = c("Store","Dept")][1L, N]

# Setup Grid Tuning & Feature Tuning data.table using a cross join of vectors
Tuning <- data.table::CJ(
  TimeWeights = c("None",0.999),
  MaxTimeGroups = c("weeks","months"),
  TargetTransformation = c("TRUE","FALSE"),
  Difference = c("TRUE","FALSE"),
  HoldoutTrain = c(6,18),
  Langevin = c("TRUE","FALSE"),
  NTrees = c(2500,5000),
  Depth = c(6,9),
  RandomStrength = c(0.75,1),
  L2_Leaf_Reg = c(3.0,4.0),
  RSM = c(0.75,"NULL"),
  GrowPolicy = c("SymmetricTree","Lossguide","Depthwise"),
  BootStrapType = c("Bayesian","MVS","No"))

# Remove options that are not compatible with GPU (skip over this otherwise)
Tuning <- Tuning[Langevin == "TRUE" | (Langevin == "FALSE" & RSM == "NULL" & BootStrapType %in% c("Bayesian","No"))]

# Randomize order of Tuning data.table
Tuning <- Tuning[order(runif(.N))]

# Load grid results and remove rows that have already been tested
if(file.exists(file.path(Path, "Walmart_CARMA_Metrics.csv"))) {
  Metrics <- data.table::fread(file.path(Path, "Walmart_CARMA_Metrics.csv"))
  temp <- data.table::rbindlist(list(Metrics,Tuning), fill = TRUE)
  temp <- unique(temp, by = c(4:(ncol(temp)-1)))
  Tuning <- temp[is.na(RunTime)][, .SD, .SDcols = names(Tuning)]
  rm(Metrics,temp)
}

# Define the total number of runs
TotalRuns <- Tuning[,.N]

# Kick off feature + grid tuning
for(Run in seq_len(TotalRuns)) {

  # Print run number
  for(zz in seq_len(100)) print(Run)

  # Use fresh data for each run
  xregs_new <- data.table::copy(xregs)
  data_new <- data.table::copy(data1)

  # Timer start
  StartTime <- Sys.time()

  # Run carma system
  CatBoostResults <- AutoQuant::AutoCatBoostCARMA(

    # data args
    data = data_new,
    TimeWeights = if(Tuning[Run, TimeWeights] == "None") NULL else as.numeric(Tuning[Run, TimeWeights]),
    TargetColumnName = "Weekly_Sales",
    DateColumnName = "Date",
    HierarchGroups = NULL,
    GroupVariables = c("Store","Dept"),
    TimeUnit = "weeks",
    TimeGroups = if(Tuning[Run, MaxTimeGroups] == "weeks") "weeks" else if(Tuning[Run, MaxTimeGroups] == "months") c("weeks","months") else c("weeks","months","quarters"),

    # Production args
    TrainOnFull = TRUE,
    SplitRatios = c(1 - Tuning[Run, HoldoutTrain] / N2, Tuning[Run, HoldoutTrain] / N2),
    PartitionType = "random",
    FC_Periods = N2-N1,
    TaskType = TaskType,
    NumGPU = 1,
    Timer = TRUE,
    DebugMode = TRUE,

    # Target variable transformations
    TargetTransformation = as.logical(Tuning[Run, TargetTransformation]),
    Methods = c('Asinh','Log','LogPlus1','Sqrt'),
    Difference = as.logical(Tuning[Run, Difference]),
    NonNegativePred = TRUE,
    RoundPreds = FALSE,

    # Calendar-related features
    CalendarVariables = c("week","wom","month","quarter"),
    HolidayVariable = c("USPublicHolidays"),
    HolidayLookback = NULL,
    HolidayLags = c(1,2,3),
    HolidayMovingAverages = c(2,3),

    # Lags, moving averages, and other rolling stats
    Lags = if(Tuning[Run, MaxTimeGroups] == "weeks") c(1,2,3,4,5,8,9,12,13,51,52,53) else if(Tuning[Run, MaxTimeGroups] == "months") list("weeks" = c(1,2,3,4,5,8,9,12,13,51,52,53), "months" = c(1,2,6,12)) else list("weeks" = c(1,2,3,4,5,8,9,12,13,51,52,53), "months" = c(1,2,6,12), "quarters" = c(1,2,3,4)),
    MA_Periods = if(Tuning[Run, MaxTimeGroups] == "weeks") c(2,3,4,5,8,9,12,13,51,52,53) else if(Tuning[Run, MaxTimeGroups] == "months") list("weeks" = c(2,3,4,5,8,9,12,13,51,52,53), "months" = c(2,6,12)) else list("weeks" = c(2,3,4,5,8,9,12,13,51,52,53), "months" = c(2,6,12), "quarters" = c(2,3,4)),
    SD_Periods = NULL,
    Skew_Periods = NULL,
    Kurt_Periods = NULL,
    Quantile_Periods = NULL,
    Quantiles_Selected = NULL,

    # Bonus features
    AnomalyDetection = NULL,
    XREGS = xregs_new,
    FourierTerms = 0,
    TimeTrendVariable = TRUE,
    ZeroPadSeries = NULL,
    DataTruncate = FALSE,

    # ML grid tuning args
    GridTune = FALSE,
    PassInGrid = NULL,
    ModelCount = 5,
    MaxRunsWithoutNewWinner = 50,
    MaxRunMinutes = 60*60,

    # ML evaluation output
    PDFOutputPath = NULL,
    SaveDataPath = NULL,
    NumOfParDepPlots = 0L,

    # ML loss functions
    EvalMetric = "RMSE",
    EvalMetricValue = 1,
    LossFunction = "RMSE",
    LossFunctionValue = 1,

    # ML tuning args
    NTrees = Tuning[Run, NTrees],
    Depth = Tuning[Run, Depth],
    L2_Leaf_Reg = Tuning[Run, L2_Leaf_Reg],
    LearningRate = 0.03,
    Langevin = as.logical(Tuning[Run, Langevin]),
    DiffusionTemperature = 10000,
    RandomStrength = Tuning[Run, RandomStrength],
    BorderCount = 254,
    RSM = if(Tuning[Run, RSM] == "NULL") NULL else as.numeric(Tuning[Run, RSM]),
    GrowPolicy = Tuning[Run, GrowPolicy],
    BootStrapType = Tuning[Run, BootStrapType],
    ModelSizeReg = 0.5,
    FeatureBorderType = "GreedyLogSum",
    SamplingUnit = "Group",
    SubSample = NULL,
    ScoreFunction = "Cosine",
    MinDataInLeaf = 1)

  # Timer End
  EndTime <- Sys.time()

  # Prepare data for evaluation
  Results <- CatBoostResults$Forecast
  data.table::setnames(Results, "Weekly_Sales", "bla")
  Results <- merge(Results, data, by = c("Store","Dept","Date"), all = FALSE)
  Results <- Results[is.na(bla)][, bla := NULL]

  # Create totals and subtotals
  Results <- data.table::groupingsets(
    x = Results,
    j = list(Predictions = sum(Predictions), Weekly_Sales = sum(Weekly_Sales)),
    by = c("Date", "Store", "Dept"),
    sets = list(c("Date", "Store", "Dept"), c("Store", "Dept"), "Store", "Dept", "Date"))
  
  # Fill NAs with "Total" for totals and subtotals
  for(cols in c("Store","Dept")) Results[, eval(cols) := data.table::fifelse(is.na(get(cols)), "Total", get(cols))]

  # Add error measures
  Results[, Weekly_MAE := abs(Weekly_Sales - Predictions)]
  Results[, Weekly_MAPE := Weekly_MAE / Weekly_Sales]

  # Weekly results
  Weekly_MAPE <- Results[, list(Weekly_MAPE = mean(Weekly_MAPE)), by = list(Store,Dept)]

  # Monthly results
  temp <- data.table::copy(Results)
  temp <- temp[, Date := lubridate::floor_date(Date, unit = "months")]
  temp <- temp[, lapply(.SD, sum), by = c("Date","Store","Dept"), .SDcols = c("Predictions", "Weekly_Sales")]
  temp[, Monthly_MAE := abs(Weekly_Sales - Predictions)]
  temp[, Monthly_MAPE := Monthly_MAE / Weekly_Sales]
  Monthly_MAPE <- temp[, list(Monthly_MAPE = mean(Monthly_MAPE)), by = list(Store,Dept)]

  # Collect metrics for Total (feel free to switch to something else or no filter at all)
  Metrics <- data.table::data.table(
    RunNumber = Run,
    Total_Weekly_MAPE = Weekly_MAPE[Store == "Total" & Dept == "Total", Weekly_MAPE],
    Total_Monthly_MAPE = Monthly_MAPE[Store == "Total" & Dept == "Total", Monthly_MAPE],
    Tuning[Run],
    RunTime = EndTime - StartTime)

  # Append to file (not overwrite)
  data.table::fwrite(Metrics, file = file.path(Path, "Walmart_CARMA_Metrics.csv"), append = TRUE)
  
  # Remove objects (clear space before new runs)
  rm(CatBoostResults, Results, temp, Weekly_MAE, Weekly_MAPE, Monthly_MAE, Monthly_MAPE)

  # Garbage collection because of GPU
  gc()
}

Code Example: AutoXGBoostCARMA()

# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# XGBoost Version ----
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

# Load data
data <- data.table::fread("https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1")

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- Rodeo::TimeSeriesFill(
  data,
  DateColumnName = "Date",
  GroupVariables = c("Store","Dept"),
  TimeUnit = "weeks",
  FillType = "maxmax",
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE)

# Set negative numbers to 0
data <- data[, Weekly_Sales := data.table::fifelse(Weekly_Sales < 0, 0, Weekly_Sales)]

# Remove IsHoliday column
data[, IsHoliday := NULL]

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c("Date", "Store", "Dept")]

# Change data types
data[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]
xregs[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]

 # Build forecast
XGBoostResults <- AutoXGBoostCARMA(

  # Data Artifacts
  data = data,
  NonNegativePred = FALSE,
  RoundPreds = FALSE,
  TargetColumnName = "Weekly_Sales",
  DateColumnName = "Date",
  HierarchGroups = NULL,
  GroupVariables = c("Store","Dept"),
  TimeUnit = "weeks",
  TimeGroups = c("weeks","months"),

  # Data Wrangling Features
  ZeroPadSeries = NULL,
  DataTruncate = FALSE,
  SplitRatios = c(1 - 10 / 138, 10 / 138),
  PartitionType = "timeseries",
  AnomalyDetection = NULL,
  EncodingMethod = "binary",

  # Productionize
  FC_Periods = 0,
  TrainOnFull = FALSE,
  NThreads = 8,
  Timer = TRUE,
  DebugMode = FALSE,
  SaveDataPath = NULL,
  PDFOutputPath = NULL,

  # Target Transformations
  TargetTransformation = TRUE,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  Difference = FALSE,

  # Features
  Lags = list("weeks" = seq(1L, 10L, 1L), "months" = seq(1L, 5L, 1L)),
  MA_Periods = list("weeks" = seq(5L, 20L, 5L), "months" = seq(2L, 10L, 2L)),
  SD_Periods = NULL,
  Skew_Periods = NULL,
  Kurt_Periods = NULL,
  Quantile_Periods = NULL,
  Quantiles_Selected = c("q5","q95"),
  XREGS = xregs,
  FourierTerms = 4,
  CalendarVariables = c("week", "wom", "month", "quarter"),
  HolidayVariable = c("USPublicHolidays","EasterGroup","ChristmasGroup","OtherEcclesticalFeasts"),
  HolidayLookback = NULL,
  HolidayLags = 1,
  HolidayMovingAverages = 1:2,
  TimeTrendVariable = TRUE,

  # ML eval args
  TreeMethod = "hist",
  EvalMetric = "RMSE",
  LossFunction = 'reg:squarederror',

  # ML grid tuning
  GridTune = FALSE,
  ModelCount = 5,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,

  # ML args
  NTrees = 300,
  LearningRate = 0.3,
  MaxDepth = 9L,
  MinChildWeight = 1.0,
  SubSample = 1.0,
  ColSampleByTree = 1.0)

Code Example: AutoLightGBMCARMA()

# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# LightGBM Version ----
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

# Load data
data <- data.table::fread('https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1')

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- Rodeo::TimeSeriesFill(
  data,
  DateColumnName = 'Date',
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  FillType = 'maxmax',
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE)

# Set negative numbers to 0
data <- data[, Weekly_Sales := data.table::fifelse(Weekly_Sales < 0, 0, Weekly_Sales)]

# Remove IsHoliday column
data[, IsHoliday := NULL]

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c('Date', 'Store', 'Dept')]

# Change data types
data[, ':=' (Store = as.character(Store), Dept = as.character(Dept))]
xregs[, ':=' (Store = as.character(Store), Dept = as.character(Dept))]

# Build forecast
Results <- AutoLightGBMCARMA(

  # Data Artifacts
  data = data,
  NonNegativePred = FALSE,
  RoundPreds = FALSE,
  TargetColumnName = 'Weekly_Sales',
  DateColumnName = 'Date',
  HierarchGroups = NULL,
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  TimeGroups = c('weeks','months'),

  # Data Wrangling Features
  EncodingMethod = 'binary',
  ZeroPadSeries = NULL,
  DataTruncate = FALSE,
  SplitRatios = c(1 - 10 / 138, 10 / 138),
  PartitionType = 'timeseries',
  AnomalyDetection = NULL,

  # Productionize
  FC_Periods = 0,
  TrainOnFull = FALSE,
  NThreads = 8,
  Timer = TRUE,
  DebugMode = FALSE,
  SaveDataPath = NULL,
  PDFOutputPath = NULL,

  # Target Transformations
  TargetTransformation = TRUE,
  Methods = c('Asinh', 'Asin', 'Log', 'LogPlus1', 'Sqrt', 'Logit'),
  Difference = FALSE,

  # Features
  Lags = list('weeks' = seq(1L, 10L, 1L), 'months' = seq(1L, 5L, 1L)),
  MA_Periods = list('weeks' = seq(5L, 20L, 5L), 'months' = seq(2L, 10L, 2L)),
  SD_Periods = NULL,
  Skew_Periods = NULL,
  Kurt_Periods = NULL,
  Quantile_Periods = NULL,
  Quantiles_Selected = c('q5','q95'),
  XREGS = xregs,
  FourierTerms = 4,
  CalendarVariables = c('week', 'wom', 'month', 'quarter'),
  HolidayVariable = c('USPublicHolidays','EasterGroup','ChristmasGroup','OtherEcclesticalFeasts'),
  HolidayLookback = NULL,
  HolidayLags = 1,
  HolidayMovingAverages = 1:2,
  TimeTrendVariable = TRUE,

  # ML eval args
  TreeMethod = 'hist',
  EvalMetric = 'RMSE',
  LossFunction = 'reg:squarederror',

  # Grid tuning args
  GridTune = FALSE,
  GridEvalMetric = 'mae',
  ModelCount = 30L,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,

  # LightGBM Args
  Device_Type = TaskType,
  LossFunction = 'regression',
  EvalMetric = 'MAE',
  Input_Model = NULL,
  Task = 'train',
  Boosting = 'gbdt',
  LinearTree = FALSE,
  Trees = 1000,
  ETA = 0.10,
  Num_Leaves = 31,
  Deterministic = TRUE,

  # Learning Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning-control-parameters
  Force_Col_Wise = FALSE,
  Force_Row_Wise = FALSE,
  Max_Depth = 6,
  Min_Data_In_Leaf = 20,
  Min_Sum_Hessian_In_Leaf = 0.001,
  Bagging_Freq = 1.0,
  Bagging_Fraction = 1.0,
  Feature_Fraction = 1.0,
  Feature_Fraction_Bynode = 1.0,
  Lambda_L1 = 0.0,
  Lambda_L2 = 0.0,
  Extra_Trees = FALSE,
  Early_Stopping_Round = 10,
  First_Metric_Only = TRUE,
  Max_Delta_Step = 0.0,
  Linear_Lambda = 0.0,
  Min_Gain_To_Split = 0,
  Drop_Rate_Dart = 0.10,
  Max_Drop_Dart = 50,
  Skip_Drop_Dart = 0.50,
  Uniform_Drop_Dart = FALSE,
  Top_Rate_Goss = FALSE,
  Other_Rate_Goss = FALSE,
  Monotone_Constraints = NULL,
  Monotone_Constraints_Method = 'advanced',
  Monotone_Penalty = 0.0,
  Forcedsplits_Filename = NULL, # use for AutoStack option; .json file
  Refit_Decay_Rate = 0.90,
  Path_Smooth = 0.0,

  # IO Dataset Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#io-parameters
  Max_Bin = 255,
  Min_Data_In_Bin = 3,
  Data_Random_Seed = 1,
  Is_Enable_Sparse = TRUE,
  Enable_Bundle = TRUE,
  Use_Missing = TRUE,
  Zero_As_Missing = FALSE,
  Two_Round = FALSE,

  # Convert Parameters
  Convert_Model = NULL,
  Convert_Model_Language = 'cpp',

  # Objective Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective-parameters
  Boost_From_Average = TRUE,
  Alpha = 0.90,
  Fair_C = 1.0,
  Poisson_Max_Delta_Step = 0.70,
  Tweedie_Variance_Power = 1.5,
  Lambdarank_Truncation_Level = 30,

  # Metric Parameters (metric is in Core)
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric-parameters
  Is_Provide_Training_Metric = TRUE,
  Eval_At = c(1,2,3,4,5),

  # Network Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#network-parameters
  Num_Machines = 1,

  # GPU Parameters
  # https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu-parameters
  Gpu_Platform_Id = -1,
  Gpu_Device_Id = -1,
  Gpu_Use_Dp = TRUE,
  Num_Gpu = 1)

Code Example: AutoH2OCARMA()

# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# H2O Version ----
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

# Load data
data <- data.table::fread("https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1")

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- Rodeo::TimeSeriesFill(
  data,
  DateColumnName = "Date",
  GroupVariables = c("Store","Dept"),
  TimeUnit = "weeks",
  FillType = "maxmax",
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE)

# Set negative numbers to 0
data <- data[, Weekly_Sales := data.table::fifelse(Weekly_Sales < 0, 0, Weekly_Sales)]

# Remove IsHoliday column
data[, IsHoliday := NULL]

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c("Date", "Store", "Dept")]

# Change data types
data[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]
xregs[, ":=" (Store = as.character(Store), Dept = as.character(Dept))]

# Build forecast
Results <- AutoQuant::AutoH2OCARMA(

  # Data Artifacts
  AlgoType = "drf",
  ExcludeAlgos = NULL,
  data = data,
  TargetColumnName = "Weekly_Sales",
  DateColumnName = "Date",
  HierarchGroups = NULL,
  GroupVariables = c("Dept"),
  TimeUnit = "week",
  TimeGroups = c("weeks","months"),

  # Data Wrangling Features
  SplitRatios = c(1 - 10 / 138, 10 / 138),
  PartitionType = "random",

  # Production args
  FC_Periods = 4L,
  TrainOnFull = FALSE,
  MaxMem = {gc();paste0(as.character(floor(max(32, as.numeric(system("awk '/MemFree/ {print $2}' /proc/meminfo", intern=TRUE)) -32) / 1000000)),"G")},
  NThreads = parallel::detectCores(),
  PDFOutputPath = NULL,
  SaveDataPath = NULL,
  Timer = TRUE,
  DebugMode = TRUE,

  # Target Transformations
  TargetTransformation = FALSE,
  Methods = c("Asinh", "Asin", "Log", "LogPlus1", "Sqrt", "Logit"),
  Difference = FALSE,
  NonNegativePred = FALSE,
  RoundPreds = FALSE,

  # Calendar features
  CalendarVariables = c("week", "wom", "month", "quarter", "year"),
  HolidayVariable = c("USPublicHolidays","EasterGroup",
    "ChristmasGroup","OtherEcclesticalFeasts"),
  HolidayLookback = NULL,
  HolidayLags = 1:7,
  HolidayMovingAverages = 2:7,
  TimeTrendVariable = TRUE,

  # Time series features
  Lags = list("weeks" = c(1:4), "months" = c(1:3)),
  MA_Periods = list("weeks" = c(2:8), "months" = c(6:12)),
  SD_Periods = NULL,
  Skew_Periods = NULL,
  Kurt_Periods = NULL,
  Quantile_Periods = NULL,
  Quantiles_Selected = NULL,

  # Bonus Features
  XREGS = NULL,
  FourierTerms = 2L,
  AnomalyDetection = NULL,
  ZeroPadSeries = NULL,
  DataTruncate = FALSE,

  # ML evaluation args
  EvalMetric = "RMSE",
  NumOfParDepPlots = 0L,

  # ML grid tuning args
  GridTune = FALSE,
  GridStrategy = "Cartesian",
  ModelCount = 5,
  MaxRuntimeSecs = 60*60*24,
  StoppingRounds = 10,

  # ML Args
  NTrees = 1000L,
  MaxDepth = 20,
  SampleRate = 0.632,
  MTries = -1,
  ColSampleRatePerTree = 1,
  ColSampleRatePerTreeLevel  = 1,
  MinRows = 1,
  NBins = 20,
  NBinsCats = 1024,
  NBinsTopLevel = 1024,
  HistogramType = "AUTO",
  CategoricalEncoding = "AUTO",
  RandomColNumbers = NULL,
  InteractionColNumbers = NULL,
  WeightsColumn = NULL,

  # ML args
  Distribution = "gaussian",
  Link = "identity",
  RandomDistribution = NULL,
  RandomLink = NULL,
  Solver = "AUTO",
  Alpha = NULL,
  Lambda = NULL,
  LambdaSearch = FALSE,
  NLambdas = -1,
  Standardize = TRUE,
  RemoveCollinearColumns = FALSE,
  InterceptInclude = TRUE,
  NonNegativeCoefficients = FALSE)

Time Series Forecasting

Expand to view content

Time Series Forecasting Description

There are three sets of functions for single series traditional time series model forecasting. The first set includes the AutoBanditSarima() and AutoBanditNNet() functions. These two offer the most robust fitting strategies. The utilize a multi-armed-bandit to help narrow the search space of available parameter settings. The next batch includes the AutoTBATS(), AutoETS(), and the AutoArfima() functions. These don't utilze the bandit framework. Rather, they run through a near exhaustive search through all their possible settings. Both the bandit set and the non-bandit set utilize parallelism to burn through as many models as possible for a fixed amount of time.

Bandit: AutoBanditSarima
Bandit: AutoBanditNNet
Exhaustive: AutoTBATS
Exhaustive: AutoETS
Exhaustive: AutoArfima

AutoBanditSarima() Example

# Build model
data <- AutoQuant::FakeDataGenerator(Correlation = 0.82, TimeSeries = TRUE, TimeSeriesTimeAgg = "1min")

# Run system
Output <- AutoQuant::AutoBanditSarima(
  data = data,
  SaveFile = NULL,
  ByDataType = FALSE,
  TargetVariableName = "Weekly_Sales",
  DateColumnName = "Date",
  TimeAggLevel = "1min",
  EvaluationMetric = "MAE",
  NumHoldOutPeriods = 12L,
  NumFCPeriods = 16L,
  MaxLags = 10L,
  MaxSeasonalLags = 0L,
  MaxMovingAverages = 3L,
  MaxSeasonalMovingAverages = 0L,
  MaxFourierPairs = 2L,
  TrainWeighting = 0.50,
  MaxConsecutiveFails = 50L,
  MaxNumberModels = 100L,
  MaxRunTimeMinutes = 10L,
  NumberCores = 12,
  DebugMode = FALSE)

# View output
Output$ForecastPlot
Output$ErrorLagMA2x2
Output$Forecast
Output$PerformanceGrid

AutoBanditNNet() Example

# Build model
data <- AutoQuant::FakeDataGenerator(Correlation = 0.82, TimeSeries = TRUE, TimeSeriesTimeAgg = "1min")

# Run system
Output <- AutoQuant::AutoBanditNNet(
  data = data,
  TargetVariableName = "Weekly_Sales",
  DateColumnName = "Date",
  TimeAggLevel = "1min",
  EvaluationMetric = "MAE",
  NumHoldOutPeriods = 12L,
  NumFCPeriods = 16L,
  MaxLags = 10L,
  MaxSeasonalLags = 0L,
  MaxFourierPairs = 2L,
  TrainWeighting = 0.50,
  MaxConsecutiveFails = 50L,
  MaxNumberModels = 100L,
  MaxRunTimeMinutes = 10L,
  NumberCores = 12)
  
# View output
Output$Forecast
Output$PerformanceGrid

AutoTBATS() Example

# Build model
data <- AutoQuant::FakeDataGenerator(Correlation = 0.82, TimeSeries = TRUE, TimeSeriesTimeAgg = "1min")

# Run system
Output <- AutoQuant::AutoTBATS(
  data = data,
  FilePath = getwd(),
  TargetVariableName = "Weekly_Sales",
  DateColumnName = "Date",
  TimeAggLevel = "1min",
  EvaluationMetric = "MAE",
  NumHoldOutPeriods = 12L,
  NumFCPeriods = 16L,
  MaxLags = 10L,
  MaxMovingAverages = 5,
  MaxSeasonalPeriods = 1,
  TrainWeighting = 0.50,
  MaxConsecutiveFails = 50L,
  MaxNumberModels = 100L,
  MaxRunTimeMinutes = 10L,
  NumberCores = 12)

# View output
Output$Forecast
Output$PerformanceGrid

AutoETS() Example

# Build model
data <- AutoQuant::FakeDataGenerator(Correlation = 0.82, TimeSeries = TRUE, TimeSeriesTimeAgg = "1min")

# Run system
Output <- AutoQuant::AutoETS(
  data = data,
  FilePath = getwd(),
  TargetVariableName = "Weekly_Sales",
  DateColumnName = "Date",
  TimeAggLevel = "1min",
  EvaluationMetric = "MAE",
  NumHoldOutPeriods = 12L,
  NumFCPeriods = 16L,
  TrainWeighting = 0.50,
  MaxConsecutiveFails = 50L,
  MaxNumberModels = 100L,
  MaxRunTimeMinutes = 10L,
  NumberCores = 12)

# View output
Output$Forecast
Output$PerformanceGrid

AutoArfima() Example

# Build model
data <- AutoQuant::FakeDataGenerator(Correlation = 0.82, TimeSeries = TRUE, TimeSeriesTimeAgg = "1min")

# Run system
Output <- AutoQuant::AutoArfima(
  data = data,
  FilePath = getwd(),
  TargetVariableName = "Weekly_Sales",
  DateColumnName = "Date",
  TimeAggLevel = "1min",
  EvaluationMetric = "MAE",
  NumHoldOutPeriods = 12L,
  NumFCPeriods = 16L,
  TrainWeighting = 0.50,
  MaxLags = 5, 
  MaxMovingAverages = 5,
  MaxConsecutiveFails = 50L,
  MaxNumberModels = 100L,
  MaxRunTimeMinutes = 10L,
  NumberCores = 12)

# View output
Output$Forecast
Output$PerformanceGrid

Top AI Repos — open-source AI, indexed and scored

Trend

From the README

AutoQuant Reference Manual

AutoQuant vNext Cookbook

vNext Quick Start

Causal Intelligence Planning

Causal Randomized ITT Estimation

Causal Randomized Design Analysis

Causal Experiment Design Planning

Causal Completed-Experiment Evidence

Supervised Learning

Scoring and Realized Outcomes

Model Bundles

Canonical Analytical Artifacts

Canonical Variable Semantics

Business Intent and Lever Management

Decision Management

Decision Valuation Intelligence

Decision Workflow Intelligence

Decision Lifecycle and Organizational Memory

Time-Series Forecasting

CatBoost Forecasting

Panel Forecasting

Hierarchical Forecasting

Panel Strategy Comparison

Intermittent-Demand Forecasting

Funnel Forecasting

Multi-Target Forecasting

Cross-Target Feature Forecasting

Forecasting Capability Planning

Forecasting Experiment Campaigns

Package QA

Legacy API Status

Detailed Architecture Documents

Background

Package Details

Installation

Install ALL R package dependencies for all functions:

Installation Troubleshooting

Code Examples

EDA

Model Readiness

Supervised Learning

CatBoost Builder Artifact Generator

Modern Report API Contract

Optional AutoNLS SHAP effect curves

Typed Artifact Schema Framework

Regression

Binary Classification

MultiClass Classification

Model Scoring

Model Evaluation

Panel Data Forecasting

Time Series Forecasting

Top contributors

Quality score

What to know before adopting

Categories

Details