Skip to main content

Bronze Audit Wrapper

Document Version: 1.0
Last Updated: 20-04-2026

01_bronze_audit_wrapper

Purpose​

01_bronze_audit_wrapper is the orchestration and audit notebook for bronze ingestion jobs.

It is usually the notebook configured directly in the Databricks Job task. Its role is to standardize execution and operational logging while delegating actual ingestion to a loader notebook.

Responsibilities​

This notebook is responsible for:

  • reading job widget parameters
  • validating required values
  • resolving the target catalog, schema, and table from target_table_name
  • choosing which loader notebook to run
  • forwarding ingestion parameters to the loader notebook
  • capturing loader success or failure
  • ensuring the audit table exists
  • appending one audit record for the run
  • failing the Databricks task when the loader fails

Execution flow​

Read widgets
  -> validate required parameters
  -> resolve loader notebook path
  -> ensure audit table exists
  -> forward parameters to loader
  -> run loader notebook
  -> parse loader response
  -> write audit row
  -> return success or raise failure

Loader resolution logic​

The notebook uses two parameters to determine which loader notebook to run:

  • default_loader_notebook
  • loader_notebook_map_json

Resolution logic:

  1. Read loader_notebook_map_json as a JSON dictionary.
  2. Check whether the current target_table_name exists as a key.
  3. If yes, use the mapped notebook path.
  4. If not, use default_loader_notebook.

This gives teams a common wrapper for all jobs while allowing exceptions for special tables.

Example​

{
  "deal_dev.bronze.br_cbs_customer": "./02_bronze_autoloader_generic",
  "deal_dev.bronze.br_special_feed": "./02_bronze_special_loader"
}

Parameters forwarded to the loader​

The wrapper forwards these parameters to the loader notebook:

  • target_table_name
  • staging_table_name
  • source_path
  • checkpoint_path
  • source_format
  • source_file_pattern
  • delimiter
  • header
  • null_value
  • output_mode
  • merge_schema
  • overwrite_schema
  • schema_file_path
  • rescued_data_column
  • load_type
  • w_run_date
  • business_keys
  • cleanup_stage_after_finalize
  • w_business_ts
  • w_source_system
  • w_job_name
  • w_task_name
  • w_job_id
  • w_job_run_id
  • w_task_run_id
  • w_job_trigger_type
  • w_job_start_ts

Audit table schema​

If the audit table does not exist, the notebook creates it as a Delta table with columns for:

  • catalog, schema, and table identity
  • target and staging table names
  • loader notebook path
  • load type
  • status
  • run start/end timestamps
  • execution duration
  • source and output row counts
  • run date and business timestamp
  • business keys
  • source/checkpoint/schema paths
  • source system
  • Databricks job metadata
  • message / failure text
  • audit insert timestamp

Audit statuses​

The wrapper effectively records these outcomes:

  • success: loader completed and returned success or no explicit error
  • failed: loader raised an exception or the wrapper could not run it

If the loader returns a JSON payload, the wrapper reads fields such as:

  • status
  • message
  • source_rows
  • run_date_rows_written
  • current_rows_written
  • previous_day_rows_copied
  • staging_table_name
  • w_business_ts

If the loader returns plain text instead of JSON, the wrapper treats it as success with the text stored as a message.

Failure behavior​

The notebook always attempts to append an audit row in the finally block.

After audit logging:

  • if status is failed, it raises an exception so the Databricks task fails
  • otherwise it exits with a JSON payload containing audit status and loader result

This is important because it means audit logging is not skipped when the load fails.

Operational guidance​

Use this notebook as the task notebook​

For consistency, configure the Databricks Job task to point to 01_bronze_audit_wrapper, not directly to the loader notebook.

Keep the audit table shared and stable​

A central audit table makes monitoring and support much easier. Avoid creating a different audit table per source unless there is a strong isolation requirement.

Make target_table_name fully qualified​

The wrapper expects target_table_name in the form:

catalog.schema.table

If it is not fully qualified, the notebook raises an error.

Use valid JSON for loader_notebook_map_json​

Even when empty, pass a valid JSON object:

{}

Common issues​

Missing required widget parameter​

Cause: a required job parameter was not defined in the Databricks task.

Fix: verify the task base parameters and make sure required values are present.

Invalid target_table_name​

Cause: the value is not in catalog.schema.table format.

Fix: provide the full three-part Unity Catalog name.

Loader path not found​

Cause: default_loader_notebook or the mapped notebook path is incorrect.

Fix: verify the workspace-relative notebook path from the job's folder context.