Get Pipeline Run Logs

Overview

Retrieve the execution logs for a specific pipeline run. This is useful for debugging failed pipelines or monitoring execution progress.

Tip: Logs can also be downloaded through the Dadosfera interface. Even pipelines created via the API are visible in the UI, where you can view and download logs directly.

Path Parameters

Parameter	Type	Required	Description
`runId`	string	Yes	The pipeline run ID

Run ID Format

The runId is a 32-character hexadecimal identifier returned from pipeline executions.

Example: 3f1a4d12ddbe456c85a48ee653e66475

You can obtain the run ID from:

The response of POST /platform/pipeline/execute
GET /platform/pipeline/:pipelineId/pipeline_run
The Dadosfera UI pipeline monitoring page

Examples

Get Logs for a Pipeline Run

GET /platform/pipeline/pipeline_run/3f1a4d12ddbe456c85a48ee653e66475/logs

Response (200 OK):

{
  "status": true,
  "data": {
    "run_id": "3f1a4d12ddbe456c85a48ee653e66475",
    "logs": [
      {
        "timestamp": "2024-12-18T10:30:01Z",
        "level": "INFO",
        "task_id": "extract_orders",
        "message": "Starting extraction from PostgreSQL"
      },
      {
        "timestamp": "2024-12-18T10:30:15Z",
        "level": "INFO",
        "task_id": "extract_orders",
        "message": "Extracted 50000 rows"
      },
      {
        "timestamp": "2024-12-18T10:31:00Z",
        "level": "INFO",
        "task_id": "load_snowflake",
        "message": "Loading data to Snowflake"
      },
      {
        "timestamp": "2024-12-18T10:31:30Z",
        "level": "INFO",
        "task_id": "load_snowflake",
        "message": "Successfully loaded 50000 rows to PUBLIC.orders"
      }
    ],
    "total_lines": 4
  }
}

Python Example

import requests

BASE_URL = "https://maestro.dadosfera.ai"
RUN_ID = "3f1a4d12ddbe456c85a48ee653e66475"

# Get pipeline run logs
response = requests.get(
    f"{BASE_URL}/platform/pipeline/pipeline_run/{RUN_ID}/logs",
    headers=headers
)

logs = response.json()['data']['logs']
for log in logs:
    print(f"[{log['level']}] {log['timestamp']} - {log['message']}")

Response Structure

Field	Type	Description
`run_id`	string	Pipeline run identifier
`logs`	array	List of log entries
`total_lines`	integer	Total number of log lines

Log Entry Object

Field	Type	Description
`timestamp`	string	ISO 8601 timestamp
`level`	string	Log level: `INFO`, `WARNING`, `ERROR`
`task_id`	string	Airflow task that generated the log
`message`	string	Log message content

Error Responses

404 Not Found

{
  "detail": {
    "status": false,
    "exception_type": "PipelineRunNotFound",
    "traceback": "Pipeline run with id 3f1a4d12ddbe456c85a48ee653e66475 not found",
    "data": null
  }
}

Common Error Messages

Error Message	Cause	Solution
`Connection timeout`	Database unreachable	Check network/VPN configuration
`OutOfMemoryError`	Insufficient memory	Increase memory via `/pipeline/{id}/memory`
`Authentication failed`	Invalid credentials	Update connection credentials
`Table not found`	Source table missing	Verify table exists in source

Accessing Logs via UI

Pipelines created via the API are fully visible in the Dadosfera interface. To download logs:

Navigate to Coletar > Pipelines
Find your pipeline in the list
Click on the pipeline to view runs
Select the specific run
Click Download Logs to get the full log file

Notes

Run IDs can be obtained from GET /platform/pipeline/:pipelineId/pipeline_run