taldbt
AI Migration Agent

pip install

Install taldbt as a Python package. Lightweight, no Docker needed.

# Core AI migration agent + CLI
pip install taldbt==0.2.1

# With Streamlit web UI
pip install taldbt[ui]==0.2.1

# With Temporal orchestration
pip install taldbt[temporal]==0.2.1

# Everything
pip install taldbt[all]==0.2.1

CLI usage:

# Launch the web UI
taldbt ui

# Discover and analyze a Talend project
taldbt discover ./my_talend_project

# Full migration to dbt
taldbt migrate ./my_talend_project ./dbt_output

# Check version
taldbt version

Requirements: Python 3.10+ and Ollama for local AI (optional, falls back to free cloud AI).

Docker Setup

The Docker stack includes Streamlit, Ollama (GPU), Temporal, and DuckDB — all pre-configured.

docker pull souravetl/taldbt:latest
docker pull ollama/ollama:latest

# Download docker-compose.yml from Docker Hub description
docker compose up -d

# Pull the AI model
docker exec taldbt-ollama ollama pull qwen3-coder:30b

# Open
# App:      http://localhost:8501
# Temporal: http://localhost:8233

No GPU? Use CPU override:

docker compose -f docker-compose.yml -f docker-compose.cpu.yml up -d

Air-gapped? Use build-dist.bat to create an offline package with tar files.

Streamlit Cloud Deployment

The live app at taldbt.streamlit.app runs on Streamlit Cloud with Cerebras AI.

How it works on cloud:

• Upload your Talend project as a ZIP file (no folder path access)
• AI translation uses Cerebras Qwen3-235B (free tier, 1M tokens/day)
• Groq Qwen3-32B auto-fallback if Cerebras rate-limits
• Temporal runs inline (dbt models in DAG order) — no CLI needed
• Download the generated dbt project as ZIP

To deploy your own instance: Contact [email protected] for enterprise licensing and private deployment.

LLM Provider Chain

taldbt auto-detects and chains LLM providers with intelligent fallback:

Priority: Ollama (local) → Cerebras → Groq → OpenRouter

How it works:

• 549 component KB entries handle ~96% of translations deterministically
• Only ~5% of Java expressions route to the LLM
• If the primary provider returns 429/503, auto-falls back to next
• Health checks cached for 5 minutes to avoid probing on every call
• <think> blocks from reasoning models are stripped automatically

DuckDB Flock Extension:

The flock extension connects DuckDB to the active LLM provider. This enables llm_complete() inside SQL queries for semantic validation — checking if generated SQL matches the original Talend intent. Auto-registers the active model (local or cloud) on every DuckDB connection.

Free providers:

• Cerebras — Qwen3-235B, 1,400 tok/s, 1M tokens/day free. Sign up
• Groq — Qwen3-32B, 535 tok/s, 1,000 req/day free. Sign up

Temporal Workflow Orchestration

taldbt translates Talend's orchestration (tRunJob, tParallelize) into Temporal workflows.

What gets generated:

• workflows.py — Parent/child workflows mirroring Talend job chains
• activities.py — run_dbt_model activity for each data job
• worker.py — Registers workflows + activities on task queue
• run_workflow.py — Triggers the root workflow

Execution flow:

Master_Job (parent)
  ├─ ParallelJobWorkflow (child)
  │   ├─ run_dbt_model("dimproducts_copy")
  │   └─ run_dbt_model("productvendor_copy")
  ├─ ProductSubJobsWorkflow (child)
  │   ├─ run_dbt_model("dimproductcosthistory_copy1")
  │   └─ run_dbt_model("load_dimprodinventory_copy")
  └─ run_dbt_model("shipmethodmysql_op")

Dashboard: Available at http://localhost:8233 when running Docker or local Temporal CLI. Shows workflow status, history, timing, and state transitions.

Frequently Asked Questions

What Talend components are supported?

549 component knowledge base entries covering tMap, tDBInput/Output (all databases), tFilterRow, tAggregateRow, tSortRow, tUniqRow, tJavaRow, tReplicate, tRunJob, tParallelize, tFlowToIterate, and more. Custom Java goes through the AI translation pipeline.

What databases does it handle?

48% MSSQL, 37% Teradata, 14% MySQL from our analysis of 1,595 .item files across 8 GitHub repos. DuckDB handles all dialect translation via sqlglot.

Does it work without a GPU?

Yes. Use cloud AI (Cerebras/Groq — free) or Ollama CPU mode. The knowledge base handles 96% deterministically without any LLM.

Can I use my own LLM?

Yes. Any OpenAI-compatible endpoint works. Set LLM_PROVIDER=custom with your base_url and API key.

Is my data secure?

All processing happens locally (Docker/desktop) or in your Streamlit Cloud instance. No data leaves your environment. API keys are stored in encrypted Streamlit secrets.