Complete toolkit to extract ALL chat, agent, and code context data from AI coding assistants for machine learning training.
Automatically discovers and extracts complete conversation history including:
- ✅ User messages & AI responses
- ✅ Code context (file paths, line numbers, snippets)
- ✅ Code diffs and suggested edits
- ✅ Multi-file contexts
- ✅ Tool use and execution results
- ✅ Timestamps and metadata
Extracts from Claude Code / Claude Desktop
- Searches:
~/.claude,~/.claude-code,~/.claude-local,~/.claude-m2,~/.claude-zai - Formats: JSONL session files
- Includes: Messages, tool use, file contexts, diffs
Extracts from Codex (if installed)
- Searches:
~/.codex,~/.codex-local - Formats: Rollout JSONL files
- Includes: User/agent messages, tool results, diffs
Extracts from Cursor (Chat + Composer + Agent) - ALL VERSIONS
- Searches:
~/Library/Application Support/Cursor(macOS) or equivalent - Formats: SQLite databases (
state.vscdb,cursorDiskKV) - Handles:
- Old Chat mode (workspace storage)
- Composer inline storage (v1.x - messages in composerData array)
- Composer separate storage (v1.x-v2.0 transition - messages in bubbleId keys)
- Latest Composer/Agent (v2.0+)
- Includes:
- Code context, selections, diffs
- Suggested edits and code blocks
- Tool results and execution outputs
Extracts from Trae
- Searches:
~/.trae,~/Library/Application Support/Trae - Formats: JSONL and SQLite databases
- Includes: Chat, agent data, tool use, diffs
Extracts from Windsurf
- Searches:
~/Library/Application Support/Windsurfor equivalent - Formats: SQLite databases (VSCode-like format)
- Includes: Chat, agent/flow conversations, code context
Extracts from Continue AI Assistant
- Searches:
~/.continue/sessions/ - Formats: JSON session files
- Includes:
- User/assistant messages
- Tool calls and results
- Reasoning blocks
- Context items
- Workspace information
Extracts from Google Gemini CLI
- Searches:
~/.gemini/tmp/[hash]/chats/ - Formats: JSON session files
- Includes:
- User/assistant messages
- Thoughts (reasoning steps with timestamps)
- Token usage breakdown
- Model information
- Project hash and workspace linking
Extracts from OpenCode (CLI + Desktop)
- Searches:
- CLI:
~/.local/share/opencode/storage/(Linux),~/Library/Application Support/opencode(macOS) - Desktop:
~/.local/share/ai.opencode.app(Linux),~/Library/Application Support/ai.opencode.app(macOS)
- CLI:
- Formats: JSON files (sessions/messages/parts) and Tauri .dat files (desktop)
- Includes:
- User/assistant messages with full conversation hierarchy
- Tool calls and tool results
- Code blocks and text content
- Token usage and cost tracking
- Model and provider information
- Agent mode and session metadata
- Project directory and version info
- Parent/child session relationships
# No dependencies required - uses Python 3 standard library
python3 --version # Ensure Python 3.6+ is installed# Extract from Claude Code
python3 extract_claude_code.py
# Extract from Cursor
python3 extract_cursor.py
# Extract from Codex
python3 extract_codex.py
# Extract from Trae
python3 extract_trae.py
# Extract from Windsurf
python3 extract_windsurf.py
# Extract from Continue
python3 extract_continue.py
# Extract from Gemini CLI
python3 extract_gemini.py
# Extract from OpenCode
python3 extract_opencode.py
# Extract from ALL tools at once
./extract_all.shAll scripts create an extracted_data/ directory with timestamped JSONL files:
extracted_data/
├── claude_code_conversations_20250116_143022.jsonl
├── cursor_complete_20250116_143045.jsonl
├── gemini_conversations_20250116_143145.jsonl
├── codex_conversations_20250116_143102.jsonl
├── trae_conversations_20250116_143115.jsonl
├── windsurf_conversations_20250116_143130.jsonl
├── continue_conversations_20250116_143145.jsonl
└── opencode_conversations_20250116_143200.jsonl
Each conversation is a single JSON line in JSONL format:
{
"messages": [
{
"role": "user",
"content": "How do I fix this TypeScript error?",
"code_context": [
{
"file": "/Users/user/project/src/index.ts",
"code": "const x: string = 123;",
"range": {
"selectionStartLineNumber": 10,
"positionLineNumber": 10
}
}
],
"timestamp": "2025-01-16T14:30:22.123Z"
},
{
"role": "assistant",
"content": "The error occurs because you're assigning a number to a string type...",
"suggested_diffs": [...],
"model": "claude-sonnet-4-5",
"timestamp": "2025-01-16T14:30:25.456Z"
}
],
"source": "cursor-composer",
"name": "TypeScript Type Error Fix",
"created_at": 1705414222000
}Each script follows this pattern:
- Detect Operating System (macOS, Linux, Windows)
- Search Common Locations:
- macOS:
~/Library/Application Support,~/.config,~/ - Linux:
~/.config,~/.local/share,~/ - Windows:
%APPDATA%,%LOCALAPPDATA%,~/
- macOS:
- Find All Installations of the target tool
- Scan Storage Locations:
- SQLite databases (
.vscdb,.db) - JSONL session files
- Project-specific directories
- SQLite databases (
- Extract Complete Data including context and diffs
- Save to Organized JSONL with timestamps
- Format: JSONL files (one event per line)
- Location:
~/.claude/projects/[project]/[session].jsonl - Structure: Event-based with type markers
- Format: SQLite databases
- Locations:
- Workspace:
~/Library/Application Support/Cursor/User/workspaceStorage/[hash]/state.vscdb - Global:
~/Library/Application Support/Cursor/User/globalStorage/state.vscdb
- Workspace:
- Tables:
ItemTable(Chat),cursorDiskKV(Composer/Agent) - Storage Evolution:
- v0.x - v1.x: Chat mode in workspace
ItemTable - v1.x: Composer inline (messages in
composerData.conversation[]) - v1.x - v2.0 transition: Composer separate (messages in
bubbleId:{composer}:{bubble}keys) - v2.0+: Latest format with enhanced metadata
- v0.x - v1.x: Chat mode in workspace
- Keys:
workbench.panel.aichat.view.aichat.chatdata(Chat mode)composerData:{uuid}(Composer metadata + conversation)bubbleId:{composer}:{bubble}(Individual messages - transitional format)codeBlockDiff:{id}(Code block diffs)
- Format: Hybrid (JSONL + SQLite)
- Location: Similar to VSCode/Cursor structure
- Structure: VSCode extension data format
user: Human developer messagesassistant: AI assistant responses
code_context: File selections and code snippetssuggested_diffs: AI-proposed code changestool_use: Code execution, file operationstool_results: Execution outputs, diffs applieddiff_histories: Full edit history
source: Which tool (e.g., "cursor-composer", "claude-code")session_id/composer_id: Unique conversation IDproject_path: Working directorytimestamp: Message timemodel: AI model used (if available)
# Combine all JSONL files
cat extracted_data/*.jsonl > all_conversations.jsonl
# Count total conversations
wc -l all_conversations.jsonl
# Count by source
grep -o '"source":"[^"]*"' all_conversations.jsonl | sort | uniq -cimport json
from datetime import datetime
with open('extracted_data/cursor_complete_20250116.jsonl') as f:
for line in f:
conv = json.loads(line)
created = conv.get('created_at', 0)
if created > 1704067200000: # After Jan 1, 2024
print(json.dumps(conv))import json
with open('extracted_data/cursor_complete.jsonl') as f:
for line in f:
conv = json.loads(line)
if any('suggested_diffs' in m or 'diff_histories' in m
for m in conv['messages']):
print(json.dumps(conv))✅ Complete Conversations:
- Both user prompts AND AI responses
- Multi-turn dialogues
- Full conversation context
✅ Code Context:
- File paths and names
- Selected code snippets
- Line number ranges
- Multi-file selections
✅ Diffs and Edits:
- Suggested code changes
- Applied diffs
- Edit histories
- File modifications
✅ Metadata:
- Timestamps
- Project paths
- Model information
- Conversation names
- Conversations without AI responses (user-only)
- Deleted or archived sessions
- Corrupted database entries
- May include proprietary code
- May include API keys/secrets
- May include personal file paths
- Scan for Secrets:
pip install detect-secrets
detect-secrets scan extracted_data/*.jsonl- Review Sensitive Data:
- Check for API keys, passwords, tokens
- Verify no proprietary code exposed
- Sanitize file paths if needed
- Storage:
- Keep on encrypted drives
- Don't commit to public repositories
- Secure backups recommended
from datasets import load_dataset
dataset = load_dataset(
'json',
data_files='extracted_data/*.jsonl',
split='train'
)
# Filter complete conversations
dataset = dataset.filter(
lambda x: any(m['role'] == 'assistant' for m in x['messages'])
)from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
"unsloth/qwen2.5-coder-7b-instruct",
max_seq_length=4096,
load_in_4bit=True,
)
def format_chat(example):
return {
'text': tokenizer.apply_chat_template(
example['messages'],
tokenize=False
)
}
dataset = dataset.map(format_chat)Problem: Script reports "No installations found"
Solutions:
- Check if the tool is actually installed
- Verify installation location manually
- Add custom path to script:
# Add to find_XXX_installations() function
locations.append(Path("/custom/path/to/tool"))Problem: Extraction completes but no data found
Solutions:
- Verify you've actually used the tool and have chat history
- Check if data is in a non-standard location
- Look for database files manually:
find ~ -name "*.vscdb" -o -name "*.db" 2>/dev/nullProblem: SQLite database is locked
Solutions:
- Close the AI tool before running extraction
- Use read-only mode:
conn = sqlite3.connect(f'file:{db_path}?mode=ro', uri=True)Problem: Cannot read certain files
Solutions:
- Run with appropriate permissions
- Check file ownership
- Copy databases to accessible location first
- Uses
~/Library/Application Supportfor most tools - May need Full Disk Access for system directories
- SQLite databases typically in
~/Library/Application Support/[Tool]/User/
- Uses
~/.configand~/.local/share - Check
~/.local/statefor some tools - May use
$XDG_CONFIG_HOMEif set
- Uses
%APPDATA%and%LOCALAPPDATA% - Paths:
C:\Users\[User]\AppData\Roaming\[Tool] - May need admin privileges for Program Files
- ✅ v2 (0.43+): Composer/Agent in
cursorDiskKV - ✅ v1: Chat in workspace
ItemTable ⚠️ Pre-v0.43: Different format, limited support
- ✅ All versions with JSONL session files
- ✅ Project-based structure
- ✅ Rollout JSONL format
- ✅ Time-based session organization
# Process in chunks
split -l 1000 all_conversations.jsonl chunk_
# Compress for storage
gzip extracted_data/*.jsonl# Use multiprocessing for large scans
from multiprocessing import Pool
with Pool() as pool:
results = pool.map(extract_from_db, db_files)Found a new storage format or tool? Contributions welcome!
- Follow existing script structure
- Add auto-discovery logic
- Extract complete data (messages + context + diffs)
- Output to organized JSONL
- Update this README
MIT License - Use freely for training ML models
This toolkit extracts YOUR OWN data from locally installed AI tools. Users are responsible for:
- Ensuring they have rights to extracted data
- Handling sensitive/proprietary information appropriately
- Complying with tool Terms of Service
- Scanning for secrets before sharing/training
Generated: January 16, 2025 Status: Production Ready Compatibility: Python 3.6+, macOS/Linux/Windows
