Building the Activity Code Analyzer

The Activity Code Analyzer (actcod) is a Python data pipeline built during the Santander internship to categorize millions of banking transactions by their activity codes (COD_ACT). It maps raw numeric codes to business categories using a reference CSV, with support for multiprocessing to handle large transaction files efficiently.

What it does

The pipeline reads transaction data and a reference mapping file (referencias.csv), then:

1. Loads references — Parses a CSV with Categorias and codigos columns, building a dictionary mapping each integer code to one or more business categories.
2. Processes transactions — Reads transaction files and looks up each row’s activity code against the reference map.
3. Fuzzy matching — Uses difflib.SequenceMatcher for near-matches when exact code lookups fail.
4. Multiprocessing — Distributes work across CPU cores via multiprocessing.Pool for throughput on large datasets.
5. Exports results — Writes categorized output with timestamps and progress logging.

Two script versions ship in the repo:

• Verbose.py — Full-featured pipeline with detailed logging, statistics, and multiprocessing
• verboseOG.py — Earlier single-threaded prototype for validation

Context

This project grew out of the CodigosActividad exploratory work archived in the Santander internship repo (txtversion), where multiple algorithm versions (simple, advanced, SQL-based) were tested before converging on the validated multiprocessing approach in this repository.

Activity code categorization supports fraud detection, customer segmentation, and regulatory reporting — turning opaque numeric codes into actionable business categories.

Tech stack

Development process

The pipeline was built in small, testable steps during the Santander internship. Each version added one capability — reference loading, matching logic, then parallel processing — with official Python docs as the guide for APIs and patterns.

First working script

• Loaded the reference CSV with pandas read_csv and parsed the Categorias and codigos columns into a lookup dictionary.
• Read transaction files row by row and matched each COD_ACT value against that dictionary.
• Wrote categorized output to disk so results could be checked against a small sample before scaling up.
• Deliverable: verboseOG.py — single-threaded prototype used to confirm mappings on limited data.

Logic validation (single-threaded)

• Refined categorization rules on smaller files without multiprocessing so bugs were easier to trace.
• Added difflib SequenceMatcher for fuzzy matches when an activity code was missing from the reference table.
• Logged match statistics (exact vs. fuzzy) to compare output against the reference categories manually.
• Goal: prove correctness on a known subset before distributing work across CPU cores.

Multiprocessing layer

• Split transaction batches and processed them with multiprocessing.Pool, sized with cpu_count().
• Kept the reference dictionary read-only and shared across workers to avoid duplicate I/O.
• Preserved progress logging and timestamped exports so long runs could be monitored in production.
• Deliverable: parallel version of the pipeline for full transaction files.

Validated production script

• Ran the multiprocessing build (Verbose.py) against the full reference dataset and spot-checked category assignments at scale.
• Compared throughput and output consistency with the single-threaded baseline.
• Marked this branch as the production-ready script in the repository.
• Documented setup, inputs, and expected outputs in the README on the main branch (code default branch remains master).