INCIDB Technical Documentation

1. Production Scale Metrics

INCIDB unifies high-precision chemical composition data across four core tables:

• 19,645 Commercial Formulations: Covering prestige skincare lines, K-Beauty functional cosmetics, and OTC clinical topicals.
• 44,816 Canonical INCI Compounds: Standardized according to International Nomenclature Cosmetic Ingredient conventions.
• 5,994 Global Brands: Enriched with ethical certifications (cruelty-free and vegan indicators).
• 323,730 Relational Junction Mappings: Preserving exact package label concentration order (position_index).

2. Public-Source Regulatory & Open-Data Architecture

Every ingredient profile is normalized from public regulatory and open-data sources:

• EU Commission CosIng Registry: Official functional categories, restricted thresholds, and CAS registry numbers (69-72-7 Salicylic Acid).
• US FDA MoCRA Allergen Registry: Flags mandatory contact dermatitis fragrance allergens (Linalool, Limonene, Geraniol).
• NLM DailyMed Clinical OTC: Formulations harvested from US National Library of Medicine topical dermatological labels.
• K-Beauty MFDS Standards: Functional whitening, anti-wrinkle, and barrier repair standards from South Korean regulations.
• Open Beauty Facts: Community product corpus (barcodes, brands, ingredient lists) under the Open Database License (ODbL).

3. Audit & Verification Results

All database releases pass rigorous 1-to-1 physical parity and string sanitization audits:

• Zero Linebreak Corruption: Stripped unescaped carriage returns (\r) and linebreaks (\n) inside string fields to guarantee exact line count correspondence (wc -l exactly equals record counts).
• 82% Parquet Compression Ratio: Compressed via Apache PyArrow snappy/zstd algorithms for rapid cloud ingestion.
• Full Foreign Key Integrity: Audited 100% referential integrity across all 323,730 junction records.

4. Quick Start — Python PyArrow & Pandas

Load high-performance Parquet datasets directly into memory for filtering allergens and functional categories:

import pyarrow.parquet as pq
import pandas as pd

# Load formulations and canonical ingredients
products = pq.read_table('products.parquet').to_pandas()
ingredients = pq.read_table('ingredients.parquet').to_pandas()

# Filter FDA MoCRA contact allergens
allergens = ingredients[ingredients['is_common_allergen'] == 1]
print(f"Flagged {len(allergens)} MoCRA contact allergens across {len(products):,} formulations.")

5. Quick Start — DuckDB In-Memory SQL

Execute analytical joins directly over flat Parquet files without database servers:

import duckdb

query = """
SELECT 
    b.name AS brand,
    p.name AS product,
    COUNT(pi.ingredient_id) AS total_ingredients,
    SUM(i.is_common_allergen) AS mocra_allergens
FROM 'products.parquet' p
JOIN 'brands.parquet' b ON p.brand_id = b.brand_id
JOIN 'product_ingredients.parquet' pi ON p.product_id = pi.product_id
JOIN 'ingredients.parquet' i ON pi.ingredient_id = i.ingredient_id
GROUP BY b.name, p.name
HAVING SUM(i.is_common_allergen) > 0
ORDER BY mocra_allergens DESC
LIMIT 5;
"""

print(duckdb.query(query).to_df())