Semantic Search Cwicr

Semantic Search in DDC CWICR Database

Business Case

Problem Statement

Construction cost estimation requires finding relevant work items from large databases. Traditional keyword search fails when:

• Users describe work in natural language
• Terminology varies across regions and languages
• Similar work items have different naming conventions

Solution

DDC CWICR provides pre-computed embeddings (BAAI/bge-m3, 1024 dimensions) enabling multilingual semantic search across 8 national bases (78,228 positions) plus the 30-market global base in 26 languages, with 48 PPP-repriced market catalogs per national base.

Business Value

• 90% faster work item lookup compared to manual search
• Multi-language: Arabic, Bulgarian, Chinese, Croatian, Czech, Danish, Dutch, English, Finnish, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mongolian, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Thai, Turkish, Vietnamese
• Higher accuracy by finding semantically similar items, not just keyword matches

Data landscape (2026)

Each base ships the 95-column CWICR master schema (rate_code, rate_original_name, rate_final_name, rate_unit, total_cost_per_position, classification hierarchy collection/department/section/subsection/category, resource_* component lines with is_material/is_machine/is_labor flags) plus 26 language editions and 48 markets/*.csv catalogs.

Latest data release: v0.4.0 (see releases).

Technical Implementation

Prerequisites

bashCopy code
pip install qdrant-client pandas sentence-transformers

Collections (2026)

The vector store uses BAAI/bge-m3 (1024-dim dense + sparse + colbert in one forward pass, MIT license, 100+ languages). Production collections are named cwicr_{LANG}_v3 (e.g. cwicr_tr_v3, cwicr_zh_v3). An ONNX-int8 variant (gpahal/bge-m3-onnx-int8, ~700 MB) is used on VPS-sized hosts.

Python Implementation

pythonCopy code
import pandas as pd
from qdrant_client import QdrantClient
from sentence_transformers import SentenceTransformer

class CWICRSemanticSearch:
    def __init__(self, host="localhost", port=6333, lang="en"):
        self.client = QdrantClient(host=host, port=port)
        self.collection = f"cwicr_{lang}_v3"
        self.model = SentenceTransformer("BAAI/bge-m3")

    def search_work_items(self, query, limit=10):
        vec = self.model.encode(query).tolist()
        hits = self.client.search(
            collection_name=self.collection,
            query_vector=vec,
            limit=limit,
        )
        return pd.DataFrame([{**h.payload, "score": h.score} for h in hits])

    def search_by_category(self, query, category, limit=10):
        vec = self.model.encode(query).tolist()
        hits = self.client.search(
            collection_name=self.collection,
            query_vector=vec,
            query_filter={"must": [{"key": "category", "match": {"value": category}}]},
            limit=limit,
        )
        return pd.DataFrame([{**h.payload, "score": h.score} for h in hits])

Inside OpenConstructionERP

The platform's costs module already exposes semantic matching:

• POST /api/v1/costs/suggest-for-element — rank cost items for a BIM element body.
• /qdrant-search — multilingual candidate retrieval for a query.
• The SQL fallback (GET /api/v1/costs/?q=...) works without Qdrant.

Database Schema (95-column master)

Key fields the payload carries:

Usage Examples

Basic Search

pythonCopy code
search = CWICRSemanticSearch(lang="tr")

# Natural language query
results = search.search_work_items("tuğla duvar örülmesi")
print(results[["rate_code", "rate_original_name", "total_cost_per_position", "score"]])

Cost Estimation

pythonCopy code
# Find work items for foundation work
foundation = search.search_work_items("reinforced concrete foundation", limit=20)

# Estimate with quantities (BIM takeoff)
quantities = {"15.115.1008": 150.0}  # m³
total = sum(quantities[c] * row["total_cost_per_position"]
            for _, row in foundation.iterrows() if row["rate_code"] in quantities)
print(f"Estimated: {total:,.2f} TRY")

Best Practices

1. Use specific queries - "reinforced concrete slab 200mm" beats "concrete"
2. Filter by category - Narrow results to relevant work types
3. Check similarity scores - Low scores need manual verification
4. Combine with QTO - Use BIM quantities for automated estimation
5. Mind the coefficient bases - Vietnam and Indonesia have no prices (rate 0); price them via a market resource sheet
6. Trust the source column - rate_original_name holds the source wording; translations live in rate_final_name

Resources

• GitHub: OpenConstructionEstimate-DDC-CWICR
• Releases: v0.4.0
• Platform: OpenConstructionERP
• Qdrant Docs: https://qdrant.tech/documentation/