feat: LinkedIn profile search engine (Django + React + Elasticsearch + ELK)

Full-stack implementation of the Cyberyan full-stack trial task. Backend (Django 5 / DRF) - Normalised PostgreSQL model (Profile, Company, School, Skill, Experience, Education) as the source of truth, with GIN + trigram indexes. - Self-healing CSV reader: the supplied export carries an injected watermark column, missing columns, rows split by broken quoting and a duplicated header. A signature-scored Needleman-Wunsch alignment reconstructs 284 clean records out of 336 physical rows; only 10 rows parse correctly without it. - Pluggable search layer (Strategy): Elasticsearch by default, PostgreSQL full-text search as a fallback selected by a single environment variable. - Keyword search with per-field boosts, fuzziness and phrase boosting, six faceted filters (OR within a filter, AND across filters), an experience range, four sort orders, highlighting and facet aggregations. - JWT auth with rotation + blacklist, three roles, PII masking, scoped rate limits, a single error envelope, OpenAPI schema and health probes. Frontend (React 18 / TypeScript / Vite) - Debounced search with type-ahead, faceted filter panel, result cards, profile drawer, URL-synced search state, automatic token refresh. - Highlight snippets rendered without dangerouslySetInnerHTML. Infrastructure - Hardened multi-stage images (non-root, read-only rootfs, healthchecks). - docker-compose for the full stack plus dev and production overlays. - ELK: structured JSON logs shipped asynchronously to Logstash, routed per event type, with a composable index template and a Kibana setup script. Quality - 61 backend tests (85% coverage), ruff/black/bandit/mypy, ESLint + tsc, GitHub Actions running lint, tests and both image builds. - Persian documentation: README plus architecture, search, API, data, security and operations guides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnFo6fNaHCTAkJbfcdkqtG
2026-09-03 08:13:10 +00:00
commit a464e550c9
133 changed files with 17019 additions and 0 deletions
@@ -0,0 +1,87 @@
+"""``python manage.py import_profiles`` - load the CSV export into PostgreSQL."""
+
+from __future__ import annotations
+
+import json
+import time
+from typing import Any
+
+from django.conf import settings
+from django.core.management.base import BaseCommand, CommandError
+from django.db import transaction
+
+from apps.profiles.importers import ProfileImporter
+from apps.profiles.models import (
+    Company,
+    Education,
+    Experience,
+    Profile,
+    ProfileSkill,
+    School,
+    Skill,
+)
+
+
+class Command(BaseCommand):
+    help = "Import (or refresh) LinkedIn profiles from the raw CSV export."
+
+    def add_arguments(self, parser) -> None:  # noqa: ANN001
+        parser.add_argument(
+            "--path",
+            default=settings.DATASET_PATH,
+            help="Path to the CSV export (defaults to DATASET_PATH).",
+        )
+        parser.add_argument(
+            "--limit", type=int, default=None, help="Import at most N profiles (smoke tests)."
+        )
+        parser.add_argument(
+            "--truncate",
+            action="store_true",
+            help="Delete existing profile data before importing.",
+        )
+        parser.add_argument(
+            "--reindex",
+            action="store_true",
+            help="Rebuild the Elasticsearch index once the import finishes.",
+        )
+
+    def handle(self, *args: Any, **options: Any) -> None:
+        path = options["path"]
+        started = time.perf_counter()
+
+        if options["truncate"]:
+            self.stdout.write("Truncating existing profile data ...")
+            with transaction.atomic():
+                ProfileSkill.objects.all().delete()
+                Experience.objects.all().delete()
+                Education.objects.all().delete()
+                Profile.objects.all().delete()
+                Skill.objects.all().delete()
+                Company.objects.all().delete()
+                School.objects.all().delete()
+
+        self.stdout.write(f"Reading dataset from {path} ...")
+        try:
+            stats = ProfileImporter(path, limit=options["limit"]).run()
+        except FileNotFoundError as exc:
+            raise CommandError(f"Dataset not found: {path}") from exc
+
+        elapsed = round(time.perf_counter() - started, 2)
+        summary = {**stats.as_dict(), "elapsed_seconds": elapsed}
+        self.stdout.write(json.dumps(summary, indent=2))
+
+        if stats.report.issues:
+            self.stdout.write(
+                self.style.WARNING(
+                    f"{len(stats.report.issues)} record(s) failed: "
+                    f"{', '.join(stats.report.issues[:10])}"
+                )
+            )
+
+        if options["reindex"]:
+            self.stdout.write("Rebuilding the search index ...")
+            from django.core.management import call_command
+
+            call_command("search_index", "--rebuild")
+
+        self.stdout.write(self.style.SUCCESS(f"Import finished in {elapsed}s."))