feat: LinkedIn profile search engine (Django + React + Elasticsearch + ELK)

Full-stack implementation of the Cyberyan full-stack trial task.

Backend (Django 5 / DRF)
- Normalised PostgreSQL model (Profile, Company, School, Skill, Experience,
  Education) as the source of truth, with GIN + trigram indexes.
- Self-healing CSV reader: the supplied export carries an injected watermark
  column, missing columns, rows split by broken quoting and a duplicated
  header. A signature-scored Needleman-Wunsch alignment reconstructs 284 clean
  records out of 336 physical rows; only 10 rows parse correctly without it.
- Pluggable search layer (Strategy): Elasticsearch by default, PostgreSQL
  full-text search as a fallback selected by a single environment variable.
- Keyword search with per-field boosts, fuzziness and phrase boosting, six
  faceted filters (OR within a filter, AND across filters), an experience
  range, four sort orders, highlighting and facet aggregations.
- JWT auth with rotation + blacklist, three roles, PII masking, scoped rate
  limits, a single error envelope, OpenAPI schema and health probes.

Frontend (React 18 / TypeScript / Vite)
- Debounced search with type-ahead, faceted filter panel, result cards,
  profile drawer, URL-synced search state, automatic token refresh.
- Highlight snippets rendered without dangerouslySetInnerHTML.

Infrastructure
- Hardened multi-stage images (non-root, read-only rootfs, healthchecks).
- docker-compose for the full stack plus dev and production overlays.
- ELK: structured JSON logs shipped asynchronously to Logstash, routed per
  event type, with a composable index template and a Kibana setup script.

Quality
- 61 backend tests (85% coverage), ruff/black/bandit/mypy, ESLint + tsc,
  GitHub Actions running lint, tests and both image builds.
- Persian documentation: README plus architecture, search, API, data,
  security and operations guides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnFo6fNaHCTAkJbfcdkqtG
This commit is contained in:
2026-09-03 08:13:10 +00:00
commit a464e550c9
133 changed files with 17019 additions and 0 deletions
View File
+7
View File
@@ -0,0 +1,7 @@
import os
from django.core.asgi import get_asgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.prod")
application = get_asgi_application()
View File
+345
View File
@@ -0,0 +1,345 @@
"""
Base settings shared by every environment.
Environment specific modules (``dev``, ``prod``, ``test``) import everything
from here and override only what differs. Nothing secret is ever hard coded:
every credential is read from the environment through ``django-environ``.
"""
from datetime import timedelta
from pathlib import Path
import environ
BASE_DIR = Path(__file__).resolve().parent.parent.parent
env = environ.Env(
DJANGO_DEBUG=(bool, False),
DJANGO_ALLOWED_HOSTS=(list, ["localhost", "127.0.0.1"]),
CORS_ALLOWED_ORIGINS=(list, ["http://localhost:5173", "http://localhost:8080"]),
CSRF_TRUSTED_ORIGINS=(list, ["http://localhost:5173", "http://localhost:8080"]),
SEARCH_BACKEND=(str, "elasticsearch"),
ELASTICSEARCH_HOSTS=(list, ["http://elasticsearch:9200"]),
ELASTICSEARCH_USERNAME=(str, ""),
ELASTICSEARCH_PASSWORD=(str, ""),
ELASTICSEARCH_INDEX=(str, "linkedin-profiles"),
ELASTICSEARCH_TIMEOUT=(int, 10),
LOGSTASH_ENABLED=(bool, False),
LOGSTASH_HOST=(str, "logstash"),
LOGSTASH_PORT=(int, 5959),
LOG_LEVEL=(str, "INFO"),
THROTTLE_ANON=(str, "30/min"),
THROTTLE_USER=(str, "300/min"),
THROTTLE_SEARCH=(str, "120/min"),
THROTTLE_LOGIN=(str, "10/min"),
SERVICE_NAME=(str, "linkedin-search-api"),
ENVIRONMENT=(str, "development"),
)
# Read the local .env file when present (docker-compose injects real env vars).
env_file = BASE_DIR.parent / ".env"
if env_file.exists():
env.read_env(str(env_file))
# ---------------------------------------------------------------------------
# Core
# ---------------------------------------------------------------------------
SECRET_KEY = env("DJANGO_SECRET_KEY")
DEBUG = env("DJANGO_DEBUG")
ALLOWED_HOSTS = env("DJANGO_ALLOWED_HOSTS")
ENVIRONMENT = env("ENVIRONMENT")
SERVICE_NAME = env("SERVICE_NAME")
INSTALLED_APPS = [
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
"django.contrib.postgres",
# Third party
"rest_framework",
"rest_framework_simplejwt.token_blacklist",
"django_filters",
"corsheaders",
"drf_spectacular",
# Local
"apps.core",
"apps.accounts",
"apps.profiles",
"apps.search",
]
MIDDLEWARE = [
"apps.core.middleware.RequestIDMiddleware",
"django.middleware.security.SecurityMiddleware",
"whitenoise.middleware.WhiteNoiseMiddleware",
"corsheaders.middleware.CorsMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"apps.core.middleware.AccessLogMiddleware",
]
ROOT_URLCONF = "config.urls"
WSGI_APPLICATION = "config.wsgi.application"
ASGI_APPLICATION = "config.asgi.application"
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"DIRS": [BASE_DIR / "templates"],
"APP_DIRS": True,
"OPTIONS": {
"context_processors": [
"django.template.context_processors.debug",
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
],
},
},
]
# ---------------------------------------------------------------------------
# Database
# ---------------------------------------------------------------------------
DATABASES = {
"default": env.db("DATABASE_URL"),
}
DATABASES["default"]["CONN_MAX_AGE"] = env.int("DATABASE_CONN_MAX_AGE", default=60)
DATABASES["default"]["ATOMIC_REQUESTS"] = False
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
# ---------------------------------------------------------------------------
# Cache (also backs DRF throttling counters)
# ---------------------------------------------------------------------------
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": env("REDIS_URL", default="redis://redis:6379/1"),
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
"IGNORE_EXCEPTIONS": True, # a cache outage must not take the API down
},
"KEY_PREFIX": "lks",
}
}
DJANGO_REDIS_IGNORE_EXCEPTIONS = True
# ---------------------------------------------------------------------------
# Authentication
# ---------------------------------------------------------------------------
AUTH_USER_MODEL = "accounts.User"
AUTH_PASSWORD_VALIDATORS = [
{"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator"},
{
"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
"OPTIONS": {"min_length": 10},
},
{"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator"},
{"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator"},
]
# Argon2 first: modern, memory hard, recommended by OWASP.
PASSWORD_HASHERS = [
"django.contrib.auth.hashers.Argon2PasswordHasher",
"django.contrib.auth.hashers.PBKDF2PasswordHasher",
"django.contrib.auth.hashers.BCryptSHA256PasswordHasher",
]
SIMPLE_JWT = {
"ACCESS_TOKEN_LIFETIME": timedelta(minutes=env.int("JWT_ACCESS_MINUTES", default=15)),
"REFRESH_TOKEN_LIFETIME": timedelta(days=env.int("JWT_REFRESH_DAYS", default=7)),
"ROTATE_REFRESH_TOKENS": True,
"BLACKLIST_AFTER_ROTATION": True,
"UPDATE_LAST_LOGIN": True,
"ALGORITHM": "HS256",
"SIGNING_KEY": env("JWT_SIGNING_KEY", default=SECRET_KEY),
"AUTH_HEADER_TYPES": ("Bearer",),
"USER_ID_FIELD": "id",
"USER_ID_CLAIM": "user_id",
"TOKEN_OBTAIN_SERIALIZER": "apps.accounts.serializers.TokenObtainPairSerializer",
}
# ---------------------------------------------------------------------------
# Django REST Framework
# ---------------------------------------------------------------------------
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
"rest_framework_simplejwt.authentication.JWTAuthentication",
),
# Deny by default; endpoints opt in to public access explicitly.
"DEFAULT_PERMISSION_CLASSES": ("rest_framework.permissions.IsAuthenticated",),
"DEFAULT_PAGINATION_CLASS": "apps.core.pagination.PageNumberPagination",
"PAGE_SIZE": 20,
"DEFAULT_FILTER_BACKENDS": ("django_filters.rest_framework.DjangoFilterBackend",),
"DEFAULT_SCHEMA_CLASS": "drf_spectacular.openapi.AutoSchema",
"DEFAULT_THROTTLE_CLASSES": (
"rest_framework.throttling.AnonRateThrottle",
"rest_framework.throttling.UserRateThrottle",
),
"DEFAULT_THROTTLE_RATES": {
"anon": env("THROTTLE_ANON"),
"user": env("THROTTLE_USER"),
"search": env("THROTTLE_SEARCH"),
"login": env("THROTTLE_LOGIN"),
},
"EXCEPTION_HANDLER": "apps.core.exceptions.api_exception_handler",
"DEFAULT_RENDERER_CLASSES": ("rest_framework.renderers.JSONRenderer",),
# URL namespace = API version, so /api/v1/... is resolved automatically and
# a future /api/v2/ is a routing change rather than a rewrite.
"DEFAULT_VERSIONING_CLASS": "rest_framework.versioning.NamespaceVersioning",
"DEFAULT_VERSION": "v1",
"ALLOWED_VERSIONS": ["v1"],
"UNAUTHENTICATED_USER": "django.contrib.auth.models.AnonymousUser",
}
SPECTACULAR_SETTINGS = {
"TITLE": "LinkedIn Profile Search API",
"DESCRIPTION": (
"Keyword search and faceted filtering over a LinkedIn profile dataset. "
"Backed by Elasticsearch with a PostgreSQL full-text fallback."
),
"VERSION": "1.0.0",
"SERVE_INCLUDE_SCHEMA": False,
"COMPONENT_SPLIT_REQUEST": True,
"SCHEMA_PATH_PREFIX": "/api/v1",
"SERVE_PERMISSIONS": ["rest_framework.permissions.AllowAny"],
}
# ---------------------------------------------------------------------------
# CORS / CSRF
# ---------------------------------------------------------------------------
CORS_ALLOWED_ORIGINS = env("CORS_ALLOWED_ORIGINS")
CORS_ALLOW_CREDENTIALS = False
CORS_ALLOW_HEADERS = (
"accept",
"authorization",
"content-type",
"origin",
"user-agent",
"x-request-id",
)
CORS_EXPOSE_HEADERS = ("x-request-id",)
CSRF_TRUSTED_ORIGINS = env("CSRF_TRUSTED_ORIGINS")
# ---------------------------------------------------------------------------
# Baseline security headers (tightened further in prod.py)
# ---------------------------------------------------------------------------
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
SECURE_CROSS_ORIGIN_OPENER_POLICY = "same-origin"
X_FRAME_OPTIONS = "DENY"
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = "Lax"
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_SAMESITE = "Lax"
DATA_UPLOAD_MAX_MEMORY_SIZE = 5 * 1024 * 1024 # 5 MB
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000
# ---------------------------------------------------------------------------
# Internationalisation
# ---------------------------------------------------------------------------
LANGUAGE_CODE = "en-us"
TIME_ZONE = "UTC"
USE_I18N = True
USE_TZ = True
# ---------------------------------------------------------------------------
# Static files
# ---------------------------------------------------------------------------
STATIC_URL = "/static/"
STATIC_ROOT = BASE_DIR / "staticfiles"
STORAGES = {
"default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
"staticfiles": {"BACKEND": "whitenoise.storage.CompressedManifestStaticFilesStorage"},
}
# ---------------------------------------------------------------------------
# Search
# ---------------------------------------------------------------------------
SEARCH = {
# "elasticsearch" (default) or "postgres" - see apps/search/backends/
"BACKEND": env("SEARCH_BACKEND"),
"ELASTICSEARCH": {
"HOSTS": env("ELASTICSEARCH_HOSTS"),
"USERNAME": env("ELASTICSEARCH_USERNAME"),
"PASSWORD": env("ELASTICSEARCH_PASSWORD"),
"INDEX": env("ELASTICSEARCH_INDEX"),
"TIMEOUT": env("ELASTICSEARCH_TIMEOUT"),
"MAX_RETRIES": 2,
},
"MAX_PAGE_SIZE": 100,
"MAX_RESULT_WINDOW": 10_000,
"FACET_SIZE": 25,
}
# Path of the dataset used by the ``import_profiles`` management command.
DATASET_PATH = env("DATASET_PATH", default=str(BASE_DIR.parent / "data" / "linkedin_profiles.csv"))
# ---------------------------------------------------------------------------
# Logging - JSON on stdout, optionally mirrored to Logstash for the ELK stack
# ---------------------------------------------------------------------------
LOG_LEVEL = env("LOG_LEVEL")
LOGSTASH_ENABLED = env("LOGSTASH_ENABLED")
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"filters": {
"request_id": {"()": "apps.core.logging.RequestIDFilter"},
},
"formatters": {
"json": {
"()": "apps.core.logging.JSONFormatter",
"service": SERVICE_NAME,
"environment": ENVIRONMENT,
},
"console": {
"format": "[{asctime}] {levelname} {name} {message}",
"style": "{",
},
},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"formatter": "json",
"filters": ["request_id"],
},
},
"root": {"handlers": ["stdout"], "level": LOG_LEVEL},
"loggers": {
"django": {"handlers": ["stdout"], "level": LOG_LEVEL, "propagate": False},
"django.db.backends": {"handlers": ["stdout"], "level": "WARNING", "propagate": False},
# Elasticsearch client is chatty at INFO; keep the noise down.
"elastic_transport": {"handlers": ["stdout"], "level": "WARNING", "propagate": False},
"apps": {"handlers": ["stdout"], "level": LOG_LEVEL, "propagate": False},
# Business level audit trail: every executed search lands here.
"audit.search": {"handlers": ["stdout"], "level": "INFO", "propagate": False},
"audit.auth": {"handlers": ["stdout"], "level": "INFO", "propagate": False},
},
}
if LOGSTASH_ENABLED:
LOGGING["handlers"]["logstash"] = {
"class": "logstash_async.handler.AsynchronousLogstashHandler",
"formatter": "json",
"filters": ["request_id"],
"transport": "logstash_async.transport.TcpTransport",
"host": env("LOGSTASH_HOST"),
"port": env("LOGSTASH_PORT"),
# In-memory buffer: no extra volume, and a Logstash outage never blocks
# a request thread.
"database_path": None,
}
for logger in ("root", *LOGGING["loggers"]):
target = LOGGING if logger == "root" else LOGGING["loggers"]
cfg = target["root"] if logger == "root" else target[logger]
cfg["handlers"] = [*cfg["handlers"], "logstash"]
+22
View File
@@ -0,0 +1,22 @@
"""Local development settings: verbose, permissive, never for production."""
from .base import * # noqa: F403
from .base import INSTALLED_APPS, REST_FRAMEWORK, env
DEBUG = True
INSTALLED_APPS += ["django_extensions"]
# Browsable API is handy while developing against the endpoints by hand.
REST_FRAMEWORK["DEFAULT_RENDERER_CLASSES"] = (
"rest_framework.renderers.JSONRenderer",
"rest_framework.renderers.BrowsableAPIRenderer",
)
ALLOWED_HOSTS = ["*"]
CORS_ALLOW_ALL_ORIGINS = env.bool("CORS_ALLOW_ALL_ORIGINS", default=True)
# Never enforce HTTPS locally.
SECURE_SSL_REDIRECT = False
SESSION_COOKIE_SECURE = False
CSRF_COOKIE_SECURE = False
+42
View File
@@ -0,0 +1,42 @@
"""
Production settings.
Everything that can be hardened is hardened here; the module intentionally
fails loudly on start-up when a required secret is missing rather than falling
back to an insecure default.
"""
from .base import * # noqa: F403
from .base import DEBUG, SECRET_KEY, env
# ---------------------------------------------------------------------------
# Fail fast on unsafe configuration
# ---------------------------------------------------------------------------
if DEBUG: # pragma: no cover - configuration guard
raise RuntimeError("DJANGO_DEBUG must be false when using production settings.")
if len(SECRET_KEY) < 50: # pragma: no cover - configuration guard
raise RuntimeError("DJANGO_SECRET_KEY must be at least 50 characters in production.")
# ---------------------------------------------------------------------------
# Transport security
# ---------------------------------------------------------------------------
SECURE_SSL_REDIRECT = env.bool("SECURE_SSL_REDIRECT", default=True)
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
SECURE_HSTS_SECONDS = 31_536_000 # one year
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SAMESITE = "Strict"
CSRF_COOKIE_SAMESITE = "Strict"
# ---------------------------------------------------------------------------
# Misc hardening
# ---------------------------------------------------------------------------
# Admin lives behind an unguessable prefix so credential stuffing bots miss it.
ADMIN_URL = env("DJANGO_ADMIN_URL", default="admin/")
# Deny by default; CORS_ALLOWED_ORIGINS must be set explicitly.
CORS_ALLOW_ALL_ORIGINS = False
+26
View File
@@ -0,0 +1,26 @@
"""Test settings: fast, hermetic, no external services required."""
import os
os.environ.setdefault("DJANGO_SECRET_KEY", "test-secret-key-not-used-anywhere-else")
os.environ.setdefault("DATABASE_URL", "postgres://postgres:postgres@localhost:5432/test_db")
from .base import * # noqa: E402, F403
from .base import LOGGING, SEARCH # noqa: E402
DEBUG = False
SEARCH["BACKEND"] = "postgres" # tests must not need a live Elasticsearch
# Cheap password hashing keeps the suite quick.
PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"] # noqa: S105
CACHES = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}}
# Throttling off by default; individual tests enable it explicitly.
REST_FRAMEWORK_THROTTLE_DISABLED = True
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
LOGGING["root"]["level"] = "CRITICAL"
for _logger in LOGGING["loggers"].values():
_logger["level"] = "CRITICAL"
+40
View File
@@ -0,0 +1,40 @@
"""
Root URL configuration.
Everything the SPA talks to lives under ``/api/v1/`` and is namespaced, which
is what DRF's ``NamespaceVersioning`` uses to resolve ``request.version``.
Adding a ``/api/v2/`` later is therefore a routing change, not a rewrite.
"""
from django.conf import settings
from django.contrib import admin
from django.urls import include, path
from drf_spectacular.views import (
SpectacularAPIView,
SpectacularRedocView,
SpectacularSwaggerView,
)
api_v1 = [
path("auth/", include("apps.accounts.urls", namespace="accounts")),
path("profiles/", include("apps.profiles.urls", namespace="profiles")),
path("search/", include("apps.search.urls", namespace="search")),
path("health/", include("apps.core.urls", namespace="core")),
]
urlpatterns = [
path(getattr(settings, "ADMIN_URL", "admin/"), admin.site.urls),
path("api/v1/", include((api_v1, "api"), namespace="v1")),
# OpenAPI schema and the two documentation renderers.
path("api/schema/", SpectacularAPIView.as_view(), name="schema"),
path(
"api/docs/",
SpectacularSwaggerView.as_view(url_name="schema"),
name="swagger-ui",
),
path("api/redoc/", SpectacularRedocView.as_view(url_name="schema"), name="redoc"),
]
admin.site.site_header = "LinkedIn Profile Search"
admin.site.site_title = "LinkedIn Profile Search"
admin.site.index_title = "Operations"
+7
View File
@@ -0,0 +1,7 @@
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.prod")
application = get_wsgi_application()