mobiniqow/cyberyan_linkedin_search
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a464e550c925a01840dc784b68612e14d2257601
cyberyan_linkedin_search/backend/config/settings/base.py
T
mobiniqow a464e550c9 feat: LinkedIn profile search engine (Django + React + Elasticsearch + ELK) 
Full-stack implementation of the Cyberyan full-stack trial task.

Backend (Django 5 / DRF)
- Normalised PostgreSQL model (Profile, Company, School, Skill, Experience,
  Education) as the source of truth, with GIN + trigram indexes.
- Self-healing CSV reader: the supplied export carries an injected watermark
  column, missing columns, rows split by broken quoting and a duplicated
  header. A signature-scored Needleman-Wunsch alignment reconstructs 284 clean
  records out of 336 physical rows; only 10 rows parse correctly without it.
- Pluggable search layer (Strategy): Elasticsearch by default, PostgreSQL
  full-text search as a fallback selected by a single environment variable.
- Keyword search with per-field boosts, fuzziness and phrase boosting, six
  faceted filters (OR within a filter, AND across filters), an experience
  range, four sort orders, highlighting and facet aggregations.
- JWT auth with rotation + blacklist, three roles, PII masking, scoped rate
  limits, a single error envelope, OpenAPI schema and health probes.

Frontend (React 18 / TypeScript / Vite)
- Debounced search with type-ahead, faceted filter panel, result cards,
  profile drawer, URL-synced search state, automatic token refresh.
- Highlight snippets rendered without dangerouslySetInnerHTML.

Infrastructure
- Hardened multi-stage images (non-root, read-only rootfs, healthchecks).
- docker-compose for the full stack plus dev and production overlays.
- ELK: structured JSON logs shipped asynchronously to Logstash, routed per
  event type, with a composable index template and a Kibana setup script.

Quality
- 61 backend tests (85% coverage), ruff/black/bandit/mypy, ESLint + tsc,
  GitHub Actions running lint, tests and both image builds.
- Persian documentation: README plus architecture, search, API, data,
  security and operations guides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GnFo6fNaHCTAkJbfcdkqtG
2026-09-03 08:13:50 +00:00

346 lines
13 KiB
Python
Raw Blame History

"""
Base settings shared by every environment.
Environment specific modules (``dev``, ``prod``, ``test``) import everything
from here and override only what differs. Nothing secret is ever hard coded:
every credential is read from the environment through ``django-environ``.
"""
from datetime import timedelta
from pathlib import Path
import environ
BASE_DIR = Path(__file__).resolve().parent.parent.parent
env = environ.Env(
DJANGO_DEBUG=(bool, False),
DJANGO_ALLOWED_HOSTS=(list, ["localhost", "127.0.0.1"]),
CORS_ALLOWED_ORIGINS=(list, ["http://localhost:5173", "http://localhost:8080"]),
CSRF_TRUSTED_ORIGINS=(list, ["http://localhost:5173", "http://localhost:8080"]),
SEARCH_BACKEND=(str, "elasticsearch"),
ELASTICSEARCH_HOSTS=(list, ["http://elasticsearch:9200"]),
ELASTICSEARCH_USERNAME=(str, ""),
ELASTICSEARCH_PASSWORD=(str, ""),
ELASTICSEARCH_INDEX=(str, "linkedin-profiles"),
ELASTICSEARCH_TIMEOUT=(int, 10),
LOGSTASH_ENABLED=(bool, False),
LOGSTASH_HOST=(str, "logstash"),
LOGSTASH_PORT=(int, 5959),
LOG_LEVEL=(str, "INFO"),
THROTTLE_ANON=(str, "30/min"),
THROTTLE_USER=(str, "300/min"),
THROTTLE_SEARCH=(str, "120/min"),
THROTTLE_LOGIN=(str, "10/min"),
SERVICE_NAME=(str, "linkedin-search-api"),
ENVIRONMENT=(str, "development"),
)
# Read the local .env file when present (docker-compose injects real env vars).
env_file = BASE_DIR.parent / ".env"
if env_file.exists():
env.read_env(str(env_file))
# ---------------------------------------------------------------------------
# Core
# ---------------------------------------------------------------------------
SECRET_KEY = env("DJANGO_SECRET_KEY")
DEBUG = env("DJANGO_DEBUG")
ALLOWED_HOSTS = env("DJANGO_ALLOWED_HOSTS")
ENVIRONMENT = env("ENVIRONMENT")
SERVICE_NAME = env("SERVICE_NAME")
INSTALLED_APPS = [
"django.contrib.admin",
"django.contrib.auth",
"django.contrib.contenttypes",
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
"django.contrib.postgres",
# Third party
"rest_framework",
"rest_framework_simplejwt.token_blacklist",
"django_filters",
"corsheaders",
"drf_spectacular",
# Local
"apps.core",
"apps.accounts",
"apps.profiles",
"apps.search",
]
MIDDLEWARE = [
"apps.core.middleware.RequestIDMiddleware",
"django.middleware.security.SecurityMiddleware",
"whitenoise.middleware.WhiteNoiseMiddleware",
"corsheaders.middleware.CorsMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
"apps.core.middleware.AccessLogMiddleware",
]
ROOT_URLCONF = "config.urls"
WSGI_APPLICATION = "config.wsgi.application"
ASGI_APPLICATION = "config.asgi.application"
TEMPLATES = [
{
"BACKEND": "django.template.backends.django.DjangoTemplates",
"DIRS": [BASE_DIR / "templates"],
"APP_DIRS": True,
"OPTIONS": {
"context_processors": [
"django.template.context_processors.debug",
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
],
},
},
]
# ---------------------------------------------------------------------------
# Database
# ---------------------------------------------------------------------------
DATABASES = {
"default": env.db("DATABASE_URL"),
}
DATABASES["default"]["CONN_MAX_AGE"] = env.int("DATABASE_CONN_MAX_AGE", default=60)
DATABASES["default"]["ATOMIC_REQUESTS"] = False
DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
# ---------------------------------------------------------------------------
# Cache (also backs DRF throttling counters)
# ---------------------------------------------------------------------------
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": env("REDIS_URL", default="redis://redis:6379/1"),
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
"IGNORE_EXCEPTIONS": True, # a cache outage must not take the API down
},
"KEY_PREFIX": "lks",
}
}
DJANGO_REDIS_IGNORE_EXCEPTIONS = True
# ---------------------------------------------------------------------------
# Authentication
# ---------------------------------------------------------------------------
AUTH_USER_MODEL = "accounts.User"
AUTH_PASSWORD_VALIDATORS = [
{"NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator"},
{
"NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
"OPTIONS": {"min_length": 10},
},
{"NAME": "django.contrib.auth.password_validation.CommonPasswordValidator"},
{"NAME": "django.contrib.auth.password_validation.NumericPasswordValidator"},
]
# Argon2 first: modern, memory hard, recommended by OWASP.
PASSWORD_HASHERS = [
"django.contrib.auth.hashers.Argon2PasswordHasher",
"django.contrib.auth.hashers.PBKDF2PasswordHasher",
"django.contrib.auth.hashers.BCryptSHA256PasswordHasher",
]
SIMPLE_JWT = {
"ACCESS_TOKEN_LIFETIME": timedelta(minutes=env.int("JWT_ACCESS_MINUTES", default=15)),
"REFRESH_TOKEN_LIFETIME": timedelta(days=env.int("JWT_REFRESH_DAYS", default=7)),
"ROTATE_REFRESH_TOKENS": True,
"BLACKLIST_AFTER_ROTATION": True,
"UPDATE_LAST_LOGIN": True,
"ALGORITHM": "HS256",
"SIGNING_KEY": env("JWT_SIGNING_KEY", default=SECRET_KEY),
"AUTH_HEADER_TYPES": ("Bearer",),
"USER_ID_FIELD": "id",
"USER_ID_CLAIM": "user_id",
"TOKEN_OBTAIN_SERIALIZER": "apps.accounts.serializers.TokenObtainPairSerializer",
}
# ---------------------------------------------------------------------------
# Django REST Framework
# ---------------------------------------------------------------------------
REST_FRAMEWORK = {
"DEFAULT_AUTHENTICATION_CLASSES": (
"rest_framework_simplejwt.authentication.JWTAuthentication",
),
# Deny by default; endpoints opt in to public access explicitly.
"DEFAULT_PERMISSION_CLASSES": ("rest_framework.permissions.IsAuthenticated",),
"DEFAULT_PAGINATION_CLASS": "apps.core.pagination.PageNumberPagination",
"PAGE_SIZE": 20,
"DEFAULT_FILTER_BACKENDS": ("django_filters.rest_framework.DjangoFilterBackend",),
"DEFAULT_SCHEMA_CLASS": "drf_spectacular.openapi.AutoSchema",
"DEFAULT_THROTTLE_CLASSES": (
"rest_framework.throttling.AnonRateThrottle",
"rest_framework.throttling.UserRateThrottle",
),
"DEFAULT_THROTTLE_RATES": {
"anon": env("THROTTLE_ANON"),
"user": env("THROTTLE_USER"),
"search": env("THROTTLE_SEARCH"),
"login": env("THROTTLE_LOGIN"),
},
"EXCEPTION_HANDLER": "apps.core.exceptions.api_exception_handler",
"DEFAULT_RENDERER_CLASSES": ("rest_framework.renderers.JSONRenderer",),
# URL namespace = API version, so /api/v1/... is resolved automatically and
# a future /api/v2/ is a routing change rather than a rewrite.
"DEFAULT_VERSIONING_CLASS": "rest_framework.versioning.NamespaceVersioning",
"DEFAULT_VERSION": "v1",
"ALLOWED_VERSIONS": ["v1"],
"UNAUTHENTICATED_USER": "django.contrib.auth.models.AnonymousUser",
}
SPECTACULAR_SETTINGS = {
"TITLE": "LinkedIn Profile Search API",
"DESCRIPTION": (
"Keyword search and faceted filtering over a LinkedIn profile dataset. "
"Backed by Elasticsearch with a PostgreSQL full-text fallback."
),
"VERSION": "1.0.0",
"SERVE_INCLUDE_SCHEMA": False,
"COMPONENT_SPLIT_REQUEST": True,
"SCHEMA_PATH_PREFIX": "/api/v1",
"SERVE_PERMISSIONS": ["rest_framework.permissions.AllowAny"],
}
# ---------------------------------------------------------------------------
# CORS / CSRF
# ---------------------------------------------------------------------------
CORS_ALLOWED_ORIGINS = env("CORS_ALLOWED_ORIGINS")
CORS_ALLOW_CREDENTIALS = False
CORS_ALLOW_HEADERS = (
"accept",
"authorization",
"content-type",
"origin",
"user-agent",
"x-request-id",
)
CORS_EXPOSE_HEADERS = ("x-request-id",)
CSRF_TRUSTED_ORIGINS = env("CSRF_TRUSTED_ORIGINS")
# ---------------------------------------------------------------------------
# Baseline security headers (tightened further in prod.py)
# ---------------------------------------------------------------------------
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
SECURE_CROSS_ORIGIN_OPENER_POLICY = "same-origin"
X_FRAME_OPTIONS = "DENY"
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = "Lax"
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_SAMESITE = "Lax"
DATA_UPLOAD_MAX_MEMORY_SIZE = 5 * 1024 * 1024 # 5 MB
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000
# ---------------------------------------------------------------------------
# Internationalisation
# ---------------------------------------------------------------------------
LANGUAGE_CODE = "en-us"
TIME_ZONE = "UTC"
USE_I18N = True
USE_TZ = True
# ---------------------------------------------------------------------------
# Static files
# ---------------------------------------------------------------------------
STATIC_URL = "/static/"
STATIC_ROOT = BASE_DIR / "staticfiles"
STORAGES = {
"default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
"staticfiles": {"BACKEND": "whitenoise.storage.CompressedManifestStaticFilesStorage"},
}
# ---------------------------------------------------------------------------
# Search
# ---------------------------------------------------------------------------
SEARCH = {
# "elasticsearch" (default) or "postgres" - see apps/search/backends/
"BACKEND": env("SEARCH_BACKEND"),
"ELASTICSEARCH": {
"HOSTS": env("ELASTICSEARCH_HOSTS"),
"USERNAME": env("ELASTICSEARCH_USERNAME"),
"PASSWORD": env("ELASTICSEARCH_PASSWORD"),
"INDEX": env("ELASTICSEARCH_INDEX"),
"TIMEOUT": env("ELASTICSEARCH_TIMEOUT"),
"MAX_RETRIES": 2,
},
"MAX_PAGE_SIZE": 100,
"MAX_RESULT_WINDOW": 10_000,
"FACET_SIZE": 25,
}
# Path of the dataset used by the ``import_profiles`` management command.
DATASET_PATH = env("DATASET_PATH", default=str(BASE_DIR.parent / "data" / "linkedin_profiles.csv"))
# ---------------------------------------------------------------------------
# Logging - JSON on stdout, optionally mirrored to Logstash for the ELK stack
# ---------------------------------------------------------------------------
LOG_LEVEL = env("LOG_LEVEL")
LOGSTASH_ENABLED = env("LOGSTASH_ENABLED")
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"filters": {
"request_id": {"()": "apps.core.logging.RequestIDFilter"},
},
"formatters": {
"json": {
"()": "apps.core.logging.JSONFormatter",
"service": SERVICE_NAME,
"environment": ENVIRONMENT,
},
"console": {
"format": "[{asctime}] {levelname} {name} {message}",
"style": "{",
},
},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"formatter": "json",
"filters": ["request_id"],
},
},
"root": {"handlers": ["stdout"], "level": LOG_LEVEL},
"loggers": {
"django": {"handlers": ["stdout"], "level": LOG_LEVEL, "propagate": False},
"django.db.backends": {"handlers": ["stdout"], "level": "WARNING", "propagate": False},
# Elasticsearch client is chatty at INFO; keep the noise down.
"elastic_transport": {"handlers": ["stdout"], "level": "WARNING", "propagate": False},
"apps": {"handlers": ["stdout"], "level": LOG_LEVEL, "propagate": False},
# Business level audit trail: every executed search lands here.
"audit.search": {"handlers": ["stdout"], "level": "INFO", "propagate": False},
"audit.auth": {"handlers": ["stdout"], "level": "INFO", "propagate": False},
},
}
if LOGSTASH_ENABLED:
LOGGING["handlers"]["logstash"] = {
"class": "logstash_async.handler.AsynchronousLogstashHandler",
"formatter": "json",
"filters": ["request_id"],
"transport": "logstash_async.transport.TcpTransport",
"host": env("LOGSTASH_HOST"),
"port": env("LOGSTASH_PORT"),
# In-memory buffer: no extra volume, and a Logstash outage never blocks
# a request thread.
"database_path": None,
}
for logger in ("root", *LOGGING["loggers"]):
target = LOGGING if logger == "root" else LOGGING["loggers"]
cfg = target["root"] if logger == "root" else target[logger]
cfg["handlers"] = [*cfg["handlers"], "logstash"]
Reference in New Issue View Git Blame Copy Permalink