Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OT Document Sync & RAG Portal

An internal admin portal for managing Operational Technology (OT) documentation updates across multiple storage systems, with automated RAG (Retrieval-Augmented Generation) pipeline integration.

Built to operate within a FedRAMP-approved Azure OpenAI environment.


Overview

This system centralizes the uploading of OT manuals (policies, procedures, EDIGs) through a secure Admin Portal. When a new document version is uploaded, the system automatically:

  1. Distributes files to all storage locations (EWFP SQL, SharePoint, OneDrive)
  2. Invalidates old embeddings in the RAG vector database
  3. Chunks and embeds the new document into Azure AI Search
  4. Notifies active users of the updated content
  5. Logs all changes to a tamper-evident audit log

Architecture

React Frontend (Azure Static Web Apps)
        │
        │  MSAL JWT Token (Admin only)
        ▼
FastAPI Backend (Azure App Service - Linux)
        │
        ├──▶ EWFP SQL Database     (BLOB storage, insert-only)
        ├──▶ SharePoint (Graph API) (file distribution)
        ├──▶ OneDrive (Graph API)   (file distribution)
        └──▶ Azure AI Search        (RAG vector store)
                │
                └──▶ Azure OpenAI   (embedding generation)

All backend-to-data traffic flows through Azure Private Endpoints — no data crosses the public internet.


Tech Stack

Layer Technology
Frontend React, Tailwind CSS, @azure/msal-react
Backend Python 3.11, FastAPI
Auth Microsoft Entra ID (MSAL / RBAC)
RAG Vector Store Azure AI Search
AI Embeddings Azure OpenAI (text-embedding-ada-002)
Document Intelligence Azure AI Document Intelligence (prebuilt-layout)
SQL Storage Microsoft SQL Server (EWFP) via pyodbc
Cloud Storage Microsoft Graph API (SharePoint + OneDrive)
Infrastructure Azure Bicep (IaC)
CI/CD GitHub Actions

Project Structure

ot-docsync/
├── backend/
│   ├── main.py               # FastAPI app: routes and orchestration
│   ├── auth.py               # MSAL JWT verification middleware
│   ├── rag_pipeline.py       # Document chunking, embedding, invalidation
│   ├── storage.py            # SQL BLOB storage (EWFP, insert-only)
│   ├── graph_api.py          # SharePoint & OneDrive upload via MS Graph
│   ├── audit_log.py          # Document_Audit_Log DB operations
│   ├── requirements.txt      # Python dependencies
│   └── .env.example          # Environment variable template
│
├── frontend/
│   ├── src/
│   │   ├── components/
│   │   │   ├── AdminDocumentUpload.jsx   # Admin upload form (MSAL-secured)
│   │   │   └── DocumentUpdateNotifier.jsx # Polling-based update banner
│   │   ├── hooks/
│   │   │   └── useDocumentPolling.js     # Polling logic (1-hour interval)
│   │   ├── authConfig.js                 # MSAL configuration
│   │   └── App.jsx                       # Root component with MsalProvider
│   ├── package.json
│   └── .env.example
│
├── infra/
│   └── main.bicep            # Azure IaC: VNet, App Service, Static Web App, RBAC
│
├── docs/
│   ├── SETUP.md              # Step-by-step environment setup guide
│   └── SQL_SCHEMA.md         # EWFP audit log table schema
│
├── .github/
│   └── workflows/
│       ├── deploy-backend.yml    # CI/CD: Deploy FastAPI to Azure App Service
│       └── deploy-frontend.yml   # CI/CD: Deploy React to Azure Static Web Apps
│
└── README.md

Quick Start

Prerequisites

  • Azure subscription with FedRAMP-compliant configuration
  • Azure CLI installed and authenticated to your tenant
  • Python 3.11+, Node.js 18+
  • Existing EWFP SQL Server, Azure OpenAI, and Azure AI Search resources

1. Provision Infrastructure

az deployment group create \
  --resource-group Your-FedRAMP-RG \
  --template-file infra/main.bicep \
  --parameters existingOpenAIName="your-openai-instance" \
               existingSqlServerName="your-ewfp-db"

2. Configure Environment Variables

# Backend
cp backend/.env.example backend/.env
# Edit .env with your Azure resource values

# Frontend
cp frontend/.env.example frontend/.env
# Edit .env with your Azure App Registration client IDs

3. Run Locally

# Backend
cd backend
pip install -r requirements.txt
uvicorn main:app --reload

# Frontend (new terminal)
cd frontend
npm install
npm start

Security Model

Endpoint Auth Required Method
POST /api/documents/upload ✅ Yes — Document.Admin role MSAL JWT via Entra ID
GET /api/updates/recent ❌ No (read-only metadata only) Open polling
  • All secrets pulled from Azure Key Vault or App Service Environment Variables — no hardcoded credentials anywhere
  • FTP deployment disabled; all deploys via CI/CD only
  • All backend traffic routes through VNet to Private Endpoints
  • TLS 1.2 minimum enforced on all App Service endpoints

Key Design Decisions

Insert-Only Logic (EWFP)

The EWFP SQL database is treated as an append-only archive. New file versions are inserted alongside old ones — no deletes, no overwrites. Version numbers are embedded in filenames (e.g., ProcessNameR14.pdf replaces ProcessNameR13.pdf without removing it).

RAG Invalidation

When a new version is uploaded, old embeddings are explicitly deleted from Azure AI Search using a source_filename metadata filter. This prevents the LLM from hallucinating by mixing context from multiple document versions.

Polling vs. WebSockets

Active-user notifications use a 1-hour polling interval (/api/updates/recent) rather than WebSockets. This keeps server load near zero and eliminates the complexity of persistent socket management in a FedRAMP environment.


Contributing

See docs/SETUP.md for full environment setup and development guidelines.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages