Queen's University — School of Computing Group 23 | AWS NetID Prefix: 25bbdf-g23

Horus-OSINT is a cloud-based conversational chatbot designed to act as an Open-Source Intelligence (OSINT) and Global Threat Analyst. It leverages a fine-tuned Meta-Llama-3-8B-Instruct LLM trained on 159,826 structured records derived from the Global Terrorism Database (GTD) and GDELT, deployed entirely on AWS infrastructure.

Step 1 → terraform apply          # Provision VPC, Subnet, SG, S3
Step 2 → aws s3 cp gtd_merged.csv # Upload raw data to S3
Step 3 → EMR Step (PySpark)       # Preprocess 20M+ records → JSONL
Step 4 → Colab Notebook (T4 GPU)  # QLoRA fine-tune → upload GGUF to S3
Step 5 → EC2 (t3.xlarge)          # Ollama + HORUS Custom UI (Nginx/Docker)
Step 6 → terraform destroy        # Teardown — avoid cost overrun

Full commands for each step are in the sections below.

S3 (Raw GTD + GDELT)
       │
       ▼
EMR Cluster [25bbdf-horus-emr-cluster]   ← PySpark preprocessing + EDA
(Status: Terminated immediately after use)
       │
       ▼
S3 (Processed JSONL — train/val/test)
       │
       │                         ┌────────────────────────────────┐
       ▼                         │  Google Colab (T4 GPU).        │
Fine-Tuning (Unsloth QLoRA).  ◄──┤  horus_osint_fine_tuning.ipynb │
       │                         └────────────────────────────────┘
       ▼
S3 (horus-llama3-osint-Q4_K_M.gguf)
       │
       ▼
EC2 [25bbdf-g23-ec2] — t3.xlarge
  ├── Ollama         (port 11434 — VPC-internal only)
  └── HORUS UI       (Dockerized Nginx, port 80 — public)
       │
       ▼
User Browser  →  http://<ec2-public-ip>

.
├── main.tf                         # Terraform — VPC, Subnet, IGW, SG, S3
├── pyspark_job.py                  # PySpark preprocessing pipeline (EMR)
├── horus_osint_fine_tuning.ipynb   # Unsloth QLoRA fine-tuning notebook (Colab)
└── README.md                       # This file

pip install torch transformers datasets trl peft accelerate bitsandbytes boto3 unsloth

Note: The fine-tuning notebook is designed for Google Colab (T4 GPU). To reproduce locally, you need a CUDA-compatible GPU with ≥16 GB VRAM and the dependencies above. All notebook cells include inline installation commands — Colab is the recommended environment.

Ensure your AWS IAM user or role has the following policies attached before running any step:

EMR also requires a service role (AmazonEMR-ServiceRole) and an EC2 instance profile (AmazonEMR-InstanceProfile) — these are created automatically when you launch EMR via the console for the first time.

Combined raw records: 20M+ → Post-ETL training samples: 159,826

# Clone the repository
git clone https://github.com/MahmoudAlyosify/Horus-OSINT
cd Horus-OSINT

# Initialise Terraform providers
terraform init

# Preview resources before creation
terraform plan

# Apply — creates VPC, Subnet, IGW, Route Table, Security Group, S3 Bucket
terraform apply -auto-approve

Save the output values (vpc_id, subnet_id, security_group_id, s3_bucket_name) — required for Steps 3 and 5.

Resources created (all prefixed 25bbdf-g23-):

# Download GTD merged CSV from the START Consortium (requires registration)
# Then upload to S3:
aws s3 cp gtd_merged.csv s3://horus-25bbdf-g23-bucket/raw/gtd/gtd_merged.csv

# GDELT is accessed directly from the AWS Public Dataset — no upload needed:
# s3://gdelt-open-data/events/  (read directly in the PySpark script)

aws s3 cp pyspark_job.py s3://horus-25bbdf-g23-bucket/scripts/pyspark_job.py

Step type : Spark application
Name      : 25bbdf-g23-preprocessing
Script    : s3://horus-25bbdf-g23-bucket/scripts/pyspark_job.py
Arguments : --bucket horus-25bbdf-g23-bucket

# Verify output files were written to S3
aws s3 ls s3://horus-25bbdf-g23-bucket/processed/ --recursive

Expected output:

processed/train.jsonl      (~150k samples, 95%)
processed/val.jsonl        (~5k samples,   2.5%)
processed/test.jsonl       (~5k samples,   2.5%)

1. Open horus_osint_fine_tuning.ipynb in Google Colab
2. Set runtime: Runtime → Change runtime type → T4 GPU
3. Add the following to Colab Secrets (icon in left sidebar):
   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • HF_TOKEN (HuggingFace token with Llama-3 access)
4. Run all cells in order

The notebook will:

• Download train.jsonl from S3
• Fine-tune Meta-Llama-3-8B-Instruct with QLoRA (4-bit NF4, Unsloth)
• Export the merged model to horus-llama3-osint-Q4_K_M.gguf
• Upload the GGUF to s3://horus-25bbdf-g23-bucket/models/
• Publish adapter weights to HuggingFace Hub

Deploy the instance into 25bbdf-g23-vpc, then SSH in:

ssh -i "Horus-key-pair.pem" ubuntu@<25bbdf-g23-ec2-public-ip>

To allow the client-side UI to call the Ollama API directly, CORS and host binding must be configured via a systemd override — not just an environment variable — so the setting persists across reboots.

# Install the Ollama LLM runner
curl -fsSL https://ollama.com/install.sh | sh

# Configure CORS and host binding via systemd override
sudo mkdir -p /etc/systemd/system/ollama.service.d
echo -e "[Service]\nEnvironment=\"OLLAMA_HOST=0.0.0.0\"\nEnvironment=\"OLLAMA_ORIGINS=*\"" \
  | sudo tee /etc/systemd/system/ollama.service.d/override.conf

# Reload daemon and restart Ollama
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Verify it is running and listening on all interfaces
sudo systemctl status ollama

# Install AWS CLI
sudo snap install aws-cli --classic

# Download GGUF from S3
aws s3 cp s3://horus-25bbdf-g23-bucket/models/horus-llama3-osint-Q4_K_M.gguf \
    ./horus-llama3-osint.gguf

# Create Modelfile and register with Ollama
echo "FROM ./horus-llama3-osint.gguf" > Modelfile
ollama create horus-osint -f Modelfile

# Verify model is registered
ollama list

Instead of a generic off-the-shelf interface, a bespoke HTML/JS Single Page Application was engineered and deployed to render streaming Markdown outputs as structured "Intelligence Report Cards" matching the HORUS brand identity.

# Install Docker
sudo apt-get update && sudo apt-get install -y docker.io
sudo systemctl enable docker && sudo systemctl start docker

# Create web directory and transfer UI assets (index.html, logo.png)
mkdir -p ~/horus-ui
# scp index.html logo.png ubuntu@<ec2-ip>:~/horus-ui/

# Deploy lightweight Nginx server on port 80 — auto-restart on reboot
sudo docker run -d \
  -p 80:80 \
  --name horus-custom-ui \
  --restart always \
  -v ~/horus-ui:/usr/share/nginx/html:ro \
  nginx:alpine

# Verify container is running
sudo docker ps

Access the interface at:

http://<25bbdf-g23-ec2-public-ip>

curl http://localhost:11434/api/generate \
  -d '{
    "model": "horus-osint",
    "prompt": "Identify the threat level of bombing incidents in Afghanistan in 2019.",
    "stream": false
  }'

# Terminate EC2 instance via console first, then:
terraform destroy -auto-approve

Always run terraform destroy after submission to prevent exhausting the AWS credit.

To demonstrate the full operational capabilities of the fine-tuned OSINT LLM, the team engineered and deployed a custom, production-ready Single Page Application (SPA) tailored to the HORUS cyber-intelligence brand identity. Hosted via a Dockerized Nginx server on the AWS EC2 instance, the interface connects directly to the Ollama backend API with custom CORS configuration and dynamic streaming rendering — converting raw Markdown into formatted Intelligence Report Cards in real time. Both desktop and mobile responsiveness were validated.

The fine-tuned model was stress-tested with two multi-dataset correlation prompts designed to validate that GDELT geopolitical context and GTD tactical incident data were correctly learned and can be synthesized in a single response:

Phase 1 — Deep Strategic Assessment:

"Execute a strategic OSINT threat assessment for Iraq covering the year 2014. Correlate GDELT political instability metrics with GTD terror incident data. Focus on identifying primary target demographics, dominant attack modalities used by insurgent groups, and the overall threat severity level. Present the findings strictly in the HORUS Intelligence Report format."

Phase 2 — Tactical Field Retrieval:

"Generate a HORUS Intelligence Report for Syria, 2015. Correlate GDELT instability with GTD tactical attacks, highlighting weapon modalities and target demographics."

In both evaluations, the model adhered to strict HORUS Intelligence Report Card formatting while accurately synthesizing multi-source threat demographics — confirming that the GTD+GDELT join performed during PySpark preprocessing successfully transferred into the model's domain knowledge. The seamless deployment of this architecture demonstrates the team's ability to bridge large-scale Big Data pipelines, advanced LLM fine-tuning, and scalable cloud infrastructure into a unified, end-to-end intelligence product.

Disclaimer on Cost Estimation: Due to the strict IAM policy restrictions enforced on our federated university AWS account, direct access to the AWS Billing and Cost Management console is restricted. Consequently, the cost metrics presented below are rigorous manual estimates. These calculations are derived from the official AWS on-demand pricing for the us-east-1 (N. Virginia) region mapped against our exact resource provisioning timelines.

To ensure strict compliance with the $100.00 USD hard budget cap, the team employed aggressive FinOps governance. This included terminating the EMR cluster immediately post-preprocessing and offloading the VRAM-intensive LoRA fine-tuning process entirely to a free-tier Google Colab (T4 GPU) environment.

Estimated AWS Cost Breakdown (USD):