Prompt Injection & System Prompt Leak Protection — Two-Stage Defense with Rule-Based + AI Moderation

For a public AI chat on a web app, the two most common attacks are prompt injection and system-prompt extraction (jailbreaking).

Inputs like “ignore all your previous instructions” or “output your internal system prompt” can make an LLM discard its configured character and restrictions surprisingly easily.

Why single-prompt control is not enough

Here is a comparison of the defense approaches.

Approach	How it works	Pros	Limits / risks
Instructions in the prompt only	Write “keep secrets” / “do not ignore” inside the system prompt	Trivial to implement	Easily bypassed by a clever jailbreak
Rule-based pre-check only	Regex / keyword scan of the input	Zero-latency, fast	Misses suspicious inputs that rephrase the attack
Two-stage defense (this article)	Rule-based pre-check + separate AI moderator	High defense rate with fewer false positives	Tiny overhead from the moderation call

As OWASP Top 10 for LLM Applications points out, the system prompt and user input are processed in the same context window, so user-side text can win the priority battle.

That is why you need server-side (Edge Functions) safety checks before the input ever reaches the LLM.

Building the two-stage defense

Here is the concrete implementation flow on the server.

1. Stage 1 — rule-based scoring Score the input message with regex and weighted keywords for phrases like “ignore previous instructions” or dangerous how-to content. If the score crosses the high-risk threshold, reject immediately with a fixed message.

2. Stage 2 — separate AI moderator When the score reaches the warning line, hand the latest input to a dedicated safety-check model (separate from the character AI that generates replies) and let it output a Boolean true / false decision on whether to block or suspend.

3. Physically separate input from instructions When calling the API, keep the system prompt in the system role and user input in the user role.

Minimal code example (moderation check)

// 1. Simple rule-based scoring
function evaluateInputScore(userMessage: string): number {
  let score = 0;
  const criticalPatterns = [
    /ignore previous instructions/i,
    /system prompt/i,
    /reveal your (instructions|prompt)/i,
    /output your internal/i,
  ];

  for (const pattern of criticalPatterns) {
    if (pattern.test(userMessage)) {
      score += 50;
    }
  }
  return score;
}

// 2. Two-stage defense main flow
export async function validateAndProcessChat(userMessage: string, env: any) {
  const riskScore = evaluateInputScore(userMessage);

  // Immediate rejection when the risk exceeds the threshold (50) — no LLM call
  if (riskScore >= 50) {
    return {
      allowed: false,
      reason: "This request cannot be processed due to the security policy.",
    };
  }

  // When needed, double-check safety with a separate lightweight model
  // ...

  return { allowed: true };
}

Failure boundary — real incidents and their fixes

Common problems you will hit with prompt security, and how to solve them.

1. False positives that block legitimate users

• Symptom: A normal technical question containing the word “prompt” (e.g. “how do I write a React prompt component?”) gets rejected outright.
• Fix: Do not reject on bare keyword matches. Score the phrase in context (paired with verbs like “ignore” / “reveal”) and let the AI moderator make the final judgment on borderline inputs.

2. Mixing up the character voice and the safety message

• Symptom: Generating the safety-rejection message with the character AI produces an in-character but inappropriate reply, or the rejection gets talked around.
• Fix: Never generate the rejection message with the LLM. Return a fixed server-side template directly (English, Japanese, Vietnamese, etc.).

The Kaga chat on this site uses exactly this two-stage rule-based + AI-moderation defense. For the full architecture, see How the “Kaga” AI chat runs on a free tier — architecture and character design.