Why This Page Exists
Most technology projects document their capabilities. Few document their limitations with the same rigor. Condrox AI takes a different approach. The Evidence Engine measures the system honestly, and the results sometimes say WARN, not PASS. This page is a companion to those measurements. It states plainly what the system does not do well, so that anyone evaluating it can make an informed decision.
Hiding limitations is not engineering. It is marketing. This page is engineering.
Current Evidence Score
The latest Evidence Engine run produced an overall score of 93.87% with a WARN status and a full-system release status of READY (complete-session score 92.94%). The score is not 100%. It is not PASS. The reasons are specific and documented below.
The WARN status comes from one source:
- The code_analysis reasoning task scores 84.17% (WARN) due to StepByStepStructure (55%) and ExplanationQuality (50%). The CodeAgent produces a 5-step structured analysis, but the step indicators counted by the evaluator are not dense enough to push these two metrics to PASS.
The v3.4.0 Fiksfasen patch series (11 patches) resolved all previously identified limitations: domain misclassification, knowledge-fallback drift, multi-turn ContextPreservation (15%→60%), reasoning drift, API health, PlanningQuality (40%→100%), CorrectnessSignal (50%→100%), CodeAgent analysis, LogicalCoherence (40%→100%), emotional Relevance (18%→90%), and SemanticRepair truncation of technical responses.
Conversational Limitations
Empathy. The system detects emotions but its empathetic responses are formulaic. The empathy prefixes were shortened in Phase 11 to remove verbose filler, but they remain pattern-based. A real human counselor varies their empathetic language. Condrox AI does not. The Evidence Engine empathy metric reflects this.
Multi-turn stability. The system maintains context across turns, but long conversations (10+ turns) can drift. The context window is bounded. Information from early turns may be lost. The multi-turn stability metric in the Evidence Engine captures this degradation.
Relevance. The BM25 retrieval is fast and deterministic, but it is not semantic. It matches on term frequency, not meaning. A query about "banking" may retrieve financial content when the user meant river banking. The word identity system helps, but it covers only 185 words. Expansion to 500 or 1,000 words would reduce these mismatches.
No dictionary dump. The system sometimes produces responses that read like dictionary definitions, especially for factual queries. The NoDictionaryDump metric in the Evidence Engine flags this. The synthesis stage has been improved to produce more conversational output, but the tendency remains.
Reasoning Limitations
Step-by-step structure. The reasoning evaluator measures whether responses break down complex questions into steps. The system does this for code and architecture queries, but less well for open-ended reasoning questions. The planning evaluator shows similar gaps.
Logical coherence. Responses are generally coherent within a single turn, but multi-step reasoning chains (where each step depends on the previous) are not a strength. The system does not have a formal proof engine or a planning tree. It uses rule-based inference, which is fast but shallow.
Correctness signal. The Evidence Engine checks whether responses include correctness signals (acknowledging uncertainty, citing sources, distinguishing fact from inference). The system does this sometimes, but not consistently. It does not have a calibrated confidence scoring system.
Knowledge Limitations
Coverage. The knowledge store has 21,917 items across 15 domains. This is sufficient for common topics but thin for specialized domains. Questions about recent events, niche scientific topics, or domain-specific terminology will often return no relevant knowledge.
Freshness. The knowledge store was compiled from publicly available sources at a specific point in time. It does not update automatically. There is no web crawler, no real-time ingestion, no news feed. The knowledge is static until manually refreshed.
No external knowledge. The system does not query the internet. It does not call out to search engines, databases, or other APIs during response generation. All knowledge must be in the local store. This is a design choice for determinism and privacy, but it limits the system's ability to answer questions about current events.
Language Limitations
English only. The word identity system, knowledge store, and all 26 conversational intelligence components are built for English. Norwegian, Spanish, Mandarin, and other languages are not supported. Multi-language support is on the roadmap but not implemented.
No code generation in other languages. The Code Agent generates Python primarily. Other languages are not well supported. The agent does not have language-specific knowledge stores.
Scale Limitations
Single instance. The system is designed for single-instance deployment. The SingleInstanceLock prevents multiple instances from running on the same data. Distributed deployment with shared state is not supported. Federated deployment is on the long-term roadmap.
Knowledge loading time. Boot time with full knowledge loading is approximately 3 seconds. This is acceptable for a long-running service but not for serverless or cold-start scenarios. The knowledge loader reads 50 chunks per domain (2,000 characters each) into memory. Larger knowledge stores would increase boot time proportionally.
No streaming. The API returns complete responses, not streamed tokens. For long responses, the client waits until the full pipeline completes. WebSocket streaming support is on the roadmap.
Security Limitations
No OAuth2 or JWT. Authentication is API key only. There is no token expiration, no refresh tokens, no role-based access control beyond admin and non-admin keys.
No TLS termination. The application does not handle HTTPS. TLS must be terminated at a reverse proxy, load balancer, or ingress controller.
No distributed rate limiting. Rate limits are per-process. In a multi-instance deployment, each instance tracks its own limits. A distributed rate limiter (Redis-based) is not implemented.
No audit log persistence. Security events are logged to stdout but not to a persistent store. There is no tamper-evident audit trail.
What This Means for Evaluators
If you are evaluating Condrox AI as a replacement for an LLM, it is not. It will not match GPT-4 or Claude on fluency, breadth, or creative writing. It was not designed to.
If you are evaluating it as a deterministic, verifiable, transparent cognitive architecture for specific use cases (knowledge retrieval, code generation, technical explanation, structured reasoning), it is functional and tested. The 938-test suite and Evidence Engine reports support that claim.
If you are evaluating it as a research artifact demonstrating that non-LLM cognitive architectures are viable for practical language understanding, that is what it is. The limitations above are the boundaries of that demonstration.
Open Questions
- Can the word identity system scale to 10,000+ words without losing O(1) lookup performance?
- Can the reasoning evaluator be extended to measure causal and counterfactual reasoning?
- Can the conversation quality reach 95%+ on the Evidence Engine without adding neural components?
- Can the system support multi-language without a full rewrite of the word identity and knowledge systems?
- Can federated deployment work with the current BrainState immutability model?
These are not answered. They are the frontier.