What is a Hypergraph?

// Define entities and attributes
prototype UnitedStates : Country;
prototype Brazil : Country;
prototype Germany : Country;

prototype Tense;
prototype Future : Tense;
prototype Past : Tense;
prototype Present : Tense;

// Define relationship structure
prototype President
{
    Country Country;
    Tense TimeFrame;
}

// Subtypes for Metagraph relationships
[SubType]
prototype PresidentOfUnitedStates : PresidentOf
{
	function IsCategorized() : bool 
	{
   return this -> { this.Country typeof UnitedStates };
}
}

[SubType]
prototype FormerPresident : President
{
function IsCategorized() : bool 
	{
   return this -> { this.TimeFrame typeof Past };
}
}

// Use relationships in the metagraph
JoeBiden typeof President;                     // true
JoeBiden typeof PresidentOfUnitedStates;       // true
AngelaMerkel typeof PresidentOfUnitedStates;   // false
DonaldTrump typeof PresidentOfUnitedStates;    // true
DonaldTrump typeof FuturePresident;            // true
DonaldTrump typeof FormerPresident;            // true

• Subtypes are automatic
• Subtypes are dynamic, they will change with the underlying data.
• Subtypes allow a combinatorial number of categorizations

// Define basic graph structure
prototype TeamMember
{
    Team Team;

    //Constructor
    function TeamMember() : void 
    {
        this.Team.Add(this);  //automatically build the Team.TeamMembers collection
    }
}
prototype Team 
{
    Collection<TeamMember> TeamMembers;
}

prototype IntelligenceFactoryTeam;
prototype TeamAmericaTeam;
prototype TeamBrazilTeam;

prototype IntelligenceFactoryTeamMember : TeamMember
{
    Team.Team = IntelligenceFactoryTeam;
}

prototype Matt : IntelligenceFactoryTeamMember;
prototype Justin : IntelligenceFactoryTeamMember;
prototype Giancarlo : IntelligenceFactoryTeamMember;
prototype Flavio : IntelligenceFactoryTeamMember;

• The ProtoScript version is a quick lookup, there is no scan. The information is already organized
• The SPARQL needs to scan the nodes, looking for one(s) that satisfy the constraint.

• Better Contextual Understanding: Representing nested and hierarchical relationships.
• Enhanced Decision-Making: Leveraging semantic memory for more informed outcomes.
• Scalability and Flexibility: Handling complex, non-pairwise relationships with ease.

• Character-Based Chunking: This approach splits text into fixed-length segments, typically measured by the number of characters or tokens. While it ensures predictable and uniform chunk sizes, it often disrupts sentences or ideas mid-way, leading to incomplete or nonsensical chunks. For example, a sentence might be split across two chunks, losing coherence and context.
• Recursive Chunking: Recursive chunking uses natural separators like paragraphs, headings, or punctuation to create chunks. This approach produces more natural divisions compared to character-based methods. However, it doesn’t guarantee that each chunk is semantically coherent, as it relies purely on structural cues rather than the meaning of the content.

• Retrieval-Augmented Generation (RAG): By creating semantically coherent chunks, RAG systems can retrieve and interpret relevant information more effectively.
• Text Summarization and Clustering: Large documents, such as books or research articles, can be grouped into clusters of related content, enabling faster insights.
• Visual Exploration: Dimensionality reduction techniques like UMAP, combined with clustering and labeling via LLMs, allow users to visualize the structure and flow of a document, providing both development insights and practical tools for analysis.

• Contextual Embeddings: Before creating embeddings for text chunks, explanatory context is added to each chunk. This context is specific to the chunk and situates it within the broader document, improving its relevance when retrieved. For example, a chunk stating "The company's revenue grew by 3%" might be augmented with the context "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023."
• Contextual BM25: BM25 is a ranking function that uses lexical matching to find exact term matches. By applying BM25 in conjunction with semantic embeddings, Contextual Retrieval ensures that both exact matches and semantic similarities are used to retrieve the most relevant chunks, improving the overall retrieval accuracy.

• Contextual Embeddings: Before creating embeddings for text chunks, explanatory context is added to each chunk. This context is specific to the chunk and situates it within the broader document, improving its relevance when retrieved. For example, a chunk stating "The company's revenue grew by 3%" might be augmented with the context "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023."
• Contextual BM25: BM25 is a ranking function that uses lexical matching to find exact term matches. By applying BM25 in conjunction with semantic embeddings, Contextual Retrieval ensures that both exact matches and semantic similarities are used to retrieve the most relevant chunks, improving the overall retrieval accuracy.

• Preprocessing for Context Clarity: Unlike standard systems, SemDB preprocesses the text before chunking or embedding. Pronouns are replaced with explicit references, long-range dependencies are resolved, and sentences are rewritten for clarity. This ensures that each sentence captures its full context independently, leading to more accurate embeddings.
• Recursive Chunking for Precision: Using recursive semantic chunking, SemDB can isolate highly specific sections without relying on comparisons between sentences. This approach enhances retrieval by ensuring that each chunk is both meaningful and distinct.
• Combining Multiple Strategies: SemDB doesn’t rely solely on contextual chunking. Its robust pipeline includes:‍‍
• Combining Multiple Strategies: SemDB doesn’t rely solely on contextual chunking. Its robust pipeline includes:‍‍
• Context Chunking: For preserving local context.
• Recursive Chunking: To create semantically coherent segments.
• Ontology-Based Enhancements: Leveraging domain-specific ontologies to enrich understanding and retrieval.
• Scalability for Large Documents: SemDB is adept at handling massive documents, such as 150+ page financial PDFs, by combining contextual embeddings with recursive chunking. This ensures that even granular details remain accessible while preserving overarching context.