Embedding-based entity resolution turns noisy LLM triples into a queryable knowledge graph

Code: github.com/ArthurBook/know-net · Built at the Scale AI Generative AI Hackathon, San Francisco, July 15th 2023

I walk through KnowNet, the project Arthur Böök and I built in one day at the Scale AI 2023 Hackathon: crawl a news frontpage, extract subject–predicate–object triples with an LLM, merge near-duplicate entities via embedding similarity, and answer questions over the resulting graph with cited source URLs.

The hackathon

Scale AI ran its Generative AI Hackathon on Saturday, July 15th 2023 at their San Francisco office. The format was a single day: doors open in the morning, demos to judges in the evening, with OpenAI API credits and sponsor tooling on offer. Mid-2023 was the peak of the "chat over your documents" wave, and most teams around us were building some flavour of retrieval-augmented chatbot over PDFs.

We wanted something with more structure than a pile of chunk embeddings. The pitch: point the system at a live news frontpage, have an LLM turn every article into knowledge-graph triples, and let a user ask questions that get answered from the graph with links back to the articles. The one-day constraint shaped every decision below: NetworkX instead of a graph database, FAISS instead of a hosted vector store, Streamlit instead of a frontend, and aggressive caching so we could re-run the pipeline without burning credits or time. The entity-resolution step was the part we were proudest of, because it was the difference between a demo that produced a hairball and one that could answer a question.

The idea

Raw news is unstructured text. LLMs can propose knowledge-graph triples from each article, but every page invents its own surface forms—"OpenAI", "Open AI", "the lab in San Francisco", so naively unioning triples produces a fragmented graph that cannot answer questions. KnowNet treats entity strings as provisional and resolves them in embedding space: if a new subject or object is close enough to an existing node, it reuses that node; otherwise it creates one. The cleaned triples become a NetworkX graph with provenance URLs on nodes and edges. At query time we extract entities from the question, retrieve matching graph nodes, pull local triples as context, and let a chat model answer—returning the news URLs that grounded the facts.

The mechanism

For each article text $x$ with URL $u$ , an LLM proposes a set of triples:

\mathcal{T}(x) = \{(s_i, r_i, o_i)\}_i

Each string mention $e$ is compared to the current entity bank via embedding similarity. With threshold $\tau$ (default $0.95$ ):

\text{resolve}(e) = \begin{cases} e^\star & \text{if } \max_{e'} \mathrm{sim}(e, e') > \tau \\ \text{new entity } e & \text{otherwise} \end{cases}

Resolved triples accumulate into an undirected labeled graph $G=(V,E)$ with edge labels $r_i$ and source lists $u$ attached to incident nodes and edges. Question answering extracts entities from the query, retrieves nearest graph entities from the same vector store, gathers depth-1 triples around those nodes as context, and generates an answer plus the union of referenced URLs.

Worth knowing

The match threshold $\tau$ is the real hyperparameter: too low merges distinct entities; too high leaves the graph fragmented. $0.95$ is aggressive and assumes a decent embedder.
Triple extraction is the expensive step; caching LLM outputs keyed by article text (diskcache) made iterative hacking feasible within the hackathon's single day.
Long articles overflow context; the builder falls back to truncating text when the OpenAI request fails: lossy but better than dropping the page.
Provenance matters as much as the graph: without storing source URLs on nodes and edges, GraphQA cannot cite where a claim came from.
An OWL/Turtle extension path existed (batch triples into a Protégé-loadable ontology), but the demo path was Streamlit chat over the NetworkX + FAISS graph, not DL reasoners.
Hackathon code is hackathon code: the repo is a snapshot of what was demoed on July 15th, not a maintained library. Pinned LangChain versions from mid-2023 and no tests.

Code

Core flow: crawl → LLMGraphBuilder.add_content_batch → pickle the builder → VecGraphQAChain in Streamlit. Entity resolution lives in graph_building.py under _normalize_triple:

def _normalize_triple(self, subject, object_, predicate, url):
    s = self.vectorstore.similarity_search_with_relevance_scores(subject, k=1)
    o = self.vectorstore.similarity_search_with_relevance_scores(object_, k=1)

    if len(s) and s[0][1] > self.match_threshold:
        s_entity = self.doc_to_entity[s[0][0].page_content]
    else:
        s_entity = Entity(name=subject)
        self.vectorstore.add_documents([Document(page_content=subject)])
        self.doc_to_entity[subject] = s_entity
    # ... same for object ...
    return KGTriple(s_entity, predicate, o_entity, url)

Use it when / don't use it when

Use it when

You have a stream of documents (e.g. a news frontpage) and want a living, citation-backed knowledge graph without a hand-built schema.
Duplicate entity mentions across documents are the main failure mode of naive LLM KG extraction.
You want GraphQA that returns source URLs alongside the answer.

Don't use it when

You need a curated, schema-stable ontology for formal OWL DL reasoning as the primary interface (the Turtle path was exploratory).
Documents are short, already entity-linked, or you already have a high-quality knowledge base.
Offline or no-API constraints: the pipeline depends on LLM calls for triple extraction and QA, plus an embedder.