:: Synapse Wisdom Engine ::

The Redux of the Original Idea that Started It All – Part I

“A key challenge in artificial intelligence is the creation of systems capable of autonomously advancing scientific understanding by exploring novel domains, identifying complex patterns, and uncovering previously unseen connections in vast scientific data.” – MIT Authors

Hypothesis: Can hidden links between disparate knowledge be drawn out through Knowledge Graphs and recent developments in Natural Language Processing, specifically Large Language Models with a strong understanding of knowledge structures?

TL;DR: A research paper was released using a number of the approaches and techniques I had implemented. The research by the team at MIT, shows that that scientific knowledge can be expanded with graphs, demonstrating a connection between symphonies and material science (the author’s specialty):

The secondary paper is an important piece as the agent-swarm is receiving, parsing and hopefully producing valuable intelligence from real Data. Through the interactions of the agents, the goal is to categorize inbound data streams into nodes and the connections between those nodes. The paper was released this month and code is available on paperswithcode.

The Researchers even included sub-sampling, though I’m unclear on its persistence. I wanted to use them ephemerally to “fingerprint” potential patterns in streams of data particularly between a person and an agent or chain-of-agents. For exploration…

Genesis:

So where did it start? About 6 months ago I found my ex looking at me bemused. Not unusual for my peculiarities but I wasn’t sure what was so amusing. It turns out that while I was visualising my thoughts, from the outside I was starting at a wall after wondering if there was a way to bridge disparate ideas and what that might look like. After about 10 minutes my ex snapped me out of it but the seed was planted. Meta cognition has been of great interest to me, particularly my own but also the philosophical and metaphysical questions of Selfhood, Consciousness and beyond.

Graphs? A graph is a simple mathematical object that is useful for studying connections between absolutely anything! Graphs can help us study everything from the internet to transportation networks, and from ecological systems to groups of friends. – Mathworld.

Modelling My Thoughts:

My personal brand of meta cognition, I am particularly well acquainted with. For better or worse, I have known it somewhat (0.0001%) for 36 years. It is for the worst when there is no off switch. Conceptually though, it seemed obvious to me that the way between ideas clustered in a certain area was that of a graph. Having little more than a layman’s interest in Mathematics, Physics and Computer science it was the natural “data structure’ of my mind. Collections of data points, blocks of data formed into principles, maybe a sprinkle of wisdom showing itself on the good days.

A Hypothesis!

Enter Knowledge Graphs and the beginning of Synapse which has become Orion Threat Intelligence and a myriad of other knowledge and semantic meaning extraction projects. Not just my own but the ecosystem is exploding. It’s required me to dig through research papers as there has been a resurgence of interest in Knowledge Graphs both as a way of injecting some determinism into a Large Language Model. That is – to keep it on track… but the emphasis with a knowledge graph is on the relationships between data.

I am finding myself using tools without a stable version and that ship updates on almost a daily basis. In concert with learning those tools it presents an intellectually satisfactory bliss. Myself and an early Collaborator were lucky enough to get 3 hours from one of the gentlemen who was with Google Research in 2013 and was working on Knowledge Graphs, thank you @Mark.

An example of a knowledge graph is the “social graph” of twitter or Facebook Brad ->> Is Dating —>> Alice and Alice + Sarah went to Grilld Burgers last week and Brad had only been there the week before. Social media companies track every action, reaction and use that with their powerful models some of which are based on graphs, others that serve different purposes with the best look under the hood coming from the Netflix Engineering blog.

So, as with all matters that are sufficiently difficult I was hung hoe into the project. I didn’t know the steps in between but I had a hypothesis that Knowledge Graphs, combined with the deep semantic understanding of Large Language Models could lead to some novel breakthroughs. Not a single model but a system, a cohort of models and ultimately with the use of “ephemeral graphs” as I called them. I was aiming for what I call ““Symmetrical Intelligence Augmentation”“` and with a novel use case could surely generate these graphs on the fly.

The idea being that a human interacting with the cohort produces the nodes and relationships through the processing pipeline and those connections made by humans supplement the relationships previously learnt. It is not necessarily a live learning exercise but the combination of human feedback and a proposed Reinforcement Learning Agent System aspire to direct the model in the right path. As with all recommendation engines, let’s use Spotify for example. You are a datapoint in high-dimensional space and live among the genres and people that listen to music like you. Spotify knows what you are looking for based on the recommendation engine and a myriad of other metrics. Again, Netflix Engineering blog is a great resource as is hacking away at your own!

So I thought KnowledgeGraph generation was a solved problem — I was mistaken. There are many graveyards of failed experiments but the skills that I have learnt, though not specialised span the entire ML/MLOps pipeline from data analysis and cleaning to Infrastructure as Code managed Cloud providers. Everything, after all doesn’t serve any purpose but to shine light further down the path. I knew that I wanted to see this applied to Research bodies and institutions for finding deep links that are not obvious to humans but across thousands/millions of nodes and petabytes of metadata perhaps there were links unseen.

Make The Thing Work Alex – Find a Use Case

Most – I don’t see the point,

Being in the world of the unknown is thrilling!

Generating a knowledge graph is traditionally done using NLP techniques liked Named Entity Recognition. Don’t quote me on this but the guy, Mr Knowledge graph used it for his though it was limited by schemas defined on schema.org. If my mad scientist theory were to be useful it would need to interpret data, structure is and find relationships if not in real time, pretty close to it.

There are algorithms optimised for such graph traversal (moving from Node A at {x, y, z} to Node B {x,y,z} where Node A is Alice and Node B is Bob. It is is said in Mathematic Folk lore that some of the early mathematicians to work in the intersecting fields of Topology and Graph Theory and developed some of the algorithms for shortest path, clustering etc as an exercise born out of boredom (as all great things are!) Side note: So grateful to be able to learn this stuff without TOO much Mathematics. Doing the math for back prop or other implementation details that scrap folk at Meta and Google did in developing pytorch or tensor flow and related libraries is intense.

Optimising this into a problem that is not big O: O(x^2) (at worst) in time complexity is hard (I am reaching the bounds of my knowledge here so mathematicians and computer scientists cut me some slack.) Having said that the Transformer architecture to my knowledge scales quadratically in much the same way because the multi-headed attention mechanism that sets it apart from previous Recurrent Neural Networks etc focuses on sets of of tokens. Attention Mechanism Follow (Thanks inter webs):

Attention(Q, K, V) = softmax(QK^T / √d_k)V

This mechanism along with other trickery and the positional encoding of embedded tokens influences how weights are updated based on the relevance of different parts of the input. That pair wise operation means for every input the entire batch has to run through the neural network and the error backpropogated (Again, sorry for any layman mistakes). Backpropogation for those unfamiliar is the step where the accumulated errors the network has seen are sent back through the network and the “weights” (strength of each connection) are updated. A fully trained model has ideal the set of weights which minimises the “loss function” – how wrong it is, or. optimizes the “reward function” – how right it is.

This is stream of consciousness.

As the number of possible nodes increases or relationships grow the connections will likely show a One: Many relationships where there are clusters around a root node that feeds the others. Let’s say an entity like RealGoodSchool and a whole bunch of Alice, Bobs and Garys’ go to that school. Professors branch off, topics etc etc. It really does depend on your use case.

This was the first crux. My dear friend Rob in London who I have yet to meet In person is gracious with his time and as humble as he is intelligent, and discerning. The rare breed of from what I know of him, combination of technical abilities and soft skill management and leadership. I have never had to hire an Engineering Manager and I hear good ones are hard to come by.

I wanted Rob to be an advisor, and then when he showed interest I was gobsmacked because to have a leader in the Executive like Rob, would be gold dust. That would require success not only at a Hobby Research Level but endorsement and then financial support. Software ain’t cheap, especially not multi-model architectures and complex algorithmic computations.

Do I feel lucky, punk?

Having said that as any sane professional would, Rob wanted me to prove it with a use case. I knew that the boundaries of Science is where I want to get but as Sir Knowledge Graph himself legend had to say “Alex just don’t bet the farm on it, this stuff comes out of MIT’s, Google’s and teams of 20PhD’s. He said it with encouragement and a truly paternal caring that I not waste this precious life pursuing something. Lucky for Me, I’d already decided that I wasn’t going to do anything beyond where it was interesting so we were in agreement. We did disagree that LLM’s were going to replace rather than supplement knowledge graphs and I will say Mr Mark, that the resurgence of GraphRAG (whyhow.ai – shout out Chia) and other techniques leveraging them, particularly with the below research – there is still hope for the humble Knowledge Graph.

After 6 weeks of experimentation I had a much better grasp of the concepts I was playing with thanks to Arxiv.org and other open research body’s as well as the Professor that never sleeps – GPT4 (and now Claude) – I would suggest that when used with rapidity and iterative tenacity productivity 10 – 20x and the ability to parallel the conversations and learn about multiple aspects of a complex concept is bliss!!!! If Spotify was my music wet dream, the “Generative Pretrained Transformer” and all the super sharp people who built them are my intellectual wet dream. Though having said that its breadth of capabilities should not be underestimated.

So began a cycle of starting down a new path, finding the complexities and storing the Results for later review. I freaking LOVE Bear Notes for this. Ultimately the Synapse platform was going to be clustering nodes, before classifying them and then analysed using an also new-to-me technology of Graph Neural Networks– and do it from scratch in my mind. In Mid September I actually achieved my goal of getting (at least according to torchviz and tensor board a cohesive model design for such shenanigans. I also have an enormous amount of data for my current use case which will go into Part 2 – The Orion Threat Intelligence Platform that much of my blogging has revolved around in recent times. Not a diversion from the initial goals and methods of Synapse but rather an application of it satisying the use case. Having said that I am under no delusions of my ability to bring something like Orion to production. I just want to see the parts work together, maybe DoRA/QLora fine tune some varying model sizes for self-supervized learning and most of all – Have the Fun!

If you’ve come this far then thank you and gratitude. Writing like this helps me organize my thought and reminds me of how far I’ve come in 6 short months of frenetic focus on a niche area that seems to be gaining a lot of attention.

Up Next in Part 2:

• Synapse V0.0.1 – > 0.0.3
• Orion Hunts
• Orion AI Threat Intelligence

Now, if anyone is out there looking for someone who has dived deep in InfoSecand AI, then please shout out. I am more than happy to take a hair cut to land in a role and learn from people who know what they are doing. I have now worked across the board of the Software Development Lifecycle and although not an Engineer, read Clean Code and Clean Architecture – being a professional is a mindset

— Snyata

• GitHub
• Mail
• GitHub
• GitHub

Citations:

SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning},
author={Alireza Ghafarollahi and Markus J. Buehler},
year={2024},
eprint={2409.05556},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2409.05556},
}

@article{buehler2024graphreasoning,
author={Markus J. Buehler},
title={Accelerating Scientific Discovery with Generative Knowledge Extraction, Graph-Based Representation, and Multimodal Intelligent Graph Reasoning},
journal={Machine Learning: Science and Technology},
year={2024},
url={http://iopscience.iop.org/article/10.1088/2632-2153/ad7228},
}