The gradient

Ok so you have a function that takes a vector and spits out a single number. A loss function, for instance. The gradient is just the vector of all partial derivatives of with respect to each input:

That’s it. Each component tells you how much changes when you nudge the corresponding input variable. The gradient as a whole points in the direction of steepest ascent. If you want to minimize (which you usually do), you walk in the opposite direction. That’s gradient descent in a nutshell.

But here’s where the physics perspective adds something useful. In physics, the gradient of a scalar field gives you a vector field. If is a potential energy surface (think: a hilly landscape), then gives you the force field. A ball placed on that surface will roll in the direction of . Gradient descent is literally simulating a ball rolling downhill on your loss surface. Not metaphorically. Literally. The math is the same.

The gradient of a scalar field is a vector field pointing in the direction of steepest ascent. Negate it and you get the force that drives optimization.

The Jacobian

The gradient handles the case where your function maps : many inputs, one output. But what if you have many outputs too? A function that maps inputs to outputs? This is exactly what a neural network layer does. It takes an input vector and produces an output vector.

The Jacobian is the natural generalization. Instead of stacking partial derivatives into a vector (gradient), you stack them into a matrix:

Each row is the gradient of the -th output with respect to all inputs. So the Jacobian is really just a stack of gradients. If (one output), the Jacobian collapses to the gradient (transposed). They’re the same thing in different clothes.

The Hessian

Ok so the gradient tells you the slope. But the slope itself can change. Is the surface curving sharply or gently? Is it curving the same way in all directions? The Hessian answers these questions.

The Hessian is the matrix of second-order partial derivatives of a scalar function :

Or more compactly: . If is twice continuously differentiable (which it usually is in ML), then and the Hessian is symmetric. This is Schwarz’s theorem and it’s actually quite important because it means the Hessian has real eigenvalues, which we can interpret.

You might have noticed something. The Hessian is the Jacobian of the gradient. Just as the Jacobian generalizes the gradient to vector-valued functions, the Hessian is what you get when you apply the Jacobian operator to the gradient vector field:

These operators are nested. They build on each other.

The Laplacian

Now for the final member of our family. The Laplacian is the trace of the Hessian. Or equivalently, the sum of the unmixed second partial derivatives:

Where the gradient gives you a vector, and the Hessian gives you a matrix, the Laplacian collapses everything back to a single scalar. It’s a measure of the average curvature at a point. Or more precisely: it tells you how much the value of at a point differs from the average value of in a small neighborhood around that point.

If at a point, the function value is below the neighborhood average (the surface is concave up, like a valley). If , the function value is above the average (concave down, like a hilltop).

How they all fit together

Let’s put the hierarchy in one place:

Operator	Input	Output	What it tells you
Gradient		Vector ()	Direction of steepest ascent
Jacobian		Matrix ()	How each output changes w.r.t. each input
Hessian		Matrix ()	Curvature of in every direction
Laplacian		Scalar ()	Average curvature (trace of Hessian)

The relationships:

• The gradient is a special case of the Jacobian (when ).
• The Hessian is the Jacobian of the gradient.
• The Laplacian is the trace of the Hessian.

Each one peels back another layer of information about how your function behaves. The gradient gives direction. The Jacobian extends that to multi-output functions. The Hessian reveals curvature. The Laplacian summarizes that curvature into a single number.

Conclusion

I’ve heard people say “you don’t need to understand the math to do machine learning.” That’s technically true, the same way you don’t need to understand combustion to drive a car. But if you’re building the engine, debugging a misfire, or trying to make the thing go faster, it helps to know what’s actually happening.

These four operators (the gradient, Jacobian, Hessian, and Laplacian) are the vocabulary that physics developed over centuries to describe how fields change in space. Machine learning borrowed this vocabulary, sometimes without realizing it. Backpropagation is Jacobian products. Adam approximates the Hessian diagonal. Diffusion models are Laplacian dynamics. Normalizing flows are Jacobian determinants. It’s all connected.

The more of this you internalize, the less “magical” these algorithms feel. And the less magical they feel, the better you get at building them, fixing them, and knowing when they’re lying to you.

Don’t hesitate to reach out if I’ve made any mistakes or if something could be explained more clearly. I’m always happy to be corrected.

What the Law Actually Does

Let’s be precise about what we’re talking about, because the details matter.

The law, pushed by far-right National Security Minister Itamar Ben-Gvir, makes execution the default sentence for Palestinians convicted of deadly attacks (Al Jazeera 2026c). But here’s where it gets truly ugly.

Israeli citizens and settlers are explicitly excluded from military court jurisdiction. They get tried in civilian courts, with civilian protections, civilian appeals processes, and civilian legal standards. Palestinians face military tribunals. Two populations. Two completely different legal systems. One land.

The procedural provisions are staggering:

• Execution must be carried out within 90 days. Ninety days. That’s three months from conviction to death, with severely limited time for appeals.
• Access to legal counsel is restricted.
• Family visits are restricted.
• Judicial discretion is gutted. Judges can only impose a life sentence instead of death in vague “exceptional cases.”
• Sentence commutation is prohibited entirely.
• Execution personnel receive legal immunity.

These provisions are documented in detail by Human Rights Watch (Human Rights Watch 2026).

Now, consider this in context. The Israeli human rights organization B’Tselem has documented that military trials of Palestinians carry an approximately 96% conviction rate. Ninety-six percent. Based largely on confessions extracted under duress and torture.

So what we’re looking at is a system where Palestinians are tried in military courts with a near-certain conviction rate, based on coerced confessions, with a mandatory death sentence, with almost no appeals window, and no possibility of commutation. And this system applies to Palestinians and Palestinians alone.

That’s not a justice system. That’s an apparatus.

The International Response

Credit where it’s due. The international community has not been silent on this one.

UN human rights experts condemned the law as constituting a “discriminatory regime of capital punishment” that “manifestly violates Israel’s obligations under international human rights law” (Office of the United Nations High Commissioner for Human Rights 2026). They called for its immediate repeal.

Human Rights Watch was characteristically direct. Their deputy Middle East director stated: “Israeli officials argue that imposing the death penalty is about security, but in reality, it entrenches discrimination and a two-tiered system of justice, both hallmarks of apartheid” (Human Rights Watch 2026).

Apartheid. That’s not a word thrown around lightly by Human Rights Watch. When they use it, they mean it in the precise legal sense.

France, Germany, Italy, and the United Kingdom have all raised concerns (Al Jazeera 2026a). Amnesty International has called for urgent EU measures (Amnesty International 2026).

And the Party of European Socialists (Party of European Socialists 2026), along with the S&D Group in the European Parliament, have taken the strongest institutional European stance. They’ve called for the immediate suspension of the EU-Israel Association Agreement. Yannis Maniatis, the S&D Group’s vice-president for foreign affairs, put it plainly: “Reintroducing the death penalty is a step back into the past and yet another blow to the values that underpin our partnership with Israel. We cannot and will not remain silent. When a partner repeatedly ignores the warnings from its friends and civil society alike, there must be consequences” (Socialists and Democrats 2026).

The S&D described the law as a “grave democratic regression.” I’d call it something worse, but that framing carries weight in the European institutional context where it matters most.

Why This Should Concern You Personally

I know what some of you might be thinking. “This is terrible, but what does it have to do with me? I’m a software engineer in Stockholm (or Berlin, or Copenhagen, or wherever).”

Here’s why it concerns you.

The EU-Israel Association Agreement, which governs trade and political relations between Europe and Israel, contains an Article 2. That article conditions the entire agreement on respect for human rights and democratic principles. It’s not decorative language. It’s a legal obligation. And every time the EU fails to enforce it, that article becomes more meaningless. Not just for Israel. For every agreement the EU signs with every partner.

When Europe trades with a state that institutes ethnically targeted death penalty laws and does nothing, it sends a message. That message is: our values are negotiable. Our principles are for show. And that message weakens the foundations of the European project itself.

Over 9,300 Palestinians are currently detained in Israeli prisons, reportedly experiencing torture, starvation, and medical neglect (Al Jazeera 2026b). This new law doesn’t exist in isolation. It’s the sharpest edge of a system that has been documented and condemned by virtually every major human rights organization on the planet.

As Europeans, we are connected to this. Our governments maintain diplomatic and trade relationships with Israel. Our tax money flows into institutions that could act but choose not to. Our silence is a form of participation.

What You Can Do

I’m not interested in writing posts that make people feel bad and then give them nothing to do with that feeling. Outrage without action is just entertainment. So here’s what you can actually do.

Contact your elected representatives. If you’re in the EU, write to your MEPs. Tell them you support the suspension of the EU-Israel Association Agreement under Article 2. Be specific. Name the law. Name the article. MEPs pay attention to constituent mail more than you’d think, especially when it’s specific and informed. You can find your MEPs at the European Parliament’s website and most of them have public email addresses.

Support the organizations doing the work. Human Rights Watch, Amnesty International, B’Tselem, and the Palestinian human rights organization Al-Haq are all on the ground documenting what’s happening. They need funding. Even small amounts matter. These organizations are the reason we have the documented facts I cited in this post.

Demand your government takes a position. France, Germany, Italy, and the UK have “raised concerns.” Concerns are not enough. Push for your government to take concrete diplomatic action. Recall ambassadors. Suspend arms sales. Support referrals to the International Criminal Court. Diplomatic language is designed to sound like action while being none. Don’t let them get away with it.

Amplify Palestinian voices. Palestinians have been telling the world about these systems for decades. Organizations like the Palestinian Centre for Human Rights, Addameer (the prisoner support organization), and Defence for Children International Palestine document these realities daily. Follow them. Share their work. Let the people most affected by this law be heard.

Talk about it. I know this sounds simple, but social pressure is real. Bring it up with friends, colleagues, at dinner. Not to lecture. Just to make it visible. The machinery of oppression relies on people looking away. The single most subversive thing you can do is refuse to look away.

Make purchasing decisions that reflect your values. Look into which companies operate in or profit from the occupation. There are well-documented guides from organizations like the BDS movement. You don’t have to agree with every position to use the information to make informed choices about where your money goes.

A Note on Courage

I spent years in the Swedish Navy special operations. One thing that experience burned into me is that courage isn’t the absence of risk. It’s the decision that something matters more than the risk.

Speaking up about this carries social risk. I know that. People will disagree. Some will be angry. Some will call this post one-sided.

But here’s what I know. When a state passes a law that imposes death on people based on their ethnicity, tried in courts with a 96% conviction rate based on coerced confessions, with 90 days to execution and no possibility of commutation, there is no “other side” that makes this acceptable. There is no framing that makes this compatible with basic human rights.

The PES and the S&D Group understood this. The UN human rights experts understood this. Human Rights Watch understood this. It’s time the rest of us acted like we understood it too.

Conclusion

This law is not an abstract policy debate. It is a concrete, operational framework for state-sanctioned killing along ethnic lines. Every major human rights institution in the world has said so. The question is whether we’re going to treat that consensus as something that demands action, or something we scroll past.

Europe has a particular responsibility here. We have the institutional mechanisms (the Association Agreement, the European Parliament, our diplomatic weight) to exert real pressure. The PES and S&D have pointed the way. The question is whether the rest of European leadership will follow.

I don’t have all the answers. I’m a physicist who builds AI systems, not a diplomat. But I know this: when something is clearly, unambiguously wrong, the people who said nothing are remembered just as well as the people who did it.

Speak up. Act. Don’t look away.

Architecture of an AI Assistant

While there are many assistants out there, very few of them are actually able to do useful stuff. By stuff, I mean like answering your emails for you, prioritizing your day, paying your bills, cancelling subscriptions, call your mom, monitor your investments etc. The reason for this is twofold (i) it’s pretty darn hard to do right and (ii) the consequence of getting it wrong can be socially and financially catastrophic.

So what do we need for this to be practically achievable? Well, we need a language model that understands tool calling (Ding and Stevens 2025). This is an architecture that can emit tokens (text) that lets whoever is receiving that text know that they would like to call a tool. How this tool is supposed to be called can be learned either by a “Skill” or directly via an MCP server. The important thing here is that none of this requires a 5 trillion parameter model. The ability to understand when to call a tool and what arguments to pass is a capability that smaller models handle perfectly well. In fact, models like Qwen3:8b and Llama 3.1:8b have been specifically trained for tool calling and do a solid job of it.

And that brings me to the question I want to explore in this post: if a small model running on your laptop can do the things you actually need, why are we collectively pretending that only the biggest models matter?

A tale of David and Goliath

To put my money where my mouth is, I’m going to run a series of common tasks using both the biggest cloud models (GPT-5.2, Deepseek 2026) and a small local model (Qwen3:8b running via Ollama on my AMD Ryzen AI 7 350 with a Radeon 860M). No cherry-picking. I’ll show you the actual responses and let you judge.

The tasks I’ve chosen are deliberately mundane. Summarization, advice, and explaining a concept. These are the kinds of things people actually use LLMs for every day, not exotic benchmarks designed to stress-test reasoning at scale. My argument is simple: for everyday tasks, the small model does the job. And it does it at a fraction of the energy cost.

Summarization task

The text we’re going to summarize is the preface from Malcolm Gladwell’s “What the dog saw” (Gladwell 2009).

echo "Summarize the text below in 2 sentences $(cat mgv1.md)" | ollama run qwen3:8b 

sudo perf stat -e power/energy-pkg/ ./summary.sh
# Performance counter stats for 'system wide':
#         794.12 Joules power/energy-pkg/
#   39.278005283 seconds time elapsed

Giving advice task

echo "I want to wash my car. The car wash is just 200 meters \
down the street. Should I walk or drive there?" | ollama run qwen3:8b 

Explaining a concept

For the third task I asked both models to explain quantum mechanics in a short paragraph. This is a knowledge task where you might expect the bigger model to have an edge due to the sheer amount of training data it has absorbed.

echo "Create a small paragraph explaining quantum mechanics." \
| ollama run qwen3:8b 

What the numbers tell us

Let’s put the results side by side.

Score: Qwen3:8b wins 2, ties 1, loses 0. Against models that are 625 times larger.

Now let’s talk about what this costs. As I showed earlier, a single ChatGPT query consumes roughly 2.1 Wh while my local query consumed about 0.2 Wh. If you scale this to an organization sending, say, 1000 queries per day, the cloud approach burns 2,100 Wh (2.1 kWh) per day while the local approach uses 200 Wh (0.2 kWh). Over a year that’s 766 kWh versus 73 kWh. Multiply that across every company, every developer, every casual user, and you start to see why the energy argument matters even if you don’t care about the environmental angle (you should, but that’s a different post).

And the energy is only part of the story. There’s also the question of where your data goes. Every query you send to OpenAI, Anthropic, or Google travels to their servers, gets logged, and feeds their systems. When you run a local model, your data stays on your machine. For anyone working with sensitive business data, customer information, or anything remotely confidential, this isn’t a nice-to-have. It’s a requirement.

Conclusion

Look, I’m not saying big models are useless. They clearly have their place in complex reasoning tasks, multilingual generation, and problems where you genuinely need the breadth of knowledge that comes from training on the entire internet. I use them myself for certain things.

What I am saying is that the narrative of “bigger is always better” is convenient for the companies selling API access at scale and inconvenient for everyone else. The truth, as these simple experiments show, is that for the vast majority of what people actually use LLMs for (summarizing text, answering questions, drafting content, explaining concepts) a small model running locally does the job just as well. Sometimes better.

This matters for Europe especially (I’ve written about this before). If we accept the premise that only trillion-parameter models from American corporations can do useful AI work, then we’ve already lost. We become permanent API consumers, sending our data and our money across the Atlantic in exchange for a service we could provide ourselves. The small model movement (and it is a movement, with Mistral, Qwen, Llama, and others leading the way) is one of the most important developments in AI precisely because it breaks this dependency.

Run your own models. Measure the actual difference on your actual tasks. I think you’ll find, as I did, that the monster under the bed isn’t nearly as scary as they want you to believe.

Finding alternatives

It’s easy to think that we don’t have alternatives here in Europe, but this is simply not true. There’s a long list of alternatives (Graf 2025) which can be utilized. Both commercial and open source software and solutions are available. Even if you don’t really know what to look for you can use the categories to navigate when looking for “Cloud” replacements for example.

This should be enough for anyone to get started. I cannot say that it doesn’t involve work, but I can say that it’s worth it. Every second of it.

What about hardware?

Hardware often comes up as a problematic area on our way towards independence, and while it’s true that when it comes to mobile phones and laptops we have few optionshere in the EU these options do exist. One example of a european phone maker is Jolla which is a bunch of ex Nokia guys making a cool new phone. Another is HMD which is part of the Better Phone Project and creates european phones.

When it comes to computers there are also alternatives of course. One cool supplier here is Slimbook who produce beautiful computers that come with Linux or Windows preinstalled. Naturally, since this is about European alternatives I would strongly suggest that you choose a Linux based operating system.

What I have done

I have a pretty large setup where I have used a bunch of different technologies but I will list the ones I think most people will want an alternative to.

mindmap
  root((European<br>Software))
    Cloud Providers
        Hetzner
        Contabo
        Nextcloud
    Communication
        Proton Mail
        Mailbox.org
        Thunderbird
    Social Networks
        Signal
        Mostodon.Social
    Office Productivity
        Libreoffice
        Nextcloud
    Photo & Video & Music
        Immich
        Synchthing
        Deezer
        JellyFin
    Browsers
        Firefox
        Zen-Browser
    Maps
        OpenStreetMap
        HERE WeGo
    Search
        Startpage
        Qwant

Conclusion

Switching to European and open-source alternatives is an important move toward two vital goals:

• Building a strong, independent Europe capable of digital self-reliance.
• Peacefully protesting harmful policies promoted by the current US administration.

As the old saying goes:

A friend who bullies us is no longer a friend.

The tools for change are already here. It’s time we used them.

Introduction

Never before have I heard so many people speak of so many things they know so little about.

A colleague said this to me over coffee a while back and it’s been stuck in my head ever since. Not because it’s clever (it is), but because I keep seeing evidence for it everywhere I look.

I open LinkedIn and someone with “AI Enthusiast” in their bio is explaining why Bayesian methods are outdated. I scroll further and a marketing director who discovered ChatGPT six months ago is telling physicists how neural networks work. I check Twitter (sorry, “X”) and an influencer with a million followers is confidently explaining that p-values prove causation. Each of these posts gets thousands of likes, hundreds of shares, and a comment section full of people nodding along.

This is what I mean by cacophony. Not silence, not even misinformation exactly. Just an overwhelming volume of confident noise from people who haven’t done the work to earn that confidence.

The problem isn’t ignorance

Let me be clear about something: I’m not calling people stupid. That’s a lazy take and it’s not what I mean. Most people sharing opinions online genuinely believe what they’re saying. They’ve read an article, watched a YouTube video, maybe skimmed a paper’s abstract. They’ve formed a view. The problem is that forming a view and understanding a topic are wildly different activities, and our current information landscape makes it almost impossible to tell them apart.

I’ve been guilty of this myself. When I first got into drug discovery I had opinions about molecular properties that I’d formed from reading a few papers. It took months of actually working with chemists, running simulations, and getting things wrong before I realized how shallow my initial understanding was. I wrote about LogP and LogD partly to force myself through that process. Writing it down was brutal because it revealed every gap I’d been casually ignoring.

And that’s the thing. The gap between “I’ve heard of this” and “I understand this” is enormous. But on social media it’s invisible. A confident 280-character take looks the same whether it comes from someone who has spent a decade on the problem or someone who read a blog post yesterday.

The internet made it worse

People have always been prone to holding opinions they can’t defend. This is not new. What is new is the scale.

When I was doing my PhD in Lund, if you wanted to spread a bad take about physics, you had to stand up in a seminar and say it out loud in front of people who could immediately challenge you. That’s a pretty effective filter. The social cost of being wrong in a room full of experts is high enough that most people either do the work or keep quiet.

Social media removed that filter entirely. You can broadcast opinions to millions of people without ever facing someone who actually knows the field. The feedback you get isn’t “that’s wrong because…” it’s likes, shares, and algorithmic amplification. The platforms are optimized for engagement, not accuracy. Confident, simple, slightly provocative takes get rewarded. Nuanced, careful, qualified statements get ignored. So the incentive structure actively selects for the kind of noise I’m complaining about.

And then there’s the confirmation bias machine. Whatever you already believe, the internet will find you a community of people who believe it too, and a steady stream of content that reinforces it. Want to believe that transformers are conscious? There’s a subreddit for that. Want to believe that all of statistics is a fraud? You’ll find your people. The information is technically available to correct these views, but it has to compete with an algorithm that knows exactly what will keep you scrolling.

AI is pouring gasoline on this

Here’s where it gets really fun. We now have AI systems that can generate fluent, confident, well-structured text on any topic. A person who knows nothing about Bayesian inference can ask an LLM to write a LinkedIn post about it and get something that sounds authoritative. They post it. People who also don’t know about Bayesian inference read it and think “this person knows their stuff.” The cycle accelerates.

I’m not anti-AI (I build AI systems for a living, that would be a weird position to take). But I do think we need to be honest about what this means for the signal-to-noise ratio. When the cost of producing confident-sounding content drops to zero, the volume of that content explodes. And if most of it is produced by people who don’t understand the topic, or by machines that can’t understand anything at all, the noise gets louder while the signal stays the same.

Writing as an antidote

Ok, enough complaining. What do we actually do about this?

I think the single most powerful thing you can do is write. And I don’t mean tweets or LinkedIn posts (though those have their place). I mean long-form writing where you have to actually develop an argument, anticipate counterpoints, and deal with the uncomfortable realization that you don’t know as much as you thought you did.

I’ve been writing this blog since 2016 and I can tell you from direct experience: nothing reveals the gaps in your understanding faster than trying to explain something to someone else in writing. When I wrote about the equivalence of Bayesian priors and Ridge regression, I thought I had a clear picture going in. Three hours later I was back in the derivations because writing it down had exposed an assumption I’d been making for years without examining it. That’s the value. Not the finished post (though that’s nice too), but the process of getting there.

Writing forces you to be specific. You can’t hide behind vague gestures when you have to put actual words on a page. “AI will change everything” is easy to say. Try writing 2000 words about exactly what will change, for whom, through what mechanisms, and with what evidence. Suddenly the confident take doesn’t feel so confident anymore.

Cultivating honest skepticism

The other thing I’d advocate for is what I’ll call honest skepticism. Not cynicism (that’s just laziness wearing a trench coat), but a genuine willingness to ask “how do you know that?” of everything, including your own beliefs.

When someone tells you something, especially something that confirms what you already think, ask yourself: what would have to be true for this to be wrong? What evidence would change my mind? If you can’t answer those questions, you don’t have a belief. You have a feeling. Feelings are fine for picking a restaurant. They’re not fine for making claims about how the world works.

This applies to experts too, by the way. I have a PhD in theoretical physics and I’m wrong about things regularly. The difference (I hope) is that I try to notice when I’m wrong and update accordingly. That’s what being a scientist actually means. Not having the right answers, but having a reliable process for getting closer to them.

Conclusion

I don’t have a tidy solution for the cacophony. I don’t think there is one. The platforms are the way they are, the incentives are the way they are, and AI is only going to make the content flood worse.

But I do think we each have a choice about how we participate. We can add to the noise or we can resist it. We can share things we haven’t read properly or we can take the time to understand before we speak. We can optimize for likes or we can optimize for truth.

Write things down. Subject your ideas to the test of putting them into words. Read things written by people who actually did the work, not by people who are good at sounding like they did. And when you catch yourself about to share a confident opinion on something you learned about twenty minutes ago, maybe pause for a second. The world has enough noise.

The calculator didn’t replace mathematicians

Let me tell you what I’ve actually seen happen. I’ve spent 25 years building neural networks. I’ve watched the field go from niche academic curiosity to front-page news to dinner table conversation. And in all that time, the people who thrived weren’t the ones who learned to use the latest tool. They were the ones who understood why the tool worked, where it broke, and what to do when it broke.

This isn’t unique to AI. When calculators showed up, nobody stopped teaching mathematics. When compilers showed up, nobody stopped teaching computer science. The tools changed the baseline of what you could accomplish, but they didn’t remove the need to understand the foundations. They raised the floor. They didn’t lower the ceiling.

AI is a tool. A powerful one, sure. Possibly the most powerful one we’ve ever built. But it is still a tool. And the thing about tools is that they’re only as good as the person wielding them. Give a chainsaw to someone who doesn’t understand wood and you’ll get firewood at best, a trip to the hospital at worst.

What I’ve learned from martial arts

I taught martial arts for over a decade. One of the things you learn very quickly as an instructor is that there’s no shortcut to skill. You can watch a thousand YouTube videos of a technique, you can have someone explain the biomechanics in perfect detail, but until your body has done the movement a few thousand times, you don’t own it. It’s not yours. Under pressure, you will revert to whatever your body actually knows, not what your head thinks it knows.

The same applies to intellectual skills. You can ask an LLM to write code for you, and it’ll produce something that looks right. But when it breaks (and it will break), you need to understand why. When the output is subtly wrong (and it often is), you need to catch it. When the problem doesn’t fit neatly into what the model has seen before (and real problems rarely do), you need to improvise. And you can’t improvise with knowledge you don’t have.

I’ve watched junior developers paste LLM output into production without understanding what it does. It works for a while. Then it doesn’t, and they have no idea where to start debugging. They skipped the part where you learn the thing, and now they’re stuck. The model gave them a fish, but nobody taught them to fish.

The skills that matter

Ok so if AI handles the routine stuff, what’s left? Quite a lot, actually.

The most valuable people I’ve worked with over the years share a common trait: they can think across domains. They see connections that specialists miss. A physicist who understands finance. A programmer who understands biology. A martial artist who understands leadership. These intersections are where the interesting work happens, and they require deep skill in multiple areas, not surface-level prompting ability in one.

Here’s what I think actually matters:

Understanding the fundamentals. Not because you’ll manually compute gradients every day (though I do enjoy it), but because when your model behaves unexpectedly, the person who understands backpropagation will diagnose the problem in minutes while the person who only knows the API will spend days guessing.

Critical thinking. AI is confident. Impressively, fluently, dangerously confident. It will give you wrong answers with the same tone and structure as right ones. If you can’t evaluate the output independently, you’re not using a tool. You’re being used by one.

Building things from scratch. The ability to start from a blank page and create something that didn’t exist before. This requires deep understanding, creativity, and the willingness to sit with discomfort while you figure it out. LLMs can help you iterate faster, but they can’t replace the generative spark.

Physical and embodied skills. This one might sound odd coming from someone who spends most of his day at a computer, but I genuinely believe it. The discipline you develop through martial arts, through sport, through any practice that demands full-body engagement, transfers directly to how you think and work. Patience, persistence, attention to detail, the ability to stay calm under pressure. These aren’t soft skills. They’re the foundation everything else is built on.

The electricity analogy (but for real this time)

People love comparing AI to electricity. Usually in a breathless “this will change everything” kind of way. But the analogy is actually useful if you take it seriously.

When electricity became widespread, it didn’t make human labor obsolete. It made new kinds of labor possible. It created entirely new industries, new professions, new ways of thinking about problems. But the people who benefited most weren’t the ones who just plugged things in. They were the electrical engineers, the inventors, the people who understood the technology deeply enough to push it into places nobody had imagined.

AI will follow the same pattern. The people who just “use” it will do fine. The people who understand it, who can build on it, extend it, and critically evaluate it, will do dramatically better. And the people who gave up learning because “AI will handle it” will find themselves dependent on systems they can’t control, can’t inspect, and can’t fix.

Conclusion

I’m not arguing against using AI. I use it every day. I’m building companies around it. What I am arguing against is the idea that AI makes learning unnecessary. That’s a story being told by people who benefit from you being a passive consumer of their product.

The truth is simpler and less comfortable: the skills that matter most are the ones that are hardest to acquire. They require time, effort, frustration, and the willingness to be bad at something before you’re good at it. No shortcut has ever changed that, and AI won’t either.

So learn the thing. Do the hard work. Build the muscle memory, whether it’s physical or intellectual. The people who do will be the ones shaping the future. The people who don’t will be along for the ride, hoping the driver knows where they’re going.

Introduction

As outlined in Sensoy et al. (2018) there’s some work done in Evidential Deep Learning to quantify uncertainty in classification applications. The evidential framework depends on the ability to specify conjugate priors for the parameters of the likelihood of interest. In classification tasks we naturally lean on the Categorical distribution in the general case and the Bernoulli distribution in the binary classification case. The Bernoulli and the Categorical distribution are both useful for describing an individual sample of a class outcome, i.e., true or false for a given class. But what if your data are not individual classification samples but rather aggregations of the number of times a given class came up in several samples? Let’s take the example of a cancer classification problem. In the individual classification example we would get a true (for cancer) or false (for not cancer) per individual we test. If we have have a dataset showing how many patients had the diagnosis cancer vs not cancer per region then we would have a Binomial distribution that would best describe this data. The same logic would hold true for the Categorical distribution which would then map to the multinomial distribution.

A quick primer on Bayesian probability

There are of course terminology in Bayesian modeling that’s necessary to know and be able to relate to. I will follow Tu (2014) and outline the most important ones to know.

The likelihood is a measure of how well a chosen distribution and it’s given parameters explain the observed data. It is strictly speaking not a probability since it usually operates of several samples from a given distribution. As such, it is not normalized.

The prior is the assumed distribution you put on your parameters before observing the data. It can be interpreted as a kind of regularization of your model to make sure it doesn’t do anything crazy.

The posterior is the distribution of your parameters after observing the data. Thus, it’s a distribution over your parameters where your assumptions and data has been weighted together to form a coherent opinion about the problem.

The evidence or marginal is the probability of observing the data after all parameters have been integrated out. This is important as it is arguable what we would like to maximize for a given problem. Note that this is distinct from the likelihood which doesn’t factor the prior assumptions in.

Mathematically we can put these items into one glorious equation

where I have chosen a parameterization that matches the one we will use for the Multinomial distribution. Usually you would have written this with instead of .

Now, there’s one more I would like to mention which is also an integration exercise where we instead are interested in the marginal predictive distribution of a new datapoint . It’s called marginal predictive. In this setting we will use as representing the entire dataset of .

In order to really seal the deal here I want to be explicit about the marginal distribution which is over an entire dataset. We will use the same formalism as in the above equation. In this case we can state the marginal data distribution as

where you can see the inclusion of each datapoint in the dataset .

The Multinomial distribution

In order to describe the multinomial distribution for a random variable we need the total number of cases as well as the probability of each class to occur . In this case and are vectors of elements representing the classes.

This is the probability mass function for the Multinomial distribution. The gamma functions you see in the equations are just better ways of expressing the numbers to the classes. Just to get a feel for what is happening it’s worth remembering that .

The Multinomial distribution is a generalized form () of the Binomial () distribution. Conversely, it’s also a generalization () of the Categorical () distribution. Table 1 shows a summary of the different distributions mentioned here.

Conjugate priors

There’s a lot of research put into the conjugate distributions for the exponential family which in my opinion is really good. In fact I would argue we still need a lot more.. Anyway, the point here is that if you looked at the equations above you might wonder where the heck the prior distribution comes from? You’d be forgiven for wondering. This can in principle be any distribution you like. As long as you can write it down. Sounds great! The problem is that it’s very often expensive to compute as the integrals we showed in the posterior and the marginals are quite hard to solve and most of the choices do not have a closed form solution. This is where conjugate priors comes into the pictures. Conjugate priors are priors matched with a likelihood which has a closed form expression for the resulting posterior. This is a big thing since we won’t have to rely on MCMC techniques to do the integrations.

Ok, but so what, what is the darn prior distribution that we want to use? Well, in the case of the Multinomial likelihood the conjugate prior is the Dirichlet distribution. In our notation above we will parameterize this distribution with . Thus the prior distribution for our parameters of the Multinomial distribution will be . It’s written out explicitely as

where

and .

The is the multivariate beta function (Wikipedia contributors 2024a) which you can read about in Wikipedia contributors (2024b). A rather convenient yet annoying fact is that the Dirichlet distribution is the conjugate prior for both the Categorical distribution as well as the Multinomial.

The closed form solutions we’ve been waiting for

With the Bayesian primer and our Multinomial distribution (Equation 4) out of the way we’re now ready to have a look at the model that this whole post is motivated by, namely the “Dirichlet-Multinomial”. It’s a model for several samples from a Categorical distribution with a prior on the class probabilities.

Since we know from the section on conjugate priors that this model indeed is conjugate we can get explicit analytical expressions for the posterior (Equation 1) and marginal (Equation 3). The posterior distribution is also Dirichlet.

where , such that and is the number of observed instances of class in sample . Thus, the posterior distribution is formed from the prior pseudocount and the observed counts from dataset .

With this it’s time to move on to the marginal. It’s where we integrate the parameters out. Below you can see the procedure for a single datapoint .

Again, like before. It’s important to note here that there’s no conditioning on the data going on here. This is because the likelihood only enters through the one here. This is the marginal data distribution. The posterior predictive distribution is achieved by utilizing the conditioned instead of .

This is crazy useful as this compound distribution allows us to model overdispersed data that the original Multinomial distribution would have a hard time capturing.

How to apply with Neural Networks

Now, none of this really matters unless we bring some conditioning in to deliver the . This is what we want a model to do. To look at input data and predict the that can be fed into the Dirichlet-Multinomial to explain the data. In this section I will be using Julia (Bezanson et al. 2017) which in my opinion is a nice language for playing around with these concepts.

First I’ll define some dummy data inspired from Guinney et al. (2015) where they define four main consensus molecular subtypes (CMSs) of Colorectal cancer.

• CMS1 (microsatellite instability immune, 14%), hypermutated, microsatellite unstable and strong immune activation.
• CMS2 (canonical, 37%), epithelial, marked WNT and MYC signaling activation.
• CMS3 (metabolic, 13%), epithelial and evident metabolic dysregulation.
• CMS4 (mesenchymal, 23%), prominent transforming growth factor–β activation, stromal invasion and angiogenesis.

I will generate a fake dataset for modeling these four types as a Multinomial distribution.

using DataFrames
using Tables
using Markdown
using MarkdownTables

df = DataFrame(Gender=[0,1], Income=[1,2], CMS1=[4,5], 
           CMS2=[10,15], CMS3=[3,7], CMS4=[15,11])

markdown_table(df)

With this small dataset from Table 2 we can start to define a model for this. We have 4 classes so we need 4 outputs. I’ll show a small piece of code below which implements a stupid overparameterized neural network for this problem. Don’t judge me. It’s for illustration purposes. But I’m also getting ahead of myself, we need to implement the marginal data distribution (which we want to maximize). Since this is indeed just the compound DirichletMultinomial from above we can implement that probability mass function and take the negative logarithm to get a kind of type II likelihood that we want to maximize (or well who’s negative likelihood to minimize).

using SpecialFunctions

function dmlogpmf(x, α)
    α₀ = sum(α, dims=1)
    n = sum(x, dims=1)
    # First term
    ft = loggamma.(α₀) + loggamma.(n .+ 1) - loggamma.(n + α₀)
    # Second term
    st = sum(loggamma.(α .+ x) - (loggamma.(α) + loggamma.(x .+ 1)), dims = 1)
    ft + st
end

dmnegloglik(x, α) = sum(-dmlogpmf(x, α))

dmnegloglik (generic function with 1 method)

Next we’ll define the neural network and the training of it.

using Flux
using Zygote

X = permutedims(Float32.(Matrix(df[:,1:2])))
Y = permutedims(Float32.(Matrix(df[:,3:end])))
n = sum(Y, dims=1)[1,:]

model = Chain(Dense(2, 100, relu), Dense(100, 4, softplus))
opt = Flux.setup(Adam(0.001), model)
println("Loss Before: $(loss(Y,model(X)))")
for epoch in 1:400
    val, grads = Flux.withgradient(model) do m
        𝛂 = m(X)
        dmnegloglik(Y, 𝛂)
    end
    Flux.update!(opt, model, grads[1])
end
println("Loss After: $(loss(Y,model(X)))")

Loss Before: 19.188679
Loss After: 12.59213

Let’s unpack this a bit. First we made sure that we put the training data in a format that Flux prefers. Then we define a loss function based on the compound distribution we got due to conjugacy. That’s the DirichletMultinomial distribution. We want a neural network that produces the parameters of this distribution. We then evaluate the negative log likelihood for the observed counts we have and update the network until it matches the data better.

As you can see the loss decreased so the network indeed got trained. I know, big surprise right? Appears that 704 parameters can help when you have 2 datapoints. 😄

Now let’s see a more understandable performance metric than likelihood. For this type of output I would just use the standard MAPE per class. We’re also interested in the epistemic uncertainty for each datapoint. This represents the models uncertainty about it’s own prediction. In order to do this we define a predict and an uncert function to use for predictions and estimating epistemic uncertainty.

function predict(m, X, n)
    𝛂 = m(X)
    α₀ = sum(𝛂, dims=1)
    p = round.(𝛂 ./ α₀ .* n', digits=0)
    return p
end

function uncert(m, X)
    𝛂 = m(X)
    α₀ = sum(𝛂, dims=1)
    u = first(size(𝛂)) ./ α₀
    return u[1,:]
end

mape(p, o) = mean(abs.(p - o) ./ o, dims=2)[:,1]

mape (generic function with 1 method)

With these functions we can now see how well our model fits the data. Since we have such a simple and nice dataset we can just print it out and inspect what the predicted numbers look like in Table 3. Comparing this to Table 2 it’s clear to see that the predicted numbers are in the vicinity of the observed ones. It’s also not obvious to see which data point that’s better or worse predicted. However, due to our ability to estimate epistemic uncertainty we can now have a computational view on uncertainty. By inspecting the uncertainty column we can see that the model is a lot more confident regarding the second datapoint.

Now we might also ask with the classwise performance is, i.e., how well are the counts in each class predicted? Here, we’ll turn to MAPE and use the function from above. The numbers are in Table 4 below.

Conclusion

So we’ve seen that using a conjugate prior distribution allows us to get an explicit solution for our posterior probability as well as the marginal data distribution. We’ve also seen how the resulting compound distribution can be used as a loss function through the negative log likelihood. This lends itself nicely to using arbitrarily complex models like neural networks to predict the parameters of the posterior distribution.

As a bonus we can readily estimate the epistemic uncertainty of our model, which in the case of cancer prediction can be pretty darn important.

Introduction

There’s a new candidate for checking for relationships between two variables out on the streets. It’s been there since 2020, but I actually just heard about it now. It’s marketed as a drop in replacement for the Pearson correlation coefficient (Pearson 1895). The name is ξ correlation and was introduced in Chatterjee (2021). In this post I’ll run through the strenghts and weaknesses of this new correlation coefficient from my point of view. Finally, as the title suggests, I’ll spend some time looking into how ξ correlation can be used for causal discovery. Though I will not touch upon it in this post you can look at Additive Noise Models (Kap et al. 2021) which is the standard workhorse in the field of causal discovery for more context.

This is also a follow up on Sumner (2024) introduction to the new metric. I recommend reading it before reading mine as he goes into more depth and intuition behind the math of the ξ correlation.

Some background

The Pearson correlation coefficient has been and continues to be a versetile tool in every data scientists analytical toolbox. It is primarily used to indicate relationships between variables of interest. Its strength lies in its simple equation and its cheap computational cost.

In this equation refers to the expectation operator, σ is the standard deviation of the respective variable and μ is the average of each variable.

To visualize this let’s say you have two vectors and , representing two samples from variable x and y.

(X = [-1.0, 1.0, 0.5], Y = [1.0, -1.0, 0.9])

In Figure 1 you can see a scatterplot of x vs y. Since we only have three data points in each variable it won’t be a very impressive scatterplot, but you’ll see in a minute why I chose only three.

In the illustration above you can see three plots. The first plot is a scatterplot of y against x. The second plot is a vector space plot where Dimension 1 corresponds to the first element of x and y respectively while dimension 2 refers to the second element. A good way to think of this would be that each new sample from the variable is a new dimension. So three samples, as we have here, equals three dimensions. The last plot is also a vector space but where we have subtracted the average of x and y respecively.

Now what is the correlation between x and y? Let’s apply the math from above and do the calculation. We end up with -0.72 which shows a negative correlation. Ok, but what does this have to do with the dimensions you say? Well it turns out that there’s an inherent link between the correlation coefficient and the angle between the vectors.

The correlation coefficient is equivalent to the dot product of the centralized vector x and y divided by their norms.

We can express this mathematically as

where and is the angle between the vectors and . This gives us a nice geometrical interpretation of the correlation coefficient, which I personally find extremely satisfying and simple to think about. It also makes intuitive sense; Imagine that the two vectors were pointing in the same direction such that the angle is close to 0. This would mean that which is also what the correlation coefficient would be since the only way the vectors can point in the same direction is if the numbers are moving in similar ways. This is irrespective of any offset between the numbers.

ξ correlation

Needless to say, correlation is extremely powerful in data analysis since we can readily asses the linear relationship between any two variables. I emphasize linear because any relationship between the variables consisting of a non-linear function will be poorly represented by the correlation coefficient.

If you take a look at Figure 2 you will quickly agree with me that these variables x and y in the scatterplot have a relationship. In fact they are all simple mathematical relationships. But they all have an effective correlation coefficient of 0. This means that scanning for relationships between variables with correlation would miss a lot of non-linear relationships. Thus, for all the wonderful abilities we get out of the correlation coefficient it still fails to capture relationships as simple and common as which is of course an issue when trying to discover unknown relationships. This is one of the reasons for a lot of research being conducted in the topic of causal discovery and independence tests between random variables. Josse and Holmes (2016) provides a recent and thourough overview of the later attempts.

Chatterjee (2021) introduced ξₙ a new rank based correlation test which has nice properties if the samples of the two continuous variables , and are i.i.d.

where .

So it would be natural to ask; If all the plots in Figure 2 have a correlation of 0. Then what would the ξ correlation yield? In this post I follow the implementation of Taelman (2023). The function looks like this in julia code.

using Distributions, StatsBase

function xicor(X::AbstractVector, Y::AbstractVector; rank=denserank, noties=false)
    n = length(Y)
    xsortperm = sortperm(X)
    r = rank(xsortperm, by=i->Y[i])
    if !noties
        l = rank(xsortperm, by=i->Y[i], rev=true)
    end
    ξn = 0.0
    for i in 1:n-1
        ξn += abs(r[i+1] - r[i])
    end
    ξn *= noties ? 3 / (n^2 - 1) : n / 2sum(li->li*(n-li), l)
    ξn = 1 - ξn
    return ξn
end

xicor (generic function with 1 method)

If you’re not that into julia you can easily port this code to R, python or whatever language you prefer. Back to our question. What is the ξ correlation compared to the person correlation using our examples in Figure 2? In Table 1 we show the ξ, Kendall, Spearman and Pearson correlation coefficient for the 9 scenarios. It’s immediately clear that the ξ correlation is much more informative, and correct, regarding the relationship between x and y in all examples except the Unit Circle and the Four Gaussians. It’s also strictly different from 0 in the Unit circle example but this has a problem that we will get back to a little later. For now let’s focus on the expected results.

[ Info: Precompiling MarkdownTables [1862ce21-31c7-451e-824c-f20fa3f90fa2] (cache misses: incompatible header (2))



SYSTEM: caught exception of type :MethodError while trying to print a failed Task notice; giving up

Correlation plots

When it comes to early exploratory data analysis scatterplots are extremely useful. Often they also come in flavors of what is known as pairplots, where you can see several variables scattered against each other at the same time. It’s common practice to also print the correlation coefficient for each of these scatterplots. We use the PIMA dataset from Venables and Ripley (2002) to illustrate a common pairplot using Kendall correlation in Figure 3. As you can see we only show the bottom left of the diagonal of the matrix of scatterplots and that’s because correlation in general is symmetric so . Thus, there is no information in the top half which is not already shown in the bottom half.

This is of course no longer true for ξ correlation. Therefor we actually want to show both halves in a correlation plot when using ξₙ correlation coefficient to evaluate the potential relationship between two variables. In Figure 4 you can see especially the two variables Ped and Age where the ξₙ coefficient is in one direction and effectively in the other.

Conclusion

I have introduced the new correlation coefficient ξₙ and showed that it indeed captures a wider range of relationships than the traditional correlation coefficients by Spearman, Kendall and Pearson. I have also illustrated some issues unique to this new metric which are not present in the traditional coefficients. So while ξₙ is definitely useful it’s not a silver bullet by any means when looking for relationships between variables. Also we clearly saw that you need to calculate both directions, i.e., you need to think about and before concluding anything about the potential relationship between the two variables X and Y.

A Quick Primer on Acidity

As most of you probably remember from chemistry in high school, knowing the acidity of a solution is quite useful. Or well, at least we were forced to measure it often enough.. Anyway, to quantify this, the metric pH was introduced by a Danish chemist named Søren Peder Lauritz, who worked at the Carlsberg Laboratory in Copenhagen at the time. It measures the acidity or alkalinity of a solution. It’s essentially a scale we use to quantify just how acidic or basic (alkaline) a water-based solution is. You may hove noticed that we snuck two terms in for the same thing, namely Alkaline and Basic. I will refer to them as equal for the remainder of the post.

The pH scale ranges from 0 to 14:

• Acidic Solutions: These have a pH less than 7. The lower the pH, the more acidic the solution. For example, lemon juice and vinegar are acidic, with pH values typically around 2 to 3.
• Neutral Solution: A pH of exactly 7 indicates a neutral solution. Pure water is a classic example of a neutral solution.
• Basic Solutions: These have a pH greater than 7. The higher the pH, the more basic the solution. For instance, baking soda in water is basic, with a pH around 8 to 9.

Ok so now you know the interpretation of the pH value but we still haven’t defined what it is. So let me do that right now.

Here you can see that pH is defined on a log scale of the concentration of hydrogen ions () in the solution where the concentration is given in moles per liter. One mole (Mole (Unit) - Wikipedia — En.wikipedia.org 2023) of any substance contains exactly entities (atoms, molecules, ions, etc.). This number is what the scientific community refers to as Avogadro’s number. You might wonder why the “-” sign in front of the logarithm? Well, it’s simply because we want the pH expressed in positive numbers. The logarithm of a number less than one, which the concentration of anything in a solution always is, is always negative. So the “-” sign converts that to a positive number instead. Ok dude, I hear your enquiring minds cry. Why did you say that the pH is always between 0 and 14 when it’s clear from your definition that we in principle can get a number much higher than 14? Well, it’s a technicality. While mathematically correct, in practice, we never see values outside of the 0–14 range. This is because, in order to reach pH levels below 0 or above 14, one would require extraordinarily acidic or basic solutions, respectively (Lunawat 2023). An example of the pH scale can be seen in Figure 1 where everything from a really basic solution to a pure acid is illustrated.

Ok fine, but so what? Quite a bit actually. pH is important in many areas of science but when it comes to drug discovery and development, for instance, the pH of different parts of the body can affect the absorption and efficacy of drugs. That’s why it’s crucial to know how our molecules react in different pH regimes.

My last point about the pH metric is that since it’s a logarithm of base 10 it means that a pH of 4 is 10 times more basic than a pH of 3 which in turn is 100 times more acidic than a pH of 5.

Ready, Set, Get to the Point!

Armed with the primer on acidity we can start diving into the metrics I started this whole blog post about. Let’s begin with talking a bit about what we want to know about potential molecules that might be useful to distinguish possible drug candidates from other molecules. One important aspect of molecules is how much they like water (hydrophilic) or fat (lipophilic). This matters because a potential drug needs to be able to pass lipid membranes and it cannot very well do that if it prefers the water outside.

LogD: Because pH Just Ain’t the Same

As I might have hinted at above in my little pH primer, pH is of some significance to drug discovery. Mainly because pH is not the same in all parts of our body. In fact it can differ quite wildly. We have a pH of around in our blood while our intenstines are more acidic with a pH between and . Our stomach sports a whoppin pH between to making it a strongly acidic environment. See Figure 1 for examples of the pH scale again.

Ok, so different areas of the body have rather different environments as highlighted by the different levels of pH. But why does this matter to us when developing drugs? Well, because a molecule is not only one molecule. Molecules are susceptible to ionization, a process that adds/removes a hydrogen to/from a functional group. In Figure 2 we can see a molecule in three different states. It features two ionizable groups. Which can be ionized individually and simultaneously depending of the pH of the solvent. All more than one ionization state of the molecule can exist in different concentrations depending on the pH of the solvent.

As it is, these ionizations also matter for the bioavailability which means that pH can directly affect the properties a given molecule has. That’s why it’s a good idea to include the pH level of the part of the body where we want the molecule to be soluble. I know I’m skipping a lot of important details here and am being a bit handwavy but I hope you’ll forgive me. So with apologies prepared let’s move on to see how we can fix the shortcomings of our LogP metric which only measures the concentration of the unionized form of the molecule.

First let’s go back to the definition of LogP which was as given below.

Thus LogP is just the logarithm of the partition coefficient . So as this only looks at the unionized form of the molecule we would like to extend the equation to also take into account the concentration of it’s ionized form in water. This can be accomplished like so.

Here is known as the dissociation coefficient. This also the reason why LogD is said to take pH into account since the amount of the molecule that is ionized depends heavily on the pH of the solvent. In Figure 3 I’m illustrating how LogD varies for Piroxicam (Multum 2022) at different levels of pH of the solvent.

Here you can see something quite typical which is that an acidic drug has the highest lipophilicity (high LogD) when the environment is acidic as well. In our case Piroxicam is indeed an acidic drug which is ionized at higher pH values leading to a lower liphophilicity (lower LogD).

There is a nice theoretical relationship between LogD, LogP and pKa. The method, known as CALlogD, can be derived from the predicted LogP and pKa values (Wang et al. 2023). The equation is

where for acids and bases, respectively. The astute reader will notice that only a single pKa is provided here which is the global pKa. In Figure 4 I show how the LogD varies with pH for two acidic and two basic drugs.

pKa: All Acids Are Not Equally Strong

Now let’s turn the attention to pKa which connects, in a way, LogP, LogD and pH. The metric pKa tells us how strong an acid is. The lower the value the stronger the acid. The pKa value tells us something about how easily an acid gives away a hydrogen ion. We define pKa like the equation below.

I guess you’re starting to see the trend with the logarithms entering all of our metrics. But, I digress. Now, the is called the acid dissociation constant which measures how easily an acid can dissociate in water. Then we just take the negative base 10 logarithm of that. That’s pKa for you! As you can see in the equation above it’s also related to the pH of the solvent. This relationship is known as the The Henderson-Hasselbalch equation. So in plain language the pKa is the negative base 10 logarithm of the concentration ratio between the acid () and its conjugate base (). Extending this reasoning you can also see that pKa is equal to pH when precisely half of the acid has dissociated. This follows from the fact that half of the acid dissociating means . This means that we can interprete pKa as the pH at which half of the acid has dissociated.

I realized I skipped something again as I was reading what I just wrote. The term “conjugate base” just found it’s way into the mix. Let me clarify what I mean by that by quoting PH and pKa — Shiken.ai (2024).

Conjugate acids are bases that gained a proton . Conversely, Conjugate bases are acids that lost a proton .

What this means is that two compounds which differs only by one are said to form a conjugate acid and base pair. I think the illustration in Figure 5 shows this relationship nicely.

One important aspect of pKa that eluded me for the longest of time was that it is intrinsically a local metric. What I mean by that is that it is a given functional group of the molecule which has a pKa value. If the pKa of a functional group is less than 7 it means that it will be positively charged at thus being acidic. A pKa larger than 7 means it would be negatively charged at which means being more basic.

So how do we actually use the pKa value in our daily work you say? I said in the beginning of this section that the lower the pKa the stronger the acid. This also means that the higher the pKa the weaker the acid. That makes sense. But what is also important to remember is that pKa is also useful for evaluating the strength of bases. Wait what? Yeah indeed but a common trap here is to just invert the meaning pKa holds for acids. Unfortunately it’s not that simple. Almost, but not quite. The interpretation is the higher the pKa the stronger the conjugate base! Ashenhurst (2010) gives a nice explanation of this.

There is a small point to be made about bases as well. The pKa equation above is coming from the Henderson–Hasselbalch equation (Henderson 1908a, 1908b). It’s basically just a rearranged version of it. The original form is

and can be used to assess the pH of a solution of a weak acid in equilibrium. The pKa is the negative logarithm of the acid dissociation constant . It models the chemical reaction where an acid HA dissociates into a conjugate base and a hydrogen ion . The reaction is given below.

But what about bases? Well, for them the same ingredients for the equation is used but we call it the association constant instead.

In the above equation you can see that is really just the inverse of . This way we can redo the derivation above but for pKb instead.

It’s important to remember that for any base or acid in water the following relationship holds. The unknown here is the self-ionization constant for water , which in many cases can be put to since that’s value takes at C in water.

Note

In drug discovery we rarely use but rather express the of the conjugate acid instead.

LogS: Water solubility is key

So far I talked a bit about why we need to know how strong an acid or a base is and that we need to know the lipophilicity of a molecule. This is all well and good but we also need to know how soluable a given molecule is in water. This is where LogS comes in. LogS is as the logarithm of the solubility of a molecule measured in mol/L. Now to complicate matters a bit there are two kinds of LogS that you need to know about. The first is “Intrinsic solubility” which is the solubility measured after equilibrium between the dissolved and solid state at a pH where the compound is neutral (DrugBank 2023). The second is pH-dependent solubility which refers to the solvation equilibrium being affected by the pH of the solution.

Why does LogS matter? Simply because a low solubility goes hand in hand with bad absorption and is also indicative of distribution problems. Therefor most drug hunters avoid molecules with low LogS scores. More than 80% of all drugs in the market have a (OrganicChemistryPortal). In Figure 7 you can see the LogS vs. pH for five different ionizable drugs. Solubility is the lowest at a pH of around 4. The highest solubility is achieved around pH 7. The species are colored in the original data. Please check out Shoghi et al. (2013) for a nicer visual. I was too lazy to create all the series. Apologies.

The actual drug in this data is Sulfadimethoxine (DrugBank 2007) which approval in the US was revoked. It’s unionized form happens at pH of around 4 which means it’s non-polar which leads to the poor solubility in water. Making the solvent more acidic creates a positively charged ion which increases solubility in water. Conversely, making the solvent more basic creates a negatively charged ion which in turn also increases solubility.

Wrapping Up

Well, that was a lot to take in. At least for me. I hope you have gotten a better understading of LogP, LogD, pKa and LogS and how they are affecting modern drug discovery. I will be the first to admit there’s still a lot more to understand, but this should be a good starting point for diving further into these metrics.

I would like to end this post with a quick cheat sheet in Table 1 for you regarding the 4 properties I have been talking about. But before checking that out I would like to draw your attention to the importance of uncerstanding the uncertainty of all measurements you might do on molecules and proteins. There are very few precise values in the microscopic world and everything is constantly moving. As such any measurement we take will have a significant error attached to it. The size of this error is important to quantify. Especially when we start to train AI models on these type of data. Don’t hesitate to reach out if I have made any mistakes or blunders in my explanation. So whenever you see someone claim that a drug has a certain effect, like killing cancer cells in a petri dish, remember that the problem is never killing the cancer cells. It’s avoiding to kill everything else in the process.

With no further ado, here’s the table.

Happy hacking!

This post is also published at Medium.

Starting the modeling exercise

In the example outlined we’re modeling the number of companies who made it to Unicorns, and we gathered a list of companies and checked how many of them are unicorns. In this case we got 3 unicorns and 7 “normal” companies. Let’s set up a model for this now. We will denote the number unicorns by where N is the number of companies in the survey and is the unknown probability of a company becoming a unicorn. The prior is given by the beta distribution . This particular setup has a closed form solution which allows us to directly infer the posterior distribution of our parameter .

Before inferring the posterior we need to choose values for our prior. In this case, and . There are a number of ways to do this, and depending on who you are and what you believe they will differ. Believe me when I say that this is a good thing. I’ll begin by showing you what would happen if we claim complete ignorence which would correspond to a flat prior which in Beta distribution language is , . Our prior, likelihood and posterior look like this.

Ehhh, wait a minute.. Where’s the posterior and likelihood? Well, remember that we said we were completely ignorant right? Right, so that would correspond to trusting the data blindly and just accepting that we have no knowledge to add. So we end up with the following results

which states unsurprisingly that the probability for a company to become a unicorn is 33%. Well, that’s total BS. We trusted the data without accounting for all the biases that happens to us when we collect or think about data. In this very example here we only have 10 companies and we just picked companies we’ve heard of many times which increases the possibility of them being unicorns.

A better prior gives a better model

Let’s try again but this time let’s pretend that we’re not vegetables and actually have some idea about how many companies actaully make it to unicorn level. I would say in general that chances are slim so I will choose a prior distribution that reflects that belief. In this case I’m going with , . We can see the immediate results below in the plot and the table.

Now it’s abundantly clear that I don’t believe it’s very probable for a given company to become a unicorn. In fact we’re shy of a posterior average change of 20%.

As a small eye-candy I’ve animated the process where we change the prior assumption from uninformed to informed and how the data interacts with it to create the posterior.

Summary

In this post we went through a few scenarios and consequences of choices for our model. It’s important to note that the process here is not something I would recommend. The prior should never be adapted to fit your data. It should be a reflection of your knowledge. I basically just wanted to show you what happens when you don’t add your knowledge to your models, as well as highlighting the dangers of maximum likelihood.

Credits

Ideas for the plots and the Beta binomial model was shamelessly stolen from The Bayes rules! book. Check it out!

The SEIR model

When dealing with epidemics and infectuous diseases in general where the time from exposure to infection is significant a 4 compartment model called the SEIR (Harko et al. 2014) model is often used. It’s an acronym that stands for Susceptible, Exposed, Infected and Recovered. Susceptible refers to the stock of the population which have the risk on being infected by the disease, i.e., people that are alive and not immune. The Exposed refers to the part of the population that have been infected, but that are not yet showing symptoms. The period from Exposed to Infected is called incubation time. The Infected is exactly what it sounds like; people that have been infected and that are showing symptoms. The last one is Recovered which refers to all subjects that either got a full recovery or died.

The model takes the shape of a Differential Equation since it models how people move between the four compartments. Mathematically we can express the model with these four equations.

The parameters and their interpretation is given in the table below. There are some subleties associated with these parameters but at least this will give you a feel for them.

Parameter name	Parameter interpretation
	This is the birth rate which is equal to the death rate in this model
	This parameter measures the conversion ratio from susceptible to exposed
	This parameter governs the incubation period
	This parameter governs the duration of the infection

So what are reasobale values for these parameters? Well, to do that we need to use our common sense a bit and do some estimations. But either way it’s all good fun. Now, let’s start with . How can we build an intuition about this? To start with we can realize that only infected people can infect others. So we need to know how effectively they can do that. Basically it will need to consist of at least two factors: (i) the probability of actually transferring the virus when they meet another person and (ii) the number of people they come into contact with on average each day. Thus we will have which will make it easier to reason about. Say an average person runs into 20 persons per day and the probability of transferring the virus is 2% we reach a . Correspondingly, if the chance of transmitting the virus is 4% we end up with .

For we need to think about incubation periods. If the incubation period is around 2–14 days, as we know it is for Covid-19 (Li et al. 2020), we have quite a spread. We can model this incubation period with an exponential distribution . This would mean that the average follows from the properties of the Exponential distribution. You can see what the incubation time distribution looks like for in the plot below.

The next parameter we’ll have a look at is the which is related to the duration of the infection. Specifically it’s the rate with which infected people recover. We’ll also build an intuition for this using number of days . Say that an average infection lasts 10 days then an individual recover once every 10 days, if we assume they can be infected again, and this brings us to . According to WHO (Organization et al. 2020) it takes between 2 to 8 weeks from symptom onset to recover, for those who died (Yes I know it sounds funny, but recovery in this model refers to death as well as actual recovery). Mild cases typically recover within 14 days. The plot in this case would look like this for a distribution

We can look at the joint distribution of and to see how much longer the infection duration is than the incubation period. We can allow ourselves to do this since we will boldly assume that .

The last parameter of the model is actually the least important and least interesting one, namely the birth rate and death rate. These numbers are given by the Danish Statistical Centre. It’s simply where 365 is needed to put it into daily levels.

So in summary now we have the following parameters to use as a starting point; but we’ll play with them more later.

There’s a number that is always quoted when dealing with epidemics and that’s the “Basic Reproduction Number” which measures how many people an infected individual will infect in turn. Mathematically it is given by this relation.

As you can see it only uses the parameters of the ODEs which means that we don’t have to solve the equations to get to . In our parameter setting referred to above we would have an . Just FYI, you really want this number to be below or the infection will most likely continue to spread. A bit of algebra will show you that in order to maintain an below you need to keep which means

where is the average number of days an individual stays infected (Velavan and Meyer 2020). This is why you should think twice before meeting a group larger than 7 if . If instead is around 4% the corresponding maximum number of contacts per day should not exceed 4. We can divide the parameter space of and into a safe and a non-safe area. This is presented in the plot below.

The way to interpret this is:

The longer the infection duration the fewer people we can allow ourselves to meet on a daily basis to keep the infection at bay.

The code

There are many packages around that can create this model and simulations from it but in this post I will show you how to implement it in Julia using just a few lines of code. Basically we need 3 functions. The first function sets up the differential equation of the model. The second one solves the ODEs and then returns an array that we can use for analysis. A quick note here to the observant; You might have noticed that I listed four equations but only define the dynamics of three in the seir_ode function. This is not a bug but a simple usage of the fact that the total population is fixed to and is known since it’s the population of Denmark. This is why is given by knowing the others. Thus, seir_ode models the Susceptible , the Exposed and the Infected , while the seir function adds the after the system has been solved.

The last function r0 is calculating the famous “Basic Reproduction Number”.

Anyway, here’s the code given in julia.

using DifferentialEquations

function seir_ode(dY,Y,p,t)
    β, σ, γ, μ = p[1], p[2], p[3], p[4]
    S, E, I = Y[1], Y[2], Y[3]
    dY[1] = μ*(1-S)-β*S*I
    dY[2] = β*S*I-(σ+μ)*E
    dY[3] = σ*E - (γ+μ)*I
end

function seir(β, σ, γ, μ, tspan=(0.0,100.0); S₀=0.99, E₀=0, I₀=0.01, R₀=0)
    par=[β, σ, γ, μ]
    init=[S₀, E₀, I₀]
    seir_prob = ODEProblem(seir_ode,init,tspan,par)
    sol=solve(seir_prob);
    R=ones(1,size(sol)[2])-sum(sol,dims=1);
    hcat(sol.t, sol[1,:], sol[2,:], sol[3,:], R')
end

r0(β, σ, γ, μ) = (σ/(σ+μ))*(β/(γ+μ))

As you might have inferred from the code there is nothing there that does any actual data fitting. We as users need to provide the parameter values to use. In this post we’ll mainly play around with some of the numbers we already know from the Coronavirus spread in the world.

Simulations

We’re going to run two simulations here where we show the importance of social distancing when fighting a disease like the Coronavirus. Each plot will show a vertical bar indicating how far we are today with respect to the progression of the Epidemic. We’re going to keep the other parameters fixed to and . Thus in both scenarios is the only parameter in the model we will vary.

In the first scenario we’re looking at people just going about as they usually do which in our case will mean that you meet around 20 people every day. This is a really bad scenario where we risk having 2 million people infected in Denmark within a couple of months

But what if we limit the number of people we meet and practice the art of Social Distancing? Well then things look a lot better. We can get the maximum number of infected people down by a factor 2. There’s also the benefit of postponing the peak giving the healthcare system time to catch up.

Let’s look at this more specifically with respect to the maximum number of infected people as a consequence of applying the social distancing. Here we say that we can bring the down from to . The peak number of projected infected people in Denmark is given by the two rightmost vertical lines for both scenarios.

As you can see we’re talking about a reduction of from 1,682,000 people down to around 928,000.

Now imagine that we could improve that even further? The leftmost vertical line in the plot above shows that we can get the peak down to around 58,000! Say we really change our lives and minimize the amount of contacts. Move all of our social activities to online events and basically stop all traveling. Where do you think we could go with respect to the maximum number of infected people?

The three scenarios discussed is presented in the table below where we have assumed and quantify what that means for the maximum number of people you can meet each day, which day the number of infected people will peak.

To really put a nail in the coffin with respect to how important social distancing is going to be for our health and economy I would like to show you the progression for the three scenarios in the table above. I’ll also show you where we are today with respect to time.

As you can see we still have time to act, and the time is now. The longer we wait the more difficult it’s going to be to reduce the and consequently to contain the disease. The question now is, how much are you willing to do to flatten the curve?

Well, that’s all well and good but what about where we are right now. It looks pretty flat and the numbers we’ve seen so far in the news have been much bigger. The numbers in the plot look flat because they are so small in comparison to where the curve is going! Let’s compare the numbers we have registered in Denmark compared to what the model says and fit our parameters based on that. The result is what you can see in the plot below where , with an incubation time of around 1 day.

We can have a look at this fitted model and predict the progression from it. Now, bear in mind that fitting data this early in a series when dealing with non-linear models is extremely uncertain, but it can serve as a flavor of what might come if we are not careful.

Not so cool, a peak infection affecting 2.32 million people.

Conclusions (what the math tells us)

So no doubt about it math is always fun, but today it’s actually also telling us something of utmost importance for our future way of life. Make no mistake, the Coronavirus is a real thing and the dangers reach far beyond people just getting “the flu”. As we outlined in the beginning of this post we can see that there are two major ways we, as in everyone, can limit the damage of this pandemic.

• Practise social distancing (Keeping the low). The rule here is that you should avoid meeting more than around people at any time and preferably avoid meeting new people physically right now. The Danish government as of 2020-03-20 recommends meeting less than people at any given time, but as we can see that’s not good enough.
• Protect ourselves when meeting people (Keeping the low). Here, there are a lot of unknows, but the gist is
  • Sneeze in your arm
  • Cover your mouth when you cough
  • Wear a mask (especially if you have the disease)
  • Wash your hands often
  • Wipe surfaces you come into contact with often (like your phone)
  • Don’t shake hands
  • Avoid touching your eyes, nose and mouth

This is something we all can and should do to help fight the pandemic. If we don’t stand together now, it may very well be too late sooner than you think.

The total and the partial derivative

These terms are typically a source of confusion for many as they are sometimes seen as equivalent and in many cases they seem indistinguishable from one another. They are however not! Let’s start by defining the partial derivative and then move on to the total derivative from there. For this purpose I will use an imaginary function where we have three variables , , and . The partial derivative answers the questions of how changes () when one variable changes by a small amount (). In this setting all other variables are assumed to be constant and static. Thus the partial derivative is denoted . In order to show what happens when we do this operation we need to first define as something. Let’s say it looks like this which incidentally is the volume of an ellipsoid. Well, perhaps not so incidentally.. Either way I have chosen a different parametrization than is commonly used. In the picture below you can see from top to left to right a sphere, spheroid, and ellipsoid respectively. In our setting we can choose for the dimensions.

The partial derivative of the volume of these geometrical spaces then becomes

where we have applied the power rule. As you see the and the was not touched since we assumed them to be fixed. Thus in the picture above we model what happens with the volume as extends or shortens by a small amount. This answers our question if they are really independent of . But what if they are not? Well in this case we need the total derivative of with respect to which is denoted by and is defined like this

where you can see the partial derivative as a part of the total one. So for illustrative purposes let’s constrain the function to a situation where . What happens with the derivative then? Well the partial derivative from before stays the same. but the two other terms we need to calculate. The first part becomes while the last part turns to . Thus now we get

by adding the terms and substituting in the last step. Now hopefully it’s apparent that and that you need to be careful before you state independence between your variables while doing your derivatives.

WAIT! I hear you cry, couldn’t we just do the substitution after we calculated the partial derivative? Indeed you could and you would get something that’s off by a factor 2 which can be substantial. Basically you would get the following insanity

This is because what we’re usually after is indeed the total derivative and not the partial. However, you could of course have done the substitution before you calculated the partial derivative. This would turn out nicely like

where we again reach consistency. Thus, you cannot plug in dependencies in your partial derivative after it’s been calculated!

The chain rule of calculus

One of the perhaps most common rules to use when calculating analytical derivatives is the chain rule. Mathematically it basically states the following

which doesn’t look impressive, but don’t let it’s simplicity fool you. It’s a workhorse without parity in the analytical world of gradients. Remember, could be anything in this setting. So could for that matter. As such this rule is applicable to everything relating to gradients.

Building your own backpropagation engine for deep neural networks

In this section I’ll take you through a simple multi-layered perceptron and a derivation of the backpropagation algorithm. There are many ways to derive this but I’ll start from the error minimization approach which basically describes the fit of a neural network by the deviance from a known target . The architecture we will solve for is shown in the image below where we have two hidden layers. We stick to this for simplicity. We’ll also only use one output instead of multiple but it’s readily generalizable.

Instead of representing our network graphically we’ll do a more formal representation here where the functional form will be stated mathematically. Basically the functional form will be

where the bold face symbols denotes vectors. The function is the sigmoid activation function with a hyperparameter which we will not tune or care about in this introduction. A small note here, disregarding the parameter here is really silly since it will fundamentally change the learning of this network. The only reason I allow myself to do it is because it’s beyond the scope to cover it at this point.

In order to train a neural network we need to update the parameters according to how much they affect the error we see. This error can be defined like this for a regression like problem for a data point .

If we look at the second last layer then we simply update the parameters according to the following rule

for each new data point. This is called Stochastic Gradient Descent (SGD). You can read a lot about that in many places so I won’t dive into it here. Suffice it to say that this process can be repeated for each parameter in each layer. So the infamous backpropagation algorithm is just an application of updating your parameters by the partial derivative of the error with respect to that very parameter. Do the partial derivatives for yourself now and see how easy you can derive it. A small trick you can use is to realize that where I’ve used the prime notation for a derivative. There’s a nice tutorial on how to do this numerically here.

Take home messages

• The total derivative and the partial derivative are related but at times fundamentally different.
• All constraints and variable substitutions have to be done before calculating the partial derivative.
• The partial derivative ignores implicit dependencies.
• The total derivative takes all dependencies into account.
• Many magic recipes, like the backpropagation algorithm, usually comes from quite simple ideas and doing it for yourself is really instructional and useful.

Prerequisites

When reading this it helps if you have a basic understanding of Neural networks and their mathematical properties. Mathwise, basic linear algebra will do for the majority of the post.

Short introductory example - Boston Housing

Instead of writing on and on about how cool this new language is I will just show you how quickly you can get a simple neural network up and running. The first example we will create is the BostonHousing dataset. This is baked into the deep learning library Knet. So let’s start by fetching the data.

using Knet;
include(Knet.dir("data","housing.jl"));
x,y = housing();

Now that we have the data we also need to define the basic functions that will make up our network. We will start with the predict function where we define \(\) and \(x\) as input. \(\) in this case is our parameters which is a 2 element array containing weights in the first element and biases in the second. The \(x\) contains the dataset which in our case is a matrix of size 506x13, i.e., 506 observations and 13 covariates.

predict(ω, x) = ω[1] * x .+ ω[2];
loss(ω, x, y) = mean(abs2, predict(ω, x)-y);
lossgradient = grad(loss);

function train(ω, data; lr=0.01) 
    for (x,y) in data
        dω = lossgradient(ω, x, y)
        for i in 1:length(ω)
            ω[i] -= dω[i]*lr
        end
    end
    return ω
end;

Let’s have a look at the first 5 variables of the data set and their relation to the response that we would like to predict. The y axis in the plots is the response variable and the x axis the respective variables values. As you can see there are some correlations which seems to indicate some kind of relation though this is not definitive proof of a relation!

using Plots;
using StatPlots;

include(Knet.dir("data","housing.jl"));
x,y = housing();
#plotly();
gr();
scatter(x', y[1,:], layout=(3,5), reg=true, size=(950,500))

Here’s the training part of the script where we define and train a perceptron, i.e., a linear neural network on the Boston Housing dataset. We track the error every 10th epoch and register it in our DataFrame.

using DataFrames
ω = Any[ 0.1*randn(1,13), 0.0 ];
errdf = DataFrame(Epoch=1:20, Error=0.0);
cntr = 1;
for i=1:200
    train(ω, [(x,y)])
    if mod(i, 10) == 0
        println("Epoch $i: $(round(loss(ω,x,y)))")
        errdf[cntr, :Epoch]=i
        errdf[cntr, :Error]=loss(ω,x,y)
        cntr+=1
    end
end

Epoch 10: 383.0
Epoch 20: 262.0
Epoch 30: 182.0
Epoch 40: 130.0
Epoch 50: 94.0
Epoch 60: 71.0
Epoch 70: 55.0
Epoch 80: 45.0
Epoch 90: 38.0
Epoch 100: 33.0
Epoch 110: 30.0
Epoch 120: 28.0
Epoch 130: 26.0
Epoch 140: 25.0
Epoch 150: 24.0
Epoch 160: 24.0
Epoch 170: 24.0
Epoch 180: 23.0
Epoch 190: 23.0
Epoch 200: 23.0

If you inspect the training error per epoch you’ll see that the error steadily goes down but flattens out after around 150. This is when we have reached the minimum error we can achieve with this small model. The plot on the right hand side shows the correlation between the predicted house prices and the real observed ones. The fit is not bad but there are definitely outliers.

p1 = scatter(errdf[:,:Epoch], errdf[:,:Error], xlabel="Epoch", ylabel="Error")
p2 = scatter(predict(ω, x)', y', reg=true, xlabel="Predicted", ylabel="Observed")
plot(p1, p2, layout=(1,2), size=(950,500))

ω1 = ω;

ω = Any[0.1f0*randn(Float32,64,13), zeros(Float32,64,1),
        0.1f0*randn(Float32,15,64), zeros(Float32,15,1),
        0.1f0*randn(Float32,1,15),  zeros(Float32,1,1)]

function predict(ω, x) 
    x = mat(x)
    for i=1:2:length(ω)-2
        x = relu.(ω[i]*x .+ ω[i+1])
    end
    return ω[end-1]*x .+ ω[end]
end

predict (generic function with 1 method)

loss(ω, x, y) = mean(abs2, predict(ω, x)-y)
lossgradient = grad(loss)

errdf = DataFrame(Epoch=1:60, Error=0.0)
cntr = 1
for i=1:600
    train(ω, [(x,y)])
    if mod(i, 10) == 0
        errdf[cntr, :Epoch]=i
        errdf[cntr, :Error]=loss(ω,x,y)
        cntr+=1
    end
end

p3 = scatter(errdf[:,:Epoch], errdf[:,:Error], xlabel="Epoch", ylabel="Error")
p4 = scatter(predict(ω, x)', y', reg=true, xlabel="Predicted", ylabel="Observed")
plot(p3, p4, layout=(1,2), size=(950,500))

plot(p2, p4, layout=(1,2), size=(950,500))

xtrn, xtst = x[:, 1:400], x[:, 401:end]
ytrn, ytst = y[:, 1:400], y[:, 401:end]

ω = Any[0.1f0*randn(Float32,64,13), zeros(Float32,64,1),
        0.1f0*randn(Float32,15,64), zeros(Float32,15,1),
        0.1f0*randn(Float32,1,15),  zeros(Float32,1,1)]
errdf = DataFrame(Epoch=1:60, TrnError=0.0, ValError=0.0)
cntr = 1
for i=1:600
    train(ω, [(xtrn, ytrn)])
    if mod(i, 10) == 0
        errdf[cntr, :Epoch]=i
        errdf[cntr, :TrnError]=loss(ω,xtrn,ytrn)
        errdf[cntr, :ValError]=loss(ω,xtst,ytst)
        cntr+=1
    end
end

using StatPlots
@df errdf[5:60,:] plot(:Epoch, [:ValError, :TrnError], xlabel="Epoch", ylabel="Error", 
                       label=["Validation" "Training"], lw=3)

Deep convolutional networks on mnist

The Boston Housing data set is, relatively speaking, a rather simple job for any well versed machine learning girl/guy. As such we need a little bit more bite to dig into what Julia can really do. So with no further ado we turn to convolutional neural networks. We will implement the LeNet which is an old fart in the game but nicely illustrates complexity and power.

using Knet
using Plots
using StatPlots

# Define necessary functions
function predict(w, x0)
    x1 = pool(relu.(conv4(w[1],x0) .+ w[2]))
    x2 = pool(relu.(conv4(w[3],x1) .+ w[4]))
    x3 = relu.(w[5]*mat(x2) .+ w[6])
    return w[7]*x3 .+ w[8]
end

loss(ω, x, ygold) = nll(predict(ω, x), ygold)

lossgradient = grad(loss)

function train(model, data, optim)
    for (x,y) in data
        grads = lossgradient(model,x,y)
        update!(model, grads, optim)
    end
end

# Load the data
include(Knet.dir("data","mnist.jl"))
xtrn, ytrn, xtst, ytst = mnist()
dtrn = minibatch(xtrn, ytrn, 100)
dtst = minibatch(xtst, ytst, 100)

# Initialise neural network
ω = Any[xavier(Float32, 5, 5, 1, 20),  zeros(Float32, 1, 1, 20, 1),
        xavier(Float32, 5, 5, 20, 50), zeros(Float32, 1, 1, 50, 1),
        xavier(Float32, 500, 800),     zeros(Float32, 500, 1),
        xavier(Float32, 10, 500),      zeros(Float32, 10, 1) ]

o = optimizers(ω, Adam)

The last part of the code that will actually do the training I will not execute because well that basically takes forever because I’m currently writing this on a non GPU based laptop :) I have reached out to the developer of Knet for an answer and it appears as though the entire library was primarily developed with a GPU computational platform in mind. This is not an unreasonable assumption due to the good fit between neural networks and GPU’s. However, that doesn’t help us when we want to do deep learning on our CPU based machine.

# println((:epoch, 0, :trn, accuracy(ω, dtrn, predict), :tst, accuracy(ω, dtst, predict)))
# for epoch=1:10
#     train(ω, dtrn, o)
#     println((:epoch, epoch, :trn, accuracy(ω, dtrn, predict), :tst, accuracy(ω, dtst, predict)))
# end

So armed with this information we look to another nice deep learning framework called MXNet. We can use this to define a convolutional neural network like we did with Knet. MXNet is a different framework so of course defining a network looks differently.

We start by loading the package in Julia and declaring a Variable in the MXNet framework called “data”. This will serve as our reference to a data set that we wish to run the network on. Before we move on I should say that MXNet allows for two programming paradigms. - Imperative - Symbolic The symbolic paradigm is the one used by Tensorflow where you define a computational graph and then do operations on it. Thus nothing gets executed until you run data through it. Imperative works more like normal programming which is line by line basis. Both approches have their benefits and caveats and fortunately MXNet and PyTorch supports both. For the remainder of this post we will use the Symbolic interface.

using MXNet
data = mx.Variable(:data)

We will again sort of replicate the LeNet model. The first convolutional layers is specified like this:

conv1 = @mx.chain mx.Convolution(data, kernel=(5,5), num_filter=20)  =>
mx.Activation(act_type=:tanh) =>
mx.Pooling(pool_type=:max, kernel=(2,2), stride=(2,2))

Notice the activation function as well as the pooling. We use 20 filters with a 5 by 5 kernel. On top of that we add a max pooling layer with another 2 by 2 kernel and a 2 by 2 stride. The second layer uses 50 filters with the same tactics. For both convolutional layers we use the tanh activation function.

conv2 = @mx.chain mx.Convolution(conv1, kernel=(5,5), num_filter=50) =>
mx.Activation(act_type=:tanh) =>
mx.Pooling(pool_type=:max, kernel=(2,2), stride=(2,2))

We end the network structure with two fully connected layers of size 500 and 10 respectively. Further the softmax function is applied to the output layer to make the outputs sum to one for each data point. This does NOT turn this into a probability mind you!

fc1   = @mx.chain mx.Flatten(conv2) =>
mx.FullyConnected(num_hidden=500) =>
mx.Activation(act_type=:tanh)

fc2   = mx.FullyConnected(fc1, num_hidden=10)

lenet = mx.SoftmaxOutput(fc2, name=:softmax)

# load data
batch_size = 100
data_name = :data
label_name = :softmax_label
filenames = mx.get_mnist_ubyte()
train_provider = mx.MNISTProvider(image=filenames[:train_data],
                                  label=filenames[:train_label],
                                  data_name=data_name, label_name=label_name,
                                  batch_size=batch_size, shuffle=true, flat=false, silent=true)
eval_provider = mx.MNISTProvider(image=filenames[:test_data],
                                 label=filenames[:test_label],
                                 data_name=data_name, label_name=label_name,
                                 batch_size=batch_size, shuffle=false, flat=false, silent=true)

model = mx.FeedForward(lenet, context=mx.cpu())
optimizer = mx.SGD(lr=0.05, momentum=0.9, weight_decay=0.00001)
# mx.fit(model, optimizer, train_provider, n_epoch=10, eval_data=eval_provider)

using JLD
probs = load("mymxnetresults.jld", "probs")
labels = load("mymxnetresults.jld", "labels")
outliers = load("mymxnetresults.jld", "outliers")

correct = 0
for i = 1:length(labels)
    # labels are 0...9
    if indmax(probs[:,i]) == labels[i]+1
        correct += 1
    end
end

accuracy = 100correct/length(labels)
println(mx.format("Accuracy on eval set: {1:.2f}%", accuracy))

Accuracy on eval set: 98.93%

tmpdf=DataFrame(outliers)

scatter(convert(Array, tmpdf[2,:])', convert(Array, tmpdf[4,:])', ylim=(0, 1), 
        ylabel="Likelihood", xlabel="Wrongly Predicted class")

histogram(convert(Array, tmpdf[4,:])', xlims=(0,1), 
          xlabel="Likelihood", ylabel="Frequency")

Conclusions

Julia is an upcoming cool language that I believe will be well suited for deep learning applications and research. For a while longer I do believe that the early adopters of the language will be research engineers and scientists. There are a lot of things missing in the current packages compared to the Python and R universe but this is not strange given how young the language is. Further, Julia takes care of a lot of issues that Python and R currently have. So give it a go and see how you like it!

Happy inferencing!

Links and resources

• Knet: (http://denizyuret.github.io/Knet.jl/latest/)
• MXNet: (http://mxnet.incubator.apache.org/)
• Julia: (https://julialang.org/)

Estimating the parameters of the model

We’re going to estimate the parameters of this model by

1. The full Bayesian treatment, i.e., we’re going to estimate
2. The Maximum likelihood, i.e., we’re going to estimate which in the tables and the plots will be referred to as “Freq” from the term “Frequentist” which I inherently dislike but I made the tables and plots a while ago so bear with me.

If you rememeber your probability theory training you know that . Sure but so what? Well, this matters a lot. In order to see why let’s dig into these terms. First off, let’s have a look at the proper full Bayesian treatment. We can express that posterior distribution using three terms, namely the

1. Likelihood,
2. the Prior and
3. the Evidence.

The Evidence is the denominator and serves as a normalization factor that allows us to talk about probabilities in the first place. The nominator consists of two terms; the Likelihood (to the left), and the prior (to the right). It’s worth noticing here that the prior for may very well depend on the covariates as such, and even on the response variable should we wish to venture into emperical priors. Explained in plain words the equation above states that we wish to estimate the posterior probability of our parameters by weigting our prior knowledge and assumptions about those parameters with the plausability of them generating a data set like ours, normalized by the plausability of the data itself under the existing mathematical model. Now doesn’t that sound reasonable? I think it does.

Now if we look into the same kind of analysis for what the Maximum Likelihood method does we find the following equation

which states that the probability of observing a data set like ours given fixed ’s is the posterior probability of the ’s divided by our prior assumptions scaled by the total plausability of the data itself. Now this also sounds reasonable, and it is. The only problem is that the quantity on the left hand side is not sampled; it is maximized in Maximum Likelihood. Hence the name.. On top of that what you do in 99% of all cases is ignore the right hand side in the equation above and just postulate that which is a rather rough statement to begin with, but let’s not dive into that right now. So when you maximize this expression, what are you actually doing? Tadam! You’re doing data fitting. This might seem like a good thing but it’s not. Basically you’re generating every conceivable hypothesis known to the model at hand and picking the one that happens to coincide the best with your, in most cases, tiny dataset. That’s not even the worst part; The worst part is that you won’t even, once the fitting is done, be able to express yourself about the uncertainty of the parameters of your model!

Now that we have skimmed through the surface of the math behind the two methodologies we’re ready to look at some results and do the real analysis.

mylm <- lm(y~WDay1+WDay2+WDay3+WDay4+WDay5+WDay6+WDay7, data=ourdata)

data {
  int< lower = 0 > N;       // Number of data points
  vector[N] y;              // The response variable
  matrix[N, 7] xweekday;    // The weekdays variables
}
 
parameters {
  real b0;  // The intercept
  vector[7] bweekday;       // The weekday regression parameters
  real< lower = 0 > sigma;  // The standard deviation
}
 
transformed parameters {
  vector[N] mu;                // Declaration
  mu = b0 + xweekday*bweekday; // The mean prediction each timestep
}
 
model {
  y ~ normal(mu, sigma);    // Likelihood
}
 
generated quantities {
  vector[N] yhat;
  yhat = b0 + xweekday * bweekdayhat;
}

require(brms)
mybrms <- brm(bf(y~WDay1+WDay2+WDay3+WDay4+WDay5+WDay6+WDay7), data=ourdata, cores = 2, chains = 4)

Results

Now to the dirty details of our calculations for the parameter estimates of the model. Throughout the results we will discuss the Bayesian estimation first and then the ML-approach. This pertains to each plot and or table. The first result we will have a look at is the estimates themselves. For the Bayesian estimates we have the average values and the uncertainty expresses as an estimation error. For the ML approach we have the estimates and a standard error. Have a look.

If you’re looking at the table above, you might think “What the damn hell!?”, Bayesian statistics makes no sense at all! Why did we get these crazy estimates? Look at the nice narrow confidence intervals on the right hand side of the table generated by the maximum likelihood estimates and compare them to the wide credibility intervals to the left. You might be forgiven for dismissing the results from the Bayesian approach, since the difference is quite subtle from a mathematical point of view. After all we are computing the exact same mathematical model. The difference is our reasoning about the parameters. If you remember correctly maximum likelihood views the parameters as fixed constants without any variation. The variation you see in maximum likelihood comes from the uncertainty about the data and not the parameters! This is important to remember. The “Std. Error” from the maximum likelihood estimate has nothing to do with uncertainty about the parameter values for the observed data set. Instead it’s uncertainty regarding what would happen to the estimates if we observed more data sets that looks like ours. Remember from the section above that, Statistically speaking, what ML does is maximize which expresses likelihood over different ’s given an observed and fixed set of parameters along with covariates .

But ok, maybe you think there’s something very fishy with this model since the estimates are so different. How could we possible end up capturing the same time series? Well, rest assured that we can. Below you can see a scatter plot between the Observed response and the predicted for the Bayesian and ML estimation. Pretty similar huh? We can also have a look at the average fitted values from the Bayesian estimation and the fitted values from the ML method. As you can see they agree to an rather high degree.

Graphs can be quite decieving though so let’s do our homework and quantify how good these models really are head to head.

Model validation and sanity checking

I’ll start by taking you through the standard measures of goodness within time series analysis. Specifically we have the following measures.

• Mean Absolute Error (MAE)
• Mean Absolute Standard Error (MASE)
• Mean Absolute Percentage Error (MAPE)
• Root Mean Square Error (RMSE)
• Normalized Root Mean Square Error (NRMSE)
• Coefficient of Variation Root Mean Square Error (CVRMSE)
• Proportion of variance explained (R²)

These are quantified in the table below and as you can see there’s virtually no difference between the two estimations. The reason for this is of course that they were built with the same open assumptions about which values that are plausible. In fact both estimation procedures almost accept anything that’s consistent with the data at hand.

So again it seems like there’s not much differentiating these models from one another. That is true while looking at the result of the average fitted values from the two estimates. However, there’s a massive difference in the interpretation of the model. What do I mean by that you might ask yourself, and it’s a good question because if the fit is apparently more or less the same we should be able to pick any of the methods right? Wrong! Remember what I said about sampling being important as it unveils structure in the parameter space that is otherwise hidden through the ML approach. In the illustration below you can see the posterior density of each for the weekday effects. Here it’s clear that they can take many different values which ends up in equally good models. This is the reason why our uncertainty is huge in the Bayesian estimation. There is really a lot of probable parameter values that could be assumed by the model. Also present in the illustration is the ML estimate indicated by a dark vertical line.

If you look closely there are at least two or three major peaks in the densities which denotes the highest probability for those parameters (In this plot we have four different MCMC chains for each parameter), so why on earth is ML so crazy sure about the parameter values? If you read my post you already know the answer, as we already discussed that the error/uncertainty expressed by the ML approach has nothing to do with the uncertainty of the parameters. It’s purely an uncertainty about the data. As such there is no probabilistic interpretation of the parameters under the ML methodology. They are considered as fixed constants. It’s the data that’s considered to be random.

There is one more important check that we need to do and that’s a posterior predictive check just to make sure that we are not biased too much in our estimation. Again inspecting the density and cumulative distribution function below we can see that we’re doing quite ok given that we only have day of week as covariates in our model.

What you should do

So now that we have established that the Bayesian approach is necessary and useful the question still remains on how to fix the estimation. We will do two things to improve upon the estimation

1. Set up informed priors for our believs about the plausability of the parameters
2. Save the sampler some time by setting a baseline for the weekdays

Basically we will modify the model like this

where and are the empirical mean and standard deviation of the response variable respectively. This is a nice practical hack since it makes sure that your priors are in the vicinity of the response you’re trying to model. The resulting code is given below. You can try it on your own daily time series. It’s quite plug and play.

data {
  int< lower = 0 > N;       // Number of data points
  vector[N] y;          // The response variable
  matrix[N, 7] xweekday; // The weekdays variables
}
 
parameters {
  real< lower = 0.01 > b0;  // The intercept
  vector[7 - 1] bweekday; // The weekday regression parameters
  real< lower = 0 > sigma;  // The standard deviation
}
 
transformed parameters {
  // Declarations
  vector[N] mu;
  vector[7] bweekdayhat;
 
  // The weekday part
  bweekdayhat[1] = 0;
  for (i in 1:(7 - 1) ) bweekdayhat[i + 1] = bweekday[i];
  
  // The mean prediction each timestep
  mu = b0 + xweekday*bweekdayhat;
}
 
model {
  // Priors
  b0 ~ normal(mean(y), sd(y));
  bweekday ~ normal(0, sd(y));
 
  // Likelihood
  y ~ normal(mu, sigma);
}
 
generated quantities {
  vector[N] yhat;
  yhat = b0 + xweekday * bweekdayhat;
}

Now let’s have a look at this model instead. A quick look into these parameters show that we have nice clean unimodal posteriors due to our prior beliefs being applied to the analysis. The same table as shown previously is not shown below with the results for the new estimation appended to the rightmost side. For clarification we name the columns Estimate and SD.

As you can see these estimates are quite different and to the naked eye makes more sense from what we know about the data set and what we can expect from intra-weekly effects. We can further check these estimates by inspecting the posteriors further. Note here the “bweekdayhat[1]” which is a delta distribution at 0. This serves as our baseline for the intra-week effect that we’re capturing. The x-axis in the plot are the estimated ’s and the y-axis for each parameter is the posterior probability density.

So from a model estimation standpoint we should be pretty happy now. But how does this new estimation compare to the others? Below I will repeat the model performance table from earlier and extend it with our new “Bayes2” estimation.

It’s evident that our new way of estimating the parameters of the model yields not only a more satisfying modeling approach but also provides us with a more actionable model without any reduction from a performance perspective. I’d call that a win win. Basically this means that our data scientist can go back with confidence and approach the manager again with robust findings and a knowledge about the space of potentially plausible parameters!

Summary and finishing remarks

Today we looked at how to use Bayesian analysis applied to a real world problem. We saw the dangers in applying the maximum likelihood method blindly. Moreover we saw that the Bayesian formalism forces you to make your assumptions explicit. If you don’t it will show you all possibilities that the mathematical models supprts given the data set. This is important to remember and it is NOT a problem with the Bayesian analysis; It is a feature! So if I can leave you with some recommendations and guidelines when dealing with models I would say this:

• There’s nothing wrong in experimenting with ML methods for speady development of prototype models but whenever you are going to quantify your trust in your model you have to and I mean have to sample it and treat it in a proper probabilistic, i.e., Bayesian formalism.
• Always make your assumptions and beliefs explicit in your final model. This will help not only you but fellow modelers who might use your model moving forward.
• Learn to understand the difference between Maximum Likelihood and Sampling the posterior probability distribution of your parameters. It might be hard at first but it will be worth it in the end.
• Accept that there is no such thing as an analysis without assumptions! When you’re doing linear regression using Maximum Likelihood you are effectively assuming that any value between minus infinity and plus infinity are equally likely and that is nonsense my friend.

Happy inferencing!

Background information

Convolutional neural networks

Before we have a look at the results let’s give a quick introduction to what convolutional neural networks really are and why they are more successful at image analysis than normal multi-layered perceptrons. The whole idea behind using convolutional neural networks is that we need the network to be translation and rotation invariant. This just means that we need to be able to recognize an object no matter where in the image it resides. One way to achieve this is to swipe a patch reacting to certain patterns over the image. Think of it as a filter that lights up when it detects something. Of course we don’t know what we are looking for and therefore these filters are learned during training. In general in deep learning we learn lower level features in the initial layers while the later layers captures more elaborate features. This is pretty cool as we can save initially trained early layers and reuse them in other models.

Results

The video below was shot from my cell phone while the modeling team was working. As you can see the model does indeed identify some objects quite successfully. But it also fails to detect many of them. There are many reasons for this. One of them is that this model is optimized for speed and not for performance.

In this short post I showed you how it’s possible to utilize a previously fitted convolutional neural network and classify objects in a retrospective video. However this can be extended into a live object detection from a webcam or a surveillance camera. If you’re interested in doing this yourself have a look at Tensorflow’s example here.

Happy hacking!

The generating model

In order to do this exercise we need to know what we’re doing and as such we will generate the data we need by simulating a stochastic process. I’m not a big fan of this since simulated data will always be, well simulated, and as such not very realistic. Data in our real world is not random people. This is worth remembering, but as the clients I work with on a daily basis are not inclined to share their precious data, and academic data sets are pointless since they are almost exclusively too nice to represent any real challenge I resort to simulated data. It’s enough to make my point. So without further ado I give you the generating model.

which is basically a gaussian mixture model. So that represents the ground truth. The time series generated looks like this

where time is on the x axis and the response variable on the y axis. The first few lines of the generated data are presented below.

So it’s apparent that we have three variables in this data set; the response variable , and the covariates and ( is just an indicator of a fake time). So the real model is just a linear model of the two variables. Now say that instead we want to go about solving this problem and we have two individuals arguing about the best solution. Let’s call them Mr. Granularity and Mr. Aggregation. Now Mr. Granularity is a fickle bastard as he always wants to split things into more fine grained buckets. Mr. Aggregation on the other hand is more kissable by nature. By that I’m refering to the Occam’s razor version of kissable, meaning “Keep It Simple Sir” (KISS).

This means that Mr. Granularity wants to estimate a parameter for each of the two variables while Mr. Aggregation wants to estimate one parameter for the sum of and .

Mr. Granularity’s solution

So let’s start out with the more complex solution. Mathematically Mr. Granularity defines the probabilistic model like this

which is implemented in Stan code below. There’s nothing funky or noteworthy going on here. Just a simple linear model.

data {
  int N;
  real x[N];
  real z[N];
  real y[N];
}
parameters {
  real b0;
  real bx;
  real bz;
  real<lower=0> sigma;
}
model {
  b0 ~ normal(0, 5);
  bx ~ normal(0, 5);
  bz ~ normal(0, 5);
  for(n in 1:N)
    y[n] ~ normal(bx*x[n]+bz*z[n]+b0, sigma);
}
generated quantities {
  real y_pred[N];
  for (n in 1:N)
    y_pred[n] = x[n]*bx+z[n]*bz+b0;
}

Mr. Aggregation’s solution

So remember that Mr. Aggregation was concerned about over-fitting and didn’t want to split things up into the most granular pieces. As such, in his solution, we will add the two variables and together and quantify them as if they were one. The resulting model is given below followed by the implementation in Stan.

data {
  int N;
  real x[N];
  real z[N];
  real y[N];
}
parameters {
  real b0;
  real br;
  real<lower=0> sigma;
}
model {
  b0 ~ normal(0, 5);
  br ~ normal(0, 5);
  for(n in 1:N)
    y[n] ~ normal(br*(x[n]+z[n])+b0, sigma);
}
generated quantities {
  real y_pred[N];
  for (n in 1:N)
    y_pred[n] = (x[n]+z[n])*br+b0;
}

Analysis

Now let’s have a look at the different solutions and what we end up with. This problem was intentionally noise to confuse even the granular approach as much as possible. We’ll start by inspecting the posteriors for the parameters of interest. They’re shown below in these caterpillar plots where the parameters are on the y-axis and the posterior density is given on the x-axis.

It is clear that the only direct comparison we can make is the intercept from both models. Now if you remember, the generating function doesn’t contain an intercept. It’s . Visually inspecting the graphs above will show you that something bad is happening to both models. Let’s put some numbers on this shall we. The Tables below will illuminate the situation.

Conclusion

I’ve shown you today the dangers of aggregating information into a single unit and what those dangers are. There is a version of the strategy shown here which brings the best of both worlds; Hierarchical pooling. This methodology pulls data with low information content towards the mean of the other more highly informative ones. The degree of pooling can be readily expressed as a prior belief on how much the different subparts should be connected. As such; don’t throw information away. If you believe they belong together, express that belief as a prior. Don’t restrict your model to the same biases as you have! In summary:

• Always add all the granularity you need to solve the problem
• Don’t be afraid of complexity; it’s part of life
• Always sample the posteriors when you have complex models
• Embrace the uncertainty that your model shows
• Be aware that the uncertainty quantified is the model’s uncertainty

Happy Inferencing!

Conclusion

This post has been about not becoming blind with respect to the mathematical restrictions we impose on the total model by sticking to a too simplistic representation. Also in this case the Bayesian formalism does not save us with it’s bimodal capabilities since the model was misspecified.

• Think about all aspects of your model before you push the inference button
• Be aware that something that might appear as a clear cut case for multimodality may actually be a pathological problem in your model
• Also, be aware that sometimes multimodality is expected and totally ok

Happy inferencing!

A probabilistic formulation

Any regression problem can be expressed as an implementation of a probabilistic formulation. For instance what we typically have at our hand is a dependent variable , a matrix of covariates and a parameter vector . The dependent variable consists of data we would like to learn something about or be able to explain. As such we wish to model it’s dynamics via the through . The joint probability distribution for these three ingredients is given simply as . This is the most general form of representing a regression problem probabilistically. However, it’s not very useful, so in order to make it a bit more tangible let’s decompose this joint probability like this.

In this view it is clear that we want to learn something about since that’s the unknowns. The other parts we have observed data on. So we would like to say something clever about . How do we go about doing that? Well for starters we need to realize that can actually be written as

which means that

and therefor

which is just a derivation of Baye’s rule. Now we actually have something a bit more useful at our hands which is ready to be interpreted and implemented. What do I mean by implemented? Seems like an odd thing to say about probability distributions right? As weird as it may seem we actually haven’t given the probability distributions a concise mathematical representation. This is of course necessary for any kind of inference. So let’s get to it. The first term I would like to describe is the likelihood i.e. the which describes the likelihood of observing the data given the covariance matrix and a set of parameters . For simplicity let’s say this probability distribution is gaussian thus taking the following form . This corresponds to setting up a measurement model where .

The second term in the nominator on the right hand side is our prior which we will also consider gaussian. Thus, we will set indicating that the parameters are independant from each other and most likely centered around with a known standard deviation of . The last term is the denominator which in this setting functions as the evidence. This is also the normalizing constant that makes sure that we can interpret the right hand side probabilistically.

That’s it! We now have the pieces we need to push the inference button. This is often for more complicated models done by utilizing Markov Chain Monte Carlo methods to sample the distributions. If we are not interested in the distribution but only the average estimates for the parameters we can just turn this into an optimization problem instead by realizing that

since just functions as a normalizing constant and doesn’t change the location of the that would yield the maximum probability. Thus we can set up the optimization problem as

and maximize this function. Normally when we solve optimization problems it’s easier and nicer to turn it into a minimization problem instead of a maximization problem. This is easily done by minimizing

as opposed to the equation before. For the sake of clarity let’s assume from now on that we only have one independent variable and only one parameter . Since we know that

we can easily unfold the logarithms to reveal

which can be more nicely written as

where . As such putting a Gaussian prior on your is equivalent penalizing solutions that differ from by a factor of i.e. 1 divided by two times the variance of the Gaussian. Thus, the smaller the variance, the higher our prior confidence is that the solution should be close to zero. The larger the variance the more uncertain we are about where the solution should end up.

Ridge regression

The problem of regression can be formulated differently than we did previously, i.e., we don’t need to formulate it probabilistically. In essance what we could do is state that we have a set of independent equations that we would like to solve like this

where the variables and parameters have the same interpretation as before. This is basically Ordinary Least Squares (OLS) which suffers from overfitting and sensitivity to outliers and multicollinearity. So what Ridge regression does is to introduce a penalty term to this set of equations like this

where is typically chosen to be . This means that all values in the parameter vector should be close to 0. Continuing along this track we can select a dumbed down version of this equation to show what’s going on for a simple application of one variable and one parameter . In this case

turns into

which you may recognize from before. Not convinced? Well let’s look into the differences.

Probabilistic formulation	Ridge regression

Here it’s pretty obvious to see that they are equivalent. The constant plays no role in the minimization of these expressions. Neither does the denominator . Thus if we set the equivalence is clear and we see that what we are really minimizing is

which concludes my point.

Summary

So I’ve just shown that ridge regression and a Bayesian formulation with Gaussian priors on the parameters are in fact equivalent mathematically and numerically. One big question remains; Why the hell would anyone in their right mind use a probabilistic formulation for something as simple as penalized OLS? The answer you are looking for here is “freedom”. What if we would have selected a different likelihood? How about a different prior? All of this would have changed and we would have ended up with a different problem. The benefit of the probabilistic approach is that it is agnostic with respect to which distributions you choose. It’s a consistent inferential framework that just allows you the freedom to model things as you see fit. Ridge regression has already made all the model choices for you which is convenient but hardly universal.

My point is this; whatever model you decide to use and however you wish to model it is your prerogative. Embrace this freedom and don’t let old school convenient tools dictate your way towards solving a specific problem. Be creative, be free and most of all: Be honest to yourself.

Happy inferencing!

Setting up the experiment

We would like to optimize the Bolognese quality by tweaking the time we cook it. As we saw above it’s not a nice function to optimize which makes it excellent for hard core testing algorithms for robustness and ability to avoid local maxima.

Since there are many optimization methods and algorithms I will try to do a benchmark on the most common ones and the ones I like to use. This is not an exhaustive list and there may be many more worthy of adding, but this is the list I know. Hopefully it can guide you in your choices moving forward.

The algorithms we will look into are primarily implemented in NLopt except the Simulated annealing which comes from the standard optim function in R. Either way the list of algorithms are:

• Simulated Annealing (SANN)
• Improved Stochastic Ranking Evolution Strategy (ISRES)
• Constrained Optimization BY Linear Approximations (COBYLA)
• Bound Optimization BY Quadratic Approximation (BOBYQA)
• Low-storage Broyden–Fletcher–Goldfarb–Shanno algorithm (LBFGS)
• Augmented Lagrangian algorithm (AUGLAG)
• Nelder-Mead Simplex (NELDERMEAD)
• Sequential Quadratic Programming (SLSQP)

As some algorithms (or even most of them) may depend heavily on initial values and starting points we run 500 optimization with randomized starting points for all algorithms. Each algorithm receive the same starting point for each iteration. Thus, each algorithm gets to start it’s optimization in 500 different locations. Seems like a fair setup to me.

Then let’s get to the results. Below you’ll see a box and whiskers plots of the Optimal Bolognese quality found by each algorithm among the 500 trials. There’s no doubt that all are pretty ok. However, the Augmented lagrangian, Simulated annealing and BFGS are the worst by far. They are also the ones who are the most incosistent. This basically means that they are very sensitive to local maxima and cannot explore the entire landscape properly. This may not be too surprising as they are based on gradient information. Second order or not, it’s still gradient information and that my friends is always local by nature.

Where did all the optimizations for the different algorithms go? Let’s make a scatterplot and find out! The continuous line is our original Bolognese function that we want to optimize with respect to the parameter timing .

That looks quite bad actually. Especially for many of the gradient based approaches as they get stuck in local maxima or in regions of 0 gradient. Most of them only find the right solution parts of the time. This is consistent with the box and whiskers plot. However, I would say it’s actually worse than what it appears like in that plot. Just to illustrate how many bad optimization you get have a look at the plot below where we jitter the points across the boxes.

So where did the optimizations end up on the Bolognese optimization on average? Well in order to see that we will use robust statistics instead to make sure we actually select values that were optmized. Robust statistics in this case will be the median. So now we illustrate the median cook times found by the different algorithms. For fun we will show you the mean results as well. Just for your educational pleasure.

The table above is pretty clear. From an optimization point of view the results are clear from both and average, median, worst and best case scenario. The only potential problem here is timing. Evolutionary algorithms are notoriously slow. So experiment with this yourself and see if you get similar results.

Happy inferencing!