The Small Honesty Project Now studyinggemma4:26b-a4b

Citizen research · offline AI

How honest can a small AI on your own laptop be made?

Not right every time. Right when it answers, and saying so when it isn't sure. One laptop, no internet, every answer read by hand.

gemma4:26b-a4b · M4 MacBook Air, 24GB · 21–22 Oct

  • right
  • “I don’t know”
  • wrong

Drag to turn · tap a point to read it

Logbook Open the record → The Small Honesty Project's 88×31 button: get it for your site

02 · Now studying

02 · Now studying

Gemma 4 26B A4B

Google DeepMind’s open model, a mixture of experts: 26B stored, about 4B working on each word. Every number on this page comes from this exact build.

The exact buildgemma4:26b-a4bdigest 48eb98ec778c · 16 GB · MLX · Apache 2.0
Download it from Ollama ↗
ollama pull gemma4:26b-a4b

From the maker: Google’s model card ↗ · Gemma 4 at Google DeepMind ↗

Ollama counts this build as 26.2B parameters; Google’s card lists 25.2B in all, 3.8B active. Pulled 20 Oct 2026 in 8 minutes; archived since, so this exact build can be run again.

Why this model

  1. Which model doesn’t make things up?

    Five models, the same trap questions: a novel that doesn’t exist, a Python function that doesn’t either. gemma4:12b was the only one to decline both and know the 2024 result. It became the one local model.

  2. Ready for an earthquake?

    Four models, 14 emergency questions, graded by hand. Only the 12B got all 14 right; the others gave at least one dangerous or useless answer.

  3. Lab scores aren’t real use

    A lighter model for everyday chat: Llama 3.1 8B won the trial, 38 of 39 on honesty probes, then invented facts in the owner’s own chats and was retired.

  4. Bigger wasn’t more expensive

    The 26B against the 12B, overnight, power and heat logged every 5 seconds. The 26B was faster, cheaper and knew more. It also invented more about things that don’t exist.

    Night 1 · the 26B against the 12B, overnight

    Busy for each chat replylower is better
    12B12.3 s
    26B4.3 s
    Energy for a short answerlower is better
    12B129 J
    26B82 J
    Right of 81, Wikipedia firsthigher is better
    12B47
    26B71
    Memory, loaded at 32Kthe cost
    12B13.7 GB
    26B19.8–20.7 GB
    “I don’t know” to 15 unknowablesthe catch
    12B14 of 15
    26B7 of 15

  5. The homework: make the smarter one honest

    Shaped with a scoped prompt, a names check and offline Wikipedia: 77 of 81 right, and the fake author’s invented career went from 12 of 12 to 0 of 8. Open the record

So it’s the test case because it’s the most capable model this laptop runs well, with an honesty gap you can measure. The model is a part, not the product: whichever proves itself on the same tests takes the bench.

The lineup one family, three sizes, so only the size changes

  • Now studyinggemma4:26b-a4bmixture of experts · 16 GB · on the bench since 20 Oct · shaped on night 3
  • The controlgemma4:12bdense · 7.6 GB · the honest one from memory (14 of 15) · measured on night 1 · night 4, planned: the night-3 suite exactly as it ran for the 26B
  • Proposed nextgemma4:e2bsmall · 7.2 GB with its encoders · out of the box it declined 4 of 13 fakes, but answered in 2.9 s for 27 J (about 7,300 answers a charge). Can shaping lift it? Night 4, planned, the same suite; not run yet.
The homework · 12 tried and cut

The homework twelve tried and cut, one on the bench, each for a reason

  • gemma4 (E4B)the old default: the 12B was smaller on disk and stronger
  • qwen3.5:9binvented a plot for a novel that doesn’t exist
  • qwen3.5:4binvented the same plot; said Harris won in 2024
  • granite4.2:8b“2 tablespoons of bleach per gallon”: a dangerous emergency answer
  • ministral-3:8btwo dangerous emergency answers
  • gemma4:e2bdeclined 4 of 13 fakes; put the San Andreas under Memorial Stadium (proposed again as the small test)
  • llama3.2:3bmade things up: declined 1 of 2 traps
  • nemotron-3-nano:4b-mlxdeclined 0 of 2 traps
  • ministral-3:3bdeclined 0 of 2 traps
  • bonsai-2:9bhonest, but slower and hotter than Llama 3.1 8B (4.5 s a reply, 103.7 °C)
  • llama3.1:8bwon the chat trial, 38/39 on honesty probes, then invented facts in real use: retired
  • medgemma1.5:4bcalled a Lyme bull’s-eye rash sunburn, 5 of 6 times

From the logbook’s model bake-off (27 Sep), the emergency comparison (29 Sep), the chat trial (18–19 Oct) and the first-aid photo kit (22 Oct).

03 · The field guide

03 · The centerpiece

The field guide to shaping

A model out of the box is a refined material, not a finished tool. Here is each kind of wrong we met, given a plain name, with the layer that moved it. Where nothing did, we say so.

    Met a kind of wrong that isn’t here? Name it. No account, no email.

    >

    Point at a row to see which layer moved it

    04 · The record

    04 · Every answer, night 3

    The record

    Every answer from night 3, one mark each: the questions down the side, the setups across the top. Tap a question for its answers, a column for its recipe. All of it is yours to take, as JSON or CSV.

        Open data · this view · the whole record

        05 · The open board

        05 · Open all the time · no account

        The open board

        A lab anyone can walk into. Run the same 81 questions on your own machine and post what you got; add a question for the next night; say when a grade looks wrong. Nobody signs up.

        • The lab’s own · from the logbook, with its recipe
        • Unverified · every post, on arrival
        • Checked · the lab re-ran a sample and agrees
        Sorted by

        The board opens to everyone soon. For now it holds the lab’s own results; anything you post stays in your browser.

        Post your run

        1. 1 Run the 81 on your machine.
        2. 2 Paste the file the kit writes, or fill in the numbers.
        3. 3 Post it. Your first post gives you a key, to post again under the same mark.
        Machine

        Three posts under one key make your mark a fellow’s: a thank-you, nothing more.

        Give it a question for the next night

        Something you’d want an offline AI to get right, or a trap it should refuse. It joins the lattice as an unlit point until it’s been asked.

        >

        Say when a grade looks wrong

        Every answer on the bench has a “say so” under it, and every answer in the record can be put on the bench. A person reads each note.

        Back to the bench

        Point at a row to see which layer moved it