We tested Citesheet on GitLab's public CAIQ

The short answer

We filled GitLab's public CAIQ using only GitLab's public policies, then checked our answers against the answers GitLab itself published.

• 158 of 163 filled answers matched GitLab's own answers.
• 0 answers carried an invented quote. Every quote is checked word for word by code against the source document. A quote that fails the check is dropped.
• Questions with no proof in the policies stayed empty and were marked "No proof found."

The headline comes from our run on October 5, 2026, after we added review drafts: a row marked "Needs review" now also gets a drafted Yes / No / NA value, shown in yellow with a "Check:" line saying what to verify. In that run 163 of the 261 rows got a value (answered rows plus drafted review rows), and 158 of those matched the value GitLab published (96.9%). All 5 that differed were yellow review rows, so none of them would go out without a person looking at it. The other 98 rows got no value.

The rest of this page explains how the test worked, the earlier runs in our decision log (before review drafts, when only fully answered rows got a value), what went wrong, and what the test does not prove.

What we tested

The questionnaire. GitLab publishes a completed CAIQ (the Cloud Security Alliance's Consensus Assessments Initiative Questionnaire) for GitLab.com on the CSA STAR Registry. We used the version dated 2025-03-25, which is CAIQ v4.0.3. That version has 261 questions across the 17 domains of the Cloud Controls Matrix. We made a blank copy: same file, same layout, GitLab's answers removed.

The documents. 28 policy and standard pages from GitLab's public handbook: access management, encryption, change management, incident response, logging, business continuity, data classification, retention, secure development, training, and others. Nothing else. No internal documents, no answer library, no past questionnaires.

The answer key. GitLab's own published answers (Yes, No or NA for each question). These are GitLab's claims, not verified facts, but they are the best public key there is: the same company wrote both the policies and the answers.

What the words mean

• Answered. Citesheet wrote an answer, backed by at least one quote that code confirmed exists word for word in one of the 28 documents.
• Needs review. Citesheet found some proof but not enough to cover the whole question, or a specific condition was unproven (for example a frequency or "all systems"). The row is filled with a "Check:" note so a person looks at it before it goes out.
• Abstained ("No proof found"). No passage in the documents was strong enough to support an answer. The row stays empty and goes on the gap list.
• Value written. The Yes / No / NA cell was filled.
• Matched. The value we wrote is the same as the value GitLab published for that question.

Detailed numbers from our decision log

Our decision log records the earlier full runs of the blank GitLab CAIQ (261 rows, default settings), before review drafts, so "value written" there counts only fully answered rows. The current answering rules are "v3". The earlier and later attempts are shown so you can see what changed.

In the v2 run the one different value was DCS-07.2: we answered No, GitLab answered NA. The v4 change was not kept because it raised answered rows by only 3, which did not meet the bar we set before the run.

Why so many rows need review. Of the 130 review rows in the v3 run, 17 name an unproven specific condition (8 a frequency, 8 a scope such as "all", 1 a named technology), and 25 name another unproven part. The other 88 read as complete, but the second model that checks coverage still said "partial." That checker is cautious on purpose: a row a person checks is better than a wrong answer nobody sees. It is also the main thing we are working to improve.

Why so many rows abstain. GitLab answered Yes to most of the CAIQ, but its public policies often point to internal pages for the details. For example, its public Access Management Policy links to an Access Management Standard in its internal handbook. Citesheet only sees what you give it, so those questions get "No proof found" rather than a guess.

The labelled test

The CAIQ run only checks the Yes / No / NA value. To check whether answers and quotes are actually right, we also keep a labelled set of 128 questions written against public policy sets (GitLab, Tailscale, PostHog, the University of California and others): 60 test questions, 59 tuning questions, and 9 that should be refused because the documents don't cover them.

Important caveat: the labels were drafted and reviewed by Claude (an AI model), not verified by a person. Every number below is provisional until they are hand-checked.

Results with the current settings:

* From the earlier runs in our decision log that measured these; the other rows are the current rules.

The AI cost is what the model calls cost us to run. It is not a price. Our own target was 90% precision, and we are below it. Most wrong answers say "Yes" where the label says "Partial": the quote is right, but the answer claims more than the quote proves.

Where it went wrong

These are the misses from the most recent run in our log that lists them one by one (the "v2" rules, 21 misses on the labelled set). Question IDs only. GitLab IDs are CAIQ question IDs; other IDs are our own labels.

• Said "Yes" where the label says "Partial" (9): GitLab CEK-01.1, IAM-03.1, DSP-09.1, DSP-16.1, plus one hard GitLab question, one PostHog and three University of California questions.
• Other wrong value (1): one Tailscale question answered "No" where the label says "Partial."
• Right value, but the quote is not one we labelled (5): GitLab IAM-02.1, two PostHog, one University of California and one hard PostHog question. The quote may be just as good; we have not reviewed these.
• Should have answered but refused (6): GitLab CCC-06.1 and one Tailscale question had no quote that passed the check. GitLab GRC-05.1, two hard PostHog and one hard Tailscale question were refused because the best passages scored too low.

After that run we changed one product rule. Standard policy wording such as "established, documented, approved" now counts as proof of that wording. Specific conditions (frequencies, "all", named standards, numbers) still need their own proof. Six labels changed from "Partial" to "Yes" under that rule, including CEK-01.1 and IAM-03.1. None of them is in the test set.

What this test does not show

• One questionnaire. The CAIQ v4.0.3 only. We have not measured other questionnaires, and GitLab's registry listing now shows a newer CAIQ v4.1.0 submission.
• One company's documents. GitLab's handbook is long, public and well organized. A small company with a few short policies will see more "No proof found" rows. That is accurate, but it means less of the file is filled.
• GitLab's answers are a key, not the truth. They are GitLab's own claims, made in March 2025. The handbook pages are from 2026, and some have changed since.
• Labels are not human-verified. The labelled set was reviewed by Claude, not by a person.
• Laptop runs. These runs were done once on a development laptop, not on our production servers. We did not measure speed here.
• Matching the value is not the whole answer. A matching "Yes" can still have a weak explanation. That is why every answer shows its quote, and why you review flagged rows before sending.

Reproduce it

Everything we used as input is public:

1. The questionnaire. GitLab.com's completed CAIQ on the CSA STAR Registry. We used the 2025-03-25 v4.0.3 submission; the listing may now show a newer one. GitLab also offers a pre-filled CAIQ through its Security Resource Center. The question text belongs to the Cloud Security Alliance, so don't republish the file.
2. The documents. GitLab's handbook repository. Our 28 pages are under content/handbook/security/, plus the acceptable use, record retention, DPIA and business continuity pages.
3. The comparison. Remove GitLab's answers, fill the blank copy with your tool, and compare the Yes / No / NA column row by row with GitLab's file. Count only rows your tool filled.

Our evaluation code and our labelled set are not public. If you want to check our numbers on your own copy, send us a message and we'll walk you through it.

How Citesheet helps

Citesheet fills your Excel questionnaire from your own policies, quotes the exact words behind every answer, and says "No proof found" when the proof isn't there.