The climbers

Meet the climbers.

Before a search engine can answer even one question, somebody has to travel the whole web and draw the map. Ours climb it. Peak to peak, link to link, they hike the entire range and come home with notes on everything: what each page says, what it shows, and who points to it.

Other engines send machines that nobody would invite in. Ours are climbers: they travel light, they stay on the trail, they obey the signs, and they write down what they find so the rest of us can search it. This page starts plain and gets deeper as it goes, the way a trail does.

Basecamps

Every climb starts from a basecamp: a small set of pages we know well and trust, the trailheads of the range. From a basecamp, a climber follows links outward, one ridge at a time. Basecamps are chosen by hand and checked often, because every map is only as honest as the ground it started from.

Trailhead signs

Most sites post a small rules file at their gate. It says which trails are open, which are closed, and how fast visitors should walk. Our climbers read the sign at every single gate and obey it, exactly, every time. A closed trail stays unclimbed, even if it looks interesting. A slow trail gets walked slowly. Good guests get invited back, and the map keeps growing because the range trusts the people drawing it.

The frontier

Every time a climber crosses a link that points somewhere new, that somewhere goes on a list: the frontier, the mountains nobody has climbed yet. The frontier is the most exciting page in the whole operation. It means the map is not finished, and it never will be, because the range grows every day. Climbers work the frontier in order of promise: a mountain pointed to by many trusted peaks gets climbed early.

Re-climbing

Mountains change. A busy peak, a news page or a busy shop, gets re-climbed every day, sometimes every hour. A quiet peak, an old poem on a still page, might be revisited once a month. Each visit compares the new notes with the old ones, so the map always shows the range as it is now, not as it was last season.

The deep end

The map is a graph

Underneath the story, the map is a link graph. Every page is a node. Every link is a directed edge from one node to another. A climber's notes on each node record the essentials: the canonical address, the title, the main content and its hash, the list of outbound edges, and a running count of inbound ones. The graph is the single source of truth that search reads from. Nothing is in the index that is not first on the map.

One mountain, one entry

The same page often answers to several addresses. Before a note is filed, its address is normalized: scheme and host lowercased, default ports dropped, fragments stripped, query parameters put in a fixed order, and trailing slashes made consistent. Addresses that normalize to the same string are the same mountain, and the map keeps one entry for it, with the other addresses written down as aliases.

Duplicates, detected by content

Two different addresses can still hold the same page. So the map does not trust addresses alone: it hashes the main content of every note. Identical hashes are exact duplicates. Near-identical content is grouped by similarity, and one member of the group is chosen as the canonical copy, the one search will show. The choice weighs the same signals the ranking does: who points to it, how it behaves, and whether it is the copy the site itself names as canonical.

Revisit math

Re-climbing everything constantly would be rude and wasteful, so the schedule is earned. Each node carries an observed change rate: how often its content hash actually changed across past visits. The revisit interval is set from that rate, proportional to how often the page truly changes and weighted by how much the page matters in the graph. A page that changed on nine of the last ten visits is revisited roughly ten times as often as one that changed once. A politeness cap per host sits above all of it: no matter how busy a mountain is, climbers never arrive faster than the trailhead sign allows.

How the map feeds Kindness Rank

Kindness Rank, our ranking system, reads the map the way a climber reads the range: a link is a recommendation, and a recommendation is worth more from a peak that is trusted and points rarely. The graph supplies the edges, the notes supply the trust signals, and the rank settles out of the whole range at once. The full mathematics lives in the K Rank white paper.

Doodle runs dispatch

Somebody has to hold the frontier list, hand out the day's routes, and make sure two climbers never knock on the same quiet door at once. That is Doodle, our head climber. She assigns the climbs, merges the notes when the teams come home, and keeps the one rule nobody breaks: we climb the web, we hike the web, and we leave every trail exactly as we found it.

Clear is being set up as a non-profit. 100% of the money goes to orphanages in South America.