Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commit8098672unknown_key

feat(BLOCK-J): J1 dependency graph

feat(BLOCK-J): J1 dependency graph

Adds drizzle/0028_repo_dependencies.sql (repo_dependencies table indexed
by (repo,ecosystem) + (name)), src/lib/deps.ts with defensive parsers for
package.json / requirements.txt / pyproject.toml / go.mod / Cargo.toml /
Gemfile / composer.json, and src/routes/deps.tsx serving
/:owner/:repo/dependencies (grouped by ecosystem) plus owner-only
reindex. Also exposes repositoriesDependingOn() for reverse-dep graphs.
646 tests pass.
Claude committed on April 15, 2026Parent: edf7c36
7 files changed+1172−180986721cc8c1b499682f05c0aafb2b314397df7
7 changed files+1172−1
ModifiedBUILD_BIBLE.md+6−1View fileUnifiedSplit
8787| Code search (ILIKE) | ✅ | per-repo + global |
8888| Semantic / embedding search | ✅ | D1 — `code_chunks` table + lexical fallback, optional Voyage `voyage-code-3`; `src/lib/semantic-search.ts`, `src/routes/semantic-search.tsx` |
8989| Symbol / xref navigation | ✅ | I8 — `src/lib/symbols.ts` regex-based extractor for ts/js/py/rs/go/rb/java/kt/swift; on-demand indexer persists top-level definitions into `code_symbols` (0025). `src/routes/symbols.tsx` serves `/:owner/:repo/symbols` overview + A–Z list, `/:owner/:repo/symbols/search?q=` prefix search, `/:owner/:repo/symbols/:name` definition detail. Owner-only reindex. |
90| Dependency graph | ✅ | J1 — `src/lib/deps.ts` parses package.json / requirements.txt / pyproject.toml / go.mod / Cargo.toml / Gemfile / composer.json without a TOML lib. `src/routes/deps.tsx` serves `/:owner/:repo/dependencies` grouped by ecosystem with per-ecosystem counts; owner-only reindex walks the default-branch tree (max 200 manifests, 1MB each). `drizzle/0028_repo_dependencies.sql` adds `repo_dependencies`. |
9091
9192### 2.3 Collaboration
9293| Feature | Status | Notes |
278279- **I9** — Repository mirroring → ✅ shipped. `drizzle/0026_repo_mirrors.sql` adds `repo_mirrors` (one-per-repo config) + `repo_mirror_runs` (audit log). `src/lib/mirrors.ts` provides URL validation (https/http/git only, no ssh/file/paths/shell metas), credentials-redaction for logs, and `runMirrorSync` that shells out to `git fetch --prune --tags` with a 5-min timeout and `GIT_TERMINAL_PROMPT=0`. `src/routes/mirrors.tsx` serves owner-only `/:owner/:repo/settings/mirror` + site-admin `/admin/mirrors/sync-all`. 17 new tests.
279280- **I10** — Enterprise SSO via OIDC → ✅ shipped. `drizzle/0027_sso_oidc.sql` adds `sso_config` (singleton `id='default'` row with issuer + authorize/token/userinfo endpoints + client credentials + scopes + optional email-domain allow-list + `auto_create_users` toggle) and `sso_user_links` (maps local user to IdP `sub`, unique per-subject). `src/lib/sso.ts` exposes `buildAuthorizeUrl`/`exchangeCode`/`fetchUserinfo`/`findOrCreateUserFromSso` pure helpers — plain OIDC auth-code flow, no XML / no signature verification dep. `src/routes/sso.tsx` serves site-admin `/admin/sso` config page, `/login/sso` initiator (state + nonce cookies, 10-min TTL), `/login/sso/callback` exchanger + session issuer, plus `POST /settings/sso/unlink` for users. `/login` renders "Sign in with <provider name>" when enabled. 24 new tests (pure helpers + route-auth smokes).
280281
282### BLOCK J — Beyond-parity advanced features
283- **J1** — Dependency graph → ✅ shipped. `drizzle/0028_repo_dependencies.sql` adds `repo_dependencies` (ecosystem + name + version_spec + manifest_path + is_dev + commit_sha) with indexes on `(repository_id, ecosystem)` + `(name)`. `src/lib/deps.ts` parses seven manifest formats (package.json, requirements.txt, pyproject.toml, go.mod, Cargo.toml, Gemfile, composer.json) without a TOML library — each parser is defensive and returns `[]` on malformed input. Walks default-branch tree (max 200 manifests, 1MB each), replaces the prior set on reindex. `src/routes/deps.tsx` serves `/:owner/:repo/dependencies` (grouped by ecosystem with per-ecosystem counts) + owner-only `POST /dependencies/reindex`. Reverse-dep lookup via `repositoriesDependingOn(ecosystem, name)` for future "who depends on me" network-graph UI. 21 new tests.
284
281285### BLOCK H — Marketplace
282286- **H1** — App marketplace → ✅ shipped. `src/routes/marketplace.tsx` + `src/lib/marketplace.ts` + `drizzle/0021_marketplace_and_apps.sql` (5 tables: `apps`, `app_installations`, `app_bots`, `app_install_tokens`, `app_events`). Routes: `GET /marketplace` (public directory with search), `GET /marketplace/:slug` (detail + install CTA), `POST /marketplace/:slug/install` (user-target install in v1), `POST /marketplace/installations/:id/uninstall`, `GET /settings/apps` (personal list), `GET+POST /developer/apps-new` (register), `GET /developer/apps/:slug/manage` (event log + install count), `POST /developer/apps/:slug/tokens/new` (show-once token). Install idempotent via soft-update on existing non-uninstalled row.
283287- **H2** — GitHub Apps equivalent (bot identities + installation tokens) → ✅ shipped. Same schema as H1: every app gets a `<slug>[bot]` row in `app_bots`. `generateBearerToken()` produces `ghi_`-prefixed bearers; `hashBearer` (sha256) is the only form persisted. `verifyInstallToken(token)` returns `{installation, app, botUsername, permissions}` or `null` (checks revoked/expired/uninstalled/suspended). Permission vocabulary: `contents:read/write`, `issues:read/write`, `pulls:read/write`, `checks:read/write`, `deployments:read/write`, `metadata:read` — `hasPermission` implements write→read implication.
292296- `src/app.tsx` — route composition, middleware order, error handlers
293297- `src/index.ts` — Bun server entry
294298- `src/lib/config.ts` — env getters (late-binding)
295- `src/db/schema.ts` — 86 tables. New tables only via new migration.
299- `src/db/schema.ts` — 87 tables. New tables only via new migration.
296300- `src/db/index.ts` — lazy proxy DB connection
297301- `src/db/migrate.ts` — migration runner
298302- `drizzle/0000_initial.sql`, `drizzle/0001_green_ecosystem.sql` — migrations
320324- `drizzle/0025_code_symbols.sql` (Block I8) — migration, never edited in place. Adds `code_symbols` table with indexes on `(repository_id, name)` + `(repository_id, path)`.
321325- `drizzle/0026_repo_mirrors.sql` (Block I9) — migration, never edited in place. Adds `repo_mirrors` (unique on `repository_id`) + `repo_mirror_runs`.
322326- `drizzle/0027_sso_oidc.sql` (Block I10) — migration, never edited in place. Adds `sso_config` singleton (`id='default'`) + `sso_user_links` (`subject` unique, FK to `users` with ON DELETE CASCADE).
327- `drizzle/0028_repo_dependencies.sql` (Block J1) — migration, never edited in place. Adds `repo_dependencies` with indexes on `(repository_id, ecosystem)` + `(name)`.
323328
324329### 4.2 Git layer (locked)
325330- `src/git/repository.ts` — tree / blob / commits / diff / branches / blame / search / raw / tags / commitsBetween
Addeddrizzle/0028_repo_dependencies.sql+31−0View fileUnifiedSplit
1-- Gluecron migration 0028: dependency graph.
2--
3-- J1 — Parses manifest files in a repo and stores a per-repo SBOM.
4-- `repo_dependencies` is a "last known state" set — each reindex REPLACES the
5-- prior rows. `ecosystem` is the package manager (npm, pypi, go, rubygems,
6-- cargo, composer). `manifest_path` is the file the dep came from.
7--
8-- One row per (repository_id, ecosystem, name, manifest_path). We don't
9-- de-dup across manifests (the same dep can legitimately appear in multiple
10-- manifests, e.g. server + client package.json).
11
12--> statement-breakpoint
13CREATE TABLE IF NOT EXISTS "repo_dependencies" (
14 "id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
15 "repository_id" uuid NOT NULL REFERENCES "repositories"("id") ON DELETE CASCADE,
16 "ecosystem" text NOT NULL, -- npm | pypi | go | rubygems | cargo | composer
17 "name" text NOT NULL,
18 "version_spec" text, -- "^1.2.3", ">=2.0", "1.2.3"
19 "manifest_path" text NOT NULL, -- "package.json", "frontend/package.json"
20 "is_dev" boolean NOT NULL DEFAULT false,
21 "commit_sha" text NOT NULL, -- commit the index was built against
22 "indexed_at" timestamp NOT NULL DEFAULT now()
23);
24
25--> statement-breakpoint
26CREATE INDEX IF NOT EXISTS "repo_dependencies_repo_id_idx"
27 ON "repo_dependencies" ("repository_id", "ecosystem");
28
29--> statement-breakpoint
30CREATE INDEX IF NOT EXISTS "repo_dependencies_name_idx"
31 ON "repo_dependencies" ("name");
Addedsrc/__tests__/deps.test.ts+261−0View fileUnifiedSplit
1/**
2 * Block J1 — Dependency graph tests.
3 *
4 * Parser smokes for each supported ecosystem + auth smokes for routes.
5 */
6
7import { describe, it, expect } from "bun:test";
8import app from "../app";
9import {
10 parsePackageJson,
11 parseRequirementsTxt,
12 parsePyprojectToml,
13 parseGoMod,
14 parseCargoToml,
15 parseGemfile,
16 parseComposerJson,
17 parseManifest,
18 isManifestPath,
19 __internal,
20} from "../lib/deps";
21
22describe("deps — isManifestPath", () => {
23 it("recognises supported manifests", () => {
24 expect(isManifestPath("package.json")).toBe(true);
25 expect(isManifestPath("frontend/package.json")).toBe(true);
26 expect(isManifestPath("requirements.txt")).toBe(true);
27 expect(isManifestPath("pyproject.toml")).toBe(true);
28 expect(isManifestPath("go.mod")).toBe(true);
29 expect(isManifestPath("Cargo.toml")).toBe(true);
30 expect(isManifestPath("Gemfile")).toBe(true);
31 expect(isManifestPath("composer.json")).toBe(true);
32 });
33
34 it("rejects unrelated files", () => {
35 expect(isManifestPath("README.md")).toBe(false);
36 expect(isManifestPath("src/index.ts")).toBe(false);
37 expect(isManifestPath("package-lock.json")).toBe(false);
38 });
39});
40
41describe("deps — parsePackageJson", () => {
42 it("extracts dependencies and devDependencies", () => {
43 const deps = parsePackageJson(
44 JSON.stringify({
45 dependencies: { hono: "^4.0.0", "drizzle-orm": "0.30.0" },
46 devDependencies: { typescript: "^5.0.0" },
47 peerDependencies: { react: ">=18" },
48 })
49 );
50 expect(deps).toHaveLength(4);
51 const hono = deps.find((d) => d.name === "hono")!;
52 expect(hono.ecosystem).toBe("npm");
53 expect(hono.versionSpec).toBe("^4.0.0");
54 expect(hono.isDev).toBe(false);
55 const ts = deps.find((d) => d.name === "typescript")!;
56 expect(ts.isDev).toBe(true);
57 });
58
59 it("returns empty on bad JSON", () => {
60 expect(parsePackageJson("not json")).toEqual([]);
61 });
62
63 it("returns empty when no deps", () => {
64 expect(parsePackageJson(JSON.stringify({ name: "x", version: "1" }))).toEqual(
65 []
66 );
67 });
68});
69
70describe("deps — parseRequirementsTxt", () => {
71 it("parses canonical form", () => {
72 const deps = parseRequirementsTxt(
73 `# comment
74requests==2.28.0
75Flask>=2.0,<3.0
76numpy
77pytest # inline comment`
78 );
79 const names = deps.map((d) => d.name);
80 expect(names).toContain("requests");
81 expect(names).toContain("Flask");
82 expect(names).toContain("numpy");
83 expect(names).toContain("pytest");
84 expect(deps.find((d) => d.name === "requests")!.versionSpec).toBe(
85 "==2.28.0"
86 );
87 });
88
89 it("skips blank lines and editable installs", () => {
90 const deps = parseRequirementsTxt(
91 `
92# just a comment
93-e git+https://example.com/foo.git#egg=foo
94--index-url https://pypi.org/simple/
95`
96 );
97 expect(deps).toHaveLength(0);
98 });
99
100 it("handles extras syntax", () => {
101 const deps = parseRequirementsTxt("requests[security]==2.0");
102 expect(deps[0].name).toBe("requests");
103 });
104});
105
106describe("deps — parsePyprojectToml", () => {
107 it("extracts project.dependencies", () => {
108 const deps = parsePyprojectToml(`
109[project]
110name = "myapp"
111dependencies = [
112 "requests>=2.0",
113 "flask"
114]
115`);
116 const names = deps.map((d) => d.name).sort();
117 expect(names).toContain("requests");
118 expect(names).toContain("flask");
119 });
120
121 it("extracts optional-dependencies as dev", () => {
122 const deps = parsePyprojectToml(`
123[project.optional-dependencies]
124dev = ["pytest", "black"]
125`);
126 expect(deps.some((d) => d.name === "pytest" && d.isDev)).toBe(true);
127 });
128});
129
130describe("deps — parseGoMod", () => {
131 it("parses require block", () => {
132 const deps = parseGoMod(`
133module example.com/foo
134
135go 1.21
136
137require (
138 github.com/gorilla/mux v1.8.0
139 github.com/jackc/pgx/v5 v5.4.3 // indirect
140)
141`);
142 expect(deps.length).toBeGreaterThanOrEqual(2);
143 expect(deps.find((d) => d.name === "github.com/gorilla/mux")!.versionSpec)
144 .toBe("v1.8.0");
145 });
146});
147
148describe("deps — parseCargoToml", () => {
149 it("parses [dependencies] and [dev-dependencies]", () => {
150 const deps = parseCargoToml(`
151[package]
152name = "myapp"
153
154[dependencies]
155serde = "1.0"
156tokio = { version = "1.35", features = ["full"] }
157
158[dev-dependencies]
159criterion = "0.5"
160`);
161 const serde = deps.find((d) => d.name === "serde")!;
162 expect(serde.versionSpec).toBe("1.0");
163 expect(serde.isDev).toBe(false);
164 const tokio = deps.find((d) => d.name === "tokio")!;
165 expect(tokio.versionSpec).toBe("1.35");
166 const criterion = deps.find((d) => d.name === "criterion")!;
167 expect(criterion.isDev).toBe(true);
168 });
169});
170
171describe("deps — parseGemfile", () => {
172 it("parses gem lines", () => {
173 const deps = parseGemfile(`
174source "https://rubygems.org"
175
176gem "rails", "7.0.0"
177gem "puma"
178
179group :development, :test do
180 gem "rspec"
181end
182`);
183 const rails = deps.find((d) => d.name === "rails")!;
184 expect(rails.versionSpec).toBe("7.0.0");
185 const rspec = deps.find((d) => d.name === "rspec")!;
186 expect(rspec.isDev).toBe(true);
187 });
188});
189
190describe("deps — parseComposerJson", () => {
191 it("parses require + require-dev, skips php", () => {
192 const deps = parseComposerJson(
193 JSON.stringify({
194 require: { php: ">=8.0", "laravel/framework": "^10.0" },
195 "require-dev": { phpunit: "^10.0" },
196 })
197 );
198 expect(deps.find((d) => d.name === "php")).toBeUndefined();
199 expect(deps.find((d) => d.name === "laravel/framework")).toBeDefined();
200 expect(deps.find((d) => d.name === "phpunit")!.isDev).toBe(true);
201 });
202});
203
204describe("deps — parseManifest (dispatch)", () => {
205 it("routes by basename", () => {
206 expect(
207 parseManifest("frontend/package.json", '{"dependencies":{"x":"1"}}')
208 ).toHaveLength(1);
209 expect(parseManifest("some/go.mod", "require foo v1")).toHaveLength(1);
210 expect(parseManifest("README.md", "# hi")).toHaveLength(0);
211 });
212
213 it("swallows parser errors and returns empty", () => {
214 expect(parseManifest("package.json", "not-json")).toEqual([]);
215 });
216});
217
218describe("deps — TOML helpers", () => {
219 const { splitTomlSections, splitTomlArrayItems, pythonRequirementToDep } =
220 __internal;
221
222 it("splits sections by header", () => {
223 const s = splitTomlSections(`[a]
224x = 1
225
226[b]
227y = 2`);
228 expect(s["a"].trim()).toContain("x = 1");
229 expect(s["b"].trim()).toContain("y = 2");
230 });
231
232 it("splits array items respecting quotes", () => {
233 expect(splitTomlArrayItems('"a, b", "c", "d"')).toEqual([
234 '"a, b"',
235 '"c"',
236 '"d"',
237 ]);
238 });
239
240 it("parses Python requirement specifiers", () => {
241 const d = pythonRequirementToDep("requests>=2.0", false)!;
242 expect(d.name).toBe("requests");
243 expect(d.versionSpec).toBe(">=2.0");
244 });
245});
246
247describe("deps — route auth", () => {
248 it("GET /:owner/:repo/dependencies for unknown repo → 404 or 500", async () => {
249 const res = await app.request("/nobody/missing/dependencies");
250 // Route is mounted — either 404 (repo missing) or 500 (no DB in test env)
251 expect([404, 500]).toContain(res.status);
252 });
253
254 it("POST /:owner/:repo/dependencies/reindex without auth → 302 /login", async () => {
255 const res = await app.request("/alice/repo/dependencies/reindex", {
256 method: "POST",
257 });
258 expect(res.status).toBe(302);
259 expect(res.headers.get("location") || "").toContain("/login");
260 });
261});
Modifiedsrc/app.tsx+4−0View fileUnifiedSplit
6969import symbolsRoutes from "./routes/symbols";
7070import mirrorsRoutes from "./routes/mirrors";
7171import ssoRoutes from "./routes/sso";
72import depsRoutes from "./routes/deps";
7273import webRoutes from "./routes/web";
7374
7475const app = new Hono();
238239// Enterprise SSO via OIDC — /admin/sso + /login/sso (Block I10)
239240app.route("/", ssoRoutes);
240241
242// Dependency graph — /:owner/:repo/dependencies (Block J1)
243app.route("/", depsRoutes);
244
241245// Insights + milestones
242246app.route("/", insightsRoutes);
243247
Modifiedsrc/db/schema.ts+35−0View fileUnifiedSplit
20752075);
20762076
20772077export type SsoUserLink = typeof ssoUserLinks.$inferSelect;
2078
2079// ----------------------------------------------------------------------------
2080// Block J1 — Dependency graph
2081// ----------------------------------------------------------------------------
2082
2083/**
2084 * Last known set of dependencies parsed from manifest files. Each reindex
2085 * replaces the prior rows for that repo. One row per (ecosystem, name,
2086 * manifest_path) — same name in multiple manifests is kept.
2087 */
2088export const repoDependencies = pgTable(
2089 "repo_dependencies",
2090 {
2091 id: uuid("id").primaryKey().defaultRandom(),
2092 repositoryId: uuid("repository_id")
2093 .notNull()
2094 .references(() => repositories.id, { onDelete: "cascade" }),
2095 ecosystem: text("ecosystem").notNull(),
2096 name: text("name").notNull(),
2097 versionSpec: text("version_spec"),
2098 manifestPath: text("manifest_path").notNull(),
2099 isDev: boolean("is_dev").default(false).notNull(),
2100 commitSha: text("commit_sha").notNull(),
2101 indexedAt: timestamp("indexed_at").defaultNow().notNull(),
2102 },
2103 (table) => [
2104 index("repo_dependencies_repo_id_idx").on(
2105 table.repositoryId,
2106 table.ecosystem
2107 ),
2108 index("repo_dependencies_name_idx").on(table.name),
2109 ]
2110);
2111
2112export type RepoDependency = typeof repoDependencies.$inferSelect;
Addedsrc/lib/deps.ts+602−0View fileUnifiedSplit
1/**
2 * Block J1 — Dependency graph.
3 *
4 * Parses common manifest files and records each dependency as a row in
5 * `repo_dependencies`. Per-reindex we REPLACE the whole set for the repo,
6 * so querying the table is always a snapshot.
7 *
8 * Supported ecosystems:
9 * - npm → package.json (dependencies + devDependencies)
10 * - pypi → requirements.txt, pyproject.toml (project.dependencies)
11 * - go → go.mod (require blocks)
12 * - cargo → Cargo.toml ([dependencies] + [dev-dependencies])
13 * - rubygems → Gemfile (gem "name", "version")
14 * - composer → composer.json (require + require-dev)
15 *
16 * We deliberately do not resolve transitive dependencies — we only surface
17 * what's declared in the manifest files. That's the GitHub "dependency
18 * graph" contract: authoritative wrt the repo, not wrt the runtime.
19 */
20
21import { and, desc, eq } from "drizzle-orm";
22import { db } from "../db";
23import {
24 repoDependencies,
25 repositories,
26 users,
27 type RepoDependency,
28} from "../db/schema";
29import {
30 getBlob,
31 getDefaultBranch,
32 getTree,
33 resolveRef,
34 type GitTreeEntry,
35} from "../git/repository";
36
37// ----------------------------------------------------------------------------
38// Types
39// ----------------------------------------------------------------------------
40
41export type Ecosystem =
42 | "npm"
43 | "pypi"
44 | "go"
45 | "rubygems"
46 | "cargo"
47 | "composer";
48
49export interface ParsedDep {
50 ecosystem: Ecosystem;
51 name: string;
52 versionSpec: string | null;
53 isDev: boolean;
54}
55
56/**
57 * Filename → (content) → ParsedDep[]. Keys are the basename; we match
58 * case-insensitively. `manifestPath` (the full path) is attached at a
59 * higher level.
60 */
61const PARSERS: Record<string, (content: string) => ParsedDep[]> = {
62 "package.json": parsePackageJson,
63 "requirements.txt": parseRequirementsTxt,
64 "pyproject.toml": parsePyprojectToml,
65 "go.mod": parseGoMod,
66 "cargo.toml": parseCargoToml,
67 gemfile: parseGemfile,
68 "composer.json": parseComposerJson,
69};
70
71// ----------------------------------------------------------------------------
72// Individual parsers — each is defensive; one bad file does not abort indexing
73// ----------------------------------------------------------------------------
74
75export function parsePackageJson(content: string): ParsedDep[] {
76 const out: ParsedDep[] = [];
77 let obj: any;
78 try {
79 obj = JSON.parse(content);
80 } catch {
81 return out;
82 }
83 if (!obj || typeof obj !== "object") return out;
84 for (const [key, isDev] of [
85 ["dependencies", false],
86 ["devDependencies", true],
87 ["peerDependencies", false],
88 ["optionalDependencies", false],
89 ] as const) {
90 const deps = obj[key];
91 if (!deps || typeof deps !== "object") continue;
92 for (const [name, spec] of Object.entries(deps)) {
93 if (typeof name !== "string" || !name) continue;
94 out.push({
95 ecosystem: "npm",
96 name,
97 versionSpec: typeof spec === "string" ? spec : null,
98 isDev,
99 });
100 }
101 }
102 return out;
103}
104
105export function parseRequirementsTxt(content: string): ParsedDep[] {
106 const out: ParsedDep[] = [];
107 for (const rawLine of content.split(/\r?\n/)) {
108 const line = rawLine.split("#")[0].trim();
109 if (!line) continue;
110 // Skip editable / url installs
111 if (line.startsWith("-e ") || line.startsWith("--") || line.startsWith("git+")) continue;
112 // Match "name", "name==1.2.3", "name>=1.0,<2.0", "name[extra]==1.0"
113 const m = line.match(
114 /^([A-Za-z0-9_][A-Za-z0-9_.\-]*)(?:\[[^\]]+\])?\s*([<>=!~].+)?$/
115 );
116 if (!m) continue;
117 out.push({
118 ecosystem: "pypi",
119 name: m[1],
120 versionSpec: m[2] ? m[2].trim() : null,
121 isDev: false,
122 });
123 }
124 return out;
125}
126
127export function parsePyprojectToml(content: string): ParsedDep[] {
128 const out: ParsedDep[] = [];
129 // We only look for `[project]` → `dependencies = [...]` and
130 // `[project.optional-dependencies]` → `dev = [...]`. Full TOML parsing
131 // is overkill — a regex-over-sections pass is sufficient here.
132 const sections = splitTomlSections(content);
133 const projectDeps = extractTomlArray(sections.project, "dependencies");
134 for (const dep of projectDeps) {
135 const parsed = pythonRequirementToDep(dep, false);
136 if (parsed) out.push(parsed);
137 }
138 const optDeps = sections["project.optional-dependencies"] || "";
139 for (const line of optDeps.split(/\r?\n/)) {
140 const keyMatch = line.match(/^\s*([A-Za-z0-9_-]+)\s*=\s*\[/);
141 if (keyMatch) {
142 // Consume until closing ]
143 const start = line.indexOf("[");
144 const tail = line.slice(start);
145 const closeIdx = tail.indexOf("]");
146 if (closeIdx > -1) {
147 const inner = tail.slice(1, closeIdx);
148 for (const item of splitTomlArrayItems(inner)) {
149 const parsed = pythonRequirementToDep(item, true);
150 if (parsed) out.push(parsed);
151 }
152 }
153 }
154 }
155 return out;
156}
157
158function pythonRequirementToDep(
159 raw: string,
160 isDev: boolean
161): ParsedDep | null {
162 const s = raw.trim().replace(/^["']|["']$/g, "");
163 if (!s) return null;
164 const m = s.match(
165 /^([A-Za-z0-9_][A-Za-z0-9_.\-]*)(?:\[[^\]]+\])?\s*([<>=!~].+)?$/
166 );
167 if (!m) return null;
168 return {
169 ecosystem: "pypi",
170 name: m[1],
171 versionSpec: m[2] ? m[2].trim() : null,
172 isDev,
173 };
174}
175
176export function parseGoMod(content: string): ParsedDep[] {
177 const out: ParsedDep[] = [];
178 // `require (` block or single-line `require foo v1.0.0`
179 const blockMatch = content.match(/require\s*\(([\s\S]*?)\)/);
180 const lines: string[] = [];
181 if (blockMatch) {
182 lines.push(...blockMatch[1].split(/\r?\n/));
183 }
184 for (const rawLine of content.split(/\r?\n/)) {
185 const m = rawLine.match(/^\s*require\s+(\S+)\s+(\S+)/);
186 if (m) lines.push(`${m[1]} ${m[2]}`);
187 }
188 for (const rawLine of lines) {
189 const line = rawLine.split("//")[0].trim();
190 if (!line) continue;
191 const m = line.match(/^(\S+)\s+(\S+)(\s+\/\/ indirect)?$/);
192 if (!m) continue;
193 out.push({
194 ecosystem: "go",
195 name: m[1],
196 versionSpec: m[2],
197 isDev: false,
198 });
199 }
200 return out;
201}
202
203export function parseCargoToml(content: string): ParsedDep[] {
204 const out: ParsedDep[] = [];
205 const sections = splitTomlSections(content);
206 for (const [header, isDev] of [
207 ["dependencies", false],
208 ["dev-dependencies", true],
209 ["build-dependencies", false],
210 ] as const) {
211 const body = sections[header];
212 if (!body) continue;
213 for (const line of body.split(/\r?\n/)) {
214 const trimmed = line.split("#")[0].trim();
215 if (!trimmed) continue;
216 // foo = "1.2.3" OR foo = { version = "1.2.3" }
217 const m = trimmed.match(/^([A-Za-z0-9_-]+)\s*=\s*(.+)$/);
218 if (!m) continue;
219 const name = m[1];
220 const rhs = m[2].trim();
221 let versionSpec: string | null = null;
222 if (rhs.startsWith('"') || rhs.startsWith("'")) {
223 const q = rhs[0];
224 const end = rhs.indexOf(q, 1);
225 if (end > 0) versionSpec = rhs.slice(1, end);
226 } else if (rhs.startsWith("{")) {
227 const vm = rhs.match(/version\s*=\s*["']([^"']+)["']/);
228 if (vm) versionSpec = vm[1];
229 }
230 out.push({
231 ecosystem: "cargo",
232 name,
233 versionSpec,
234 isDev,
235 });
236 }
237 }
238 return out;
239}
240
241export function parseGemfile(content: string): ParsedDep[] {
242 const out: ParsedDep[] = [];
243 let inDevGroup = false;
244 for (const rawLine of content.split(/\r?\n/)) {
245 const line = rawLine.split("#")[0].trim();
246 if (!line) continue;
247 // group :development, :test do
248 if (/^group\s+.*(:development|:test)/i.test(line)) {
249 inDevGroup = true;
250 continue;
251 }
252 if (/^end\b/.test(line)) {
253 inDevGroup = false;
254 continue;
255 }
256 const m = line.match(
257 /^gem\s+["']([^"']+)["'](?:\s*,\s*["']([^"']+)["'])?/
258 );
259 if (!m) continue;
260 out.push({
261 ecosystem: "rubygems",
262 name: m[1],
263 versionSpec: m[2] || null,
264 isDev: inDevGroup,
265 });
266 }
267 return out;
268}
269
270export function parseComposerJson(content: string): ParsedDep[] {
271 const out: ParsedDep[] = [];
272 let obj: any;
273 try {
274 obj = JSON.parse(content);
275 } catch {
276 return out;
277 }
278 if (!obj || typeof obj !== "object") return out;
279 for (const [key, isDev] of [
280 ["require", false],
281 ["require-dev", true],
282 ] as const) {
283 const deps = obj[key];
284 if (!deps || typeof deps !== "object") continue;
285 for (const [name, spec] of Object.entries(deps)) {
286 if (!name || name === "php") continue;
287 out.push({
288 ecosystem: "composer",
289 name,
290 versionSpec: typeof spec === "string" ? spec : null,
291 isDev,
292 });
293 }
294 }
295 return out;
296}
297
298// ----------------------------------------------------------------------------
299// TOML helpers — intentionally minimal
300// ----------------------------------------------------------------------------
301
302function splitTomlSections(content: string): Record<string, string> {
303 const out: Record<string, string> = { __root__: "" };
304 let current = "__root__";
305 for (const rawLine of content.split(/\r?\n/)) {
306 const headerMatch = rawLine.match(/^\s*\[([^\]]+)\]\s*$/);
307 if (headerMatch) {
308 current = headerMatch[1].trim();
309 if (!out[current]) out[current] = "";
310 continue;
311 }
312 out[current] = (out[current] || "") + rawLine + "\n";
313 }
314 // Also expose `project` when it appears at the root.
315 if (!out["project"] && out["__root__"]) {
316 const m = out["__root__"].match(/\[project\]([\s\S]*)/);
317 if (m) out["project"] = m[1];
318 }
319 return out;
320}
321
322function extractTomlArray(body: string | undefined, key: string): string[] {
323 if (!body) return [];
324 const start = body.search(new RegExp(`^\\s*${key}\\s*=\\s*\\[`, "m"));
325 if (start < 0) return [];
326 const open = body.indexOf("[", start);
327 if (open < 0) return [];
328 const close = findMatchingBracket(body, open);
329 if (close < 0) return [];
330 const inner = body.slice(open + 1, close);
331 return splitTomlArrayItems(inner);
332}
333
334function findMatchingBracket(s: string, open: number): number {
335 let depth = 0;
336 for (let i = open; i < s.length; i++) {
337 if (s[i] === "[") depth++;
338 else if (s[i] === "]") {
339 depth--;
340 if (depth === 0) return i;
341 }
342 }
343 return -1;
344}
345
346function splitTomlArrayItems(inner: string): string[] {
347 // Strip newlines + split by commas; handle simple quoted strings.
348 const items: string[] = [];
349 let cur = "";
350 let inQ: string | null = null;
351 for (const ch of inner) {
352 if (inQ) {
353 cur += ch;
354 if (ch === inQ) inQ = null;
355 continue;
356 }
357 if (ch === '"' || ch === "'") {
358 inQ = ch;
359 cur += ch;
360 continue;
361 }
362 if (ch === ",") {
363 if (cur.trim()) items.push(cur);
364 cur = "";
365 continue;
366 }
367 cur += ch;
368 }
369 if (cur.trim()) items.push(cur);
370 return items.map((s) => s.trim()).filter(Boolean);
371}
372
373// ----------------------------------------------------------------------------
374// File detection + tree walk
375// ----------------------------------------------------------------------------
376
377const MANIFEST_BASENAMES = new Set(Object.keys(PARSERS));
378const MAX_MANIFEST_BYTES = 1_000_000;
379const MAX_MANIFESTS = 200;
380
381export function isManifestPath(path: string): boolean {
382 const base = path.split("/").pop()?.toLowerCase() || "";
383 return MANIFEST_BASENAMES.has(base);
384}
385
386export function parseManifest(
387 path: string,
388 content: string
389): ParsedDep[] {
390 const base = path.split("/").pop()?.toLowerCase() || "";
391 const parser = PARSERS[base];
392 if (!parser) return [];
393 try {
394 return parser(content);
395 } catch {
396 return [];
397 }
398}
399
400async function walkManifestPaths(
401 owner: string,
402 repo: string,
403 ref: string
404): Promise<Array<{ path: string; size?: number }>> {
405 const out: Array<{ path: string; size?: number }> = [];
406 const queue: string[] = [""];
407 while (queue.length && out.length < MAX_MANIFESTS) {
408 const dir = queue.shift()!;
409 let entries: GitTreeEntry[] = [];
410 try {
411 entries = await getTree(owner, repo, ref, dir);
412 } catch {
413 continue;
414 }
415 for (const e of entries) {
416 const p = dir ? `${dir}/${e.name}` : e.name;
417 if (e.type === "tree") {
418 const base = e.name.toLowerCase();
419 if (
420 base === "node_modules" ||
421 base === ".git" ||
422 base === "dist" ||
423 base === "build" ||
424 base === "vendor" ||
425 base === "target" ||
426 base === "__pycache__"
427 ) {
428 continue;
429 }
430 queue.push(p);
431 } else if (e.type === "blob") {
432 if (!isManifestPath(p)) continue;
433 if (e.size !== undefined && e.size > MAX_MANIFEST_BYTES) continue;
434 out.push({ path: p, size: e.size });
435 if (out.length >= MAX_MANIFESTS) break;
436 }
437 }
438 }
439 return out;
440}
441
442// ----------------------------------------------------------------------------
443// Reindex + queries
444// ----------------------------------------------------------------------------
445
446export async function indexRepositoryDependencies(
447 repositoryId: string
448): Promise<
449 | {
450 indexed: number;
451 manifests: number;
452 commitSha: string;
453 }
454 | null
455> {
456 try {
457 const [repo] = await db
458 .select()
459 .from(repositories)
460 .where(eq(repositories.id, repositoryId))
461 .limit(1);
462 if (!repo) return null;
463
464 const [owner] = await db
465 .select({ username: users.username })
466 .from(users)
467 .where(eq(users.id, repo.ownerId))
468 .limit(1);
469 if (!owner) return null;
470
471 const defaultBranch =
472 (await getDefaultBranch(owner.username, repo.name)) || "main";
473 const head = await resolveRef(owner.username, repo.name, defaultBranch);
474 if (!head) return null;
475
476 const manifests = await walkManifestPaths(
477 owner.username,
478 repo.name,
479 head
480 );
481
482 const rows: Array<Omit<RepoDependency, "id" | "indexedAt">> = [];
483 for (const f of manifests) {
484 const blob = await getBlob(owner.username, repo.name, head, f.path).catch(
485 () => null
486 );
487 if (!blob) continue;
488 const content =
489 typeof blob === "string"
490 ? blob
491 : new TextDecoder().decode(blob as any);
492 const deps = parseManifest(f.path, content);
493 for (const dep of deps) {
494 rows.push({
495 repositoryId,
496 ecosystem: dep.ecosystem,
497 name: dep.name,
498 versionSpec: dep.versionSpec,
499 manifestPath: f.path,
500 isDev: dep.isDev,
501 commitSha: head,
502 });
503 }
504 }
505
506 await db
507 .delete(repoDependencies)
508 .where(eq(repoDependencies.repositoryId, repositoryId));
509
510 // Insert in chunks to stay under parameter limits.
511 const CHUNK = 500;
512 for (let i = 0; i < rows.length; i += CHUNK) {
513 const slice = rows.slice(i, i + CHUNK);
514 if (slice.length) await db.insert(repoDependencies).values(slice);
515 }
516
517 return {
518 indexed: rows.length,
519 manifests: manifests.length,
520 commitSha: head,
521 };
522 } catch (err) {
523 console.error("[deps] indexRepositoryDependencies:", err);
524 return null;
525 }
526}
527
528export async function listDependenciesForRepo(
529 repositoryId: string
530): Promise<RepoDependency[]> {
531 try {
532 return await db
533 .select()
534 .from(repoDependencies)
535 .where(eq(repoDependencies.repositoryId, repositoryId))
536 .orderBy(desc(repoDependencies.ecosystem), desc(repoDependencies.name));
537 } catch {
538 return [];
539 }
540}
541
542export interface EcosystemSummary {
543 ecosystem: string;
544 count: number;
545}
546
547export async function summarizeDependencies(
548 repositoryId: string
549): Promise<EcosystemSummary[]> {
550 const rows = await listDependenciesForRepo(repositoryId);
551 const counts = new Map<string, number>();
552 for (const r of rows) {
553 counts.set(r.ecosystem, (counts.get(r.ecosystem) || 0) + 1);
554 }
555 return Array.from(counts.entries())
556 .map(([ecosystem, count]) => ({ ecosystem, count }))
557 .sort((a, b) => b.count - a.count);
558}
559
560/** Look up reverse deps — which repos list this package? (Network graph.) */
561export async function repositoriesDependingOn(
562 ecosystem: string,
563 name: string,
564 limit = 50
565): Promise<
566 Array<{
567 repositoryId: string;
568 versionSpec: string | null;
569 manifestPath: string;
570 }>
571> {
572 try {
573 const rows = await db
574 .select({
575 repositoryId: repoDependencies.repositoryId,
576 versionSpec: repoDependencies.versionSpec,
577 manifestPath: repoDependencies.manifestPath,
578 })
579 .from(repoDependencies)
580 .where(
581 and(
582 eq(repoDependencies.ecosystem, ecosystem),
583 eq(repoDependencies.name, name)
584 )
585 )
586 .limit(limit);
587 return rows;
588 } catch {
589 return [];
590 }
591}
592
593// ----------------------------------------------------------------------------
594// Test-only exports
595// ----------------------------------------------------------------------------
596
597export const __internal = {
598 splitTomlSections,
599 extractTomlArray,
600 splitTomlArrayItems,
601 pythonRequirementToDep,
602};
Addedsrc/routes/deps.tsx+233−0View fileUnifiedSplit
1/**
2 * Block J1 — Dependency graph routes.
3 *
4 * GET /:owner/:repo/dependencies — grouped list + summary
5 * POST /:owner/:repo/dependencies/reindex — owner-only, walk manifests
6 */
7
8import { Hono } from "hono";
9import { and, eq } from "drizzle-orm";
10import { db } from "../db";
11import { repositories, users } from "../db/schema";
12import { Layout } from "../views/layout";
13import { RepoHeader, RepoNav } from "../views/components";
14import { softAuth, requireAuth } from "../middleware/auth";
15import type { AuthEnv } from "../middleware/auth";
16import {
17 indexRepositoryDependencies,
18 listDependenciesForRepo,
19 summarizeDependencies,
20} from "../lib/deps";
21
22const deps = new Hono<AuthEnv>();
23deps.use("*", softAuth);
24
25async function loadRepo(ownerName: string, repoName: string) {
26 const [owner] = await db
27 .select()
28 .from(users)
29 .where(eq(users.username, ownerName))
30 .limit(1);
31 if (!owner) return null;
32 const [repo] = await db
33 .select()
34 .from(repositories)
35 .where(
36 and(eq(repositories.ownerId, owner.id), eq(repositories.name, repoName))
37 )
38 .limit(1);
39 if (!repo) return null;
40 return { owner, repo };
41}
42
43// ---------- Overview ----------
44
45deps.get("/:owner/:repo/dependencies", async (c) => {
46 const user = c.get("user");
47 const { owner: ownerName, repo: repoName } = c.req.param();
48 const ctx = await loadRepo(ownerName, repoName);
49 if (!ctx) return c.notFound();
50 const { repo } = ctx;
51 if (repo.isPrivate && (!user || user.id !== repo.ownerId)) {
52 return c.notFound();
53 }
54
55 const all = await listDependenciesForRepo(repo.id);
56 const summary = await summarizeDependencies(repo.id);
57 const isOwner = !!user && user.id === repo.ownerId;
58 const message = c.req.query("message");
59 const error = c.req.query("error");
60
61 // Group by ecosystem
62 const grouped = new Map<string, typeof all>();
63 for (const d of all) {
64 const list = grouped.get(d.ecosystem) || [];
65 list.push(d);
66 grouped.set(d.ecosystem, list);
67 }
68
69 return c.html(
70 <Layout
71 title={`Dependencies — ${ownerName}/${repoName}`}
72 user={user}
73 >
74 <RepoHeader owner={ownerName} repo={repoName} />
75 <RepoNav owner={ownerName} repo={repoName} active="code" />
76 <div class="settings-container">
77 <div style="display:flex;justify-content:space-between;align-items:center">
78 <h2 style="margin:0">Dependencies</h2>
79 {isOwner && (
80 <form
81 method="POST"
82 action={`/${ownerName}/${repoName}/dependencies/reindex`}
83 >
84 <button type="submit" class="btn btn-primary btn-sm">
85 Reindex
86 </button>
87 </form>
88 )}
89 </div>
90 {message && (
91 <div class="auth-success" style="margin-top:12px">
92 {decodeURIComponent(message)}
93 </div>
94 )}
95 {error && (
96 <div class="auth-error" style="margin-top:12px">
97 {decodeURIComponent(error)}
98 </div>
99 )}
100 <p style="color:var(--text-muted);margin-top:8px">
101 Parsed from <code>package.json</code>, <code>requirements.txt</code>,{" "}
102 <code>pyproject.toml</code>, <code>go.mod</code>,{" "}
103 <code>Cargo.toml</code>, <code>Gemfile</code>, and{" "}
104 <code>composer.json</code> on the default branch. Transitive
105 dependencies are not resolved.
106 </p>
107
108 {all.length === 0 ? (
109 <div class="panel-empty" style="padding:24px">
110 No dependencies indexed yet.
111 {isOwner && " Click Reindex to scan the repository."}
112 </div>
113 ) : (
114 <>
115 <div
116 style="display:grid;grid-template-columns:repeat(auto-fit,minmax(140px,1fr));gap:8px;margin:16px 0"
117 >
118 <div class="panel" style="padding:12px;text-align:center">
119 <div style="font-size:20px;font-weight:700">
120 {all.length}
121 </div>
122 <div
123 style="font-size:11px;color:var(--text-muted);text-transform:uppercase"
124 >
125 Dependencies
126 </div>
127 </div>
128 {summary.map((s) => (
129 <div class="panel" style="padding:12px;text-align:center">
130 <div style="font-size:20px;font-weight:700">
131 {s.count}
132 </div>
133 <div
134 style="font-size:11px;color:var(--text-muted);text-transform:uppercase"
135 >
136 {s.ecosystem}
137 </div>
138 </div>
139 ))}
140 </div>
141
142 {Array.from(grouped.entries()).map(([ecosystem, list]) => (
143 <>
144 <h3 style="margin-top:20px">
145 {ecosystem}{" "}
146 <span
147 style="font-size:12px;color:var(--text-muted);font-weight:400"
148 >
149 ({list.length})
150 </span>
151 </h3>
152 <div class="panel" style="margin-bottom:12px">
153 {list.map((d) => (
154 <div
155 class="panel-item"
156 style="justify-content:space-between;flex-wrap:wrap;gap:6px"
157 >
158 <div>
159 <span
160 style="font-family:var(--font-mono);font-weight:600"
161 >
162 {d.name}
163 </span>
164 {d.versionSpec && (
165 <span
166 style="margin-left:8px;font-size:12px;color:var(--text-muted);font-family:var(--font-mono)"
167 >
168 {d.versionSpec}
169 </span>
170 )}
171 {d.isDev && (
172 <span
173 style="margin-left:8px;font-size:10px;padding:2px 6px;background:var(--bg-subtle);border-radius:3px;color:var(--text-muted);text-transform:uppercase"
174 >
175 dev
176 </span>
177 )}
178 </div>
179 <a
180 href={`/${ownerName}/${repoName}/blob/HEAD/${d.manifestPath}`}
181 style="font-size:12px;color:var(--text-muted);font-family:var(--font-mono)"
182 >
183 {d.manifestPath}
184 </a>
185 </div>
186 ))}
187 </div>
188 </>
189 ))}
190 </>
191 )}
192 </div>
193 </Layout>
194 );
195});
196
197// ---------- Reindex (owner-only) ----------
198
199deps.post("/:owner/:repo/dependencies/reindex", requireAuth, async (c) => {
200 const user = c.get("user");
201 const { owner: ownerName, repo: repoName } = c.req.param();
202 const ctx = await loadRepo(ownerName, repoName);
203 if (!ctx) return c.notFound();
204 const { repo } = ctx;
205 if (!user || user.id !== repo.ownerId) {
206 return c.html(
207 <Layout title="Forbidden" user={user}>
208 <div class="empty-state">
209 <h2>403</h2>
210 <p>Only the repository owner can reindex dependencies.</p>
211 </div>
212 </Layout>,
213 403
214 );
215 }
216
217 const result = await indexRepositoryDependencies(repo.id);
218 const to = `/${ownerName}/${repoName}/dependencies`;
219 if (!result) {
220 return c.redirect(
221 `${to}?error=${encodeURIComponent(
222 "Reindex failed — is the default branch empty?"
223 )}`
224 );
225 }
226 return c.redirect(
227 `${to}?message=${encodeURIComponent(
228 `Indexed ${result.indexed} dependencies across ${result.manifests} manifests.`
229 )}`
230 );
231});
232
233export default deps;
0234