Pre-launch — Gluecron is in final validation. Public signups and git hosting for non-owner users open after launch review.
ccantynz/Gluecron.com
Commit892cb53unknown_key

feat(BLOCK-L): GitHub importer — pull existing repos into Gluecron

feat(BLOCK-L): GitHub importer — pull existing repos into Gluecron

- drizzle/0039_github_imports.sql — per-run ledger
- src/lib/github-import.ts — pure walkers + mappers + runImport
  orchestrator; paginated Link-header follower with per-endpoint caps;
  authed clone URL builder that rejects CR/LF + URL-encodes the token;
  never-throws contract
- src/routes/github-import.tsx — GET/POST /new/import synchronous
  clone --mirror + metadata walk + green bootstrap + audit log;
  GET /api/imports/:id JSON poll
- schema.ts: githubImports table
- app.tsx: mounted before webRoutes catch-all
- 31 new tests (1098/1098 pass)

BUILD_BIBLE updated with §4.8 Block L locked files + §3 plan row.
Claude committed on April 17, 2026Parent: 8c9470a
7 files changed+1619−4892cb5323c93bc676cfa954e6ffcd784d0af6ceb
7 changed files+1619−4
ModifiedBUILD_BIBLE.md+13−4View fileUnifiedSplit
328328- **K11** — Cross-product identity (Gluecron/Crontech/Gatetest SSO) → ✅ shipped. `drizzle/0038_cross_product_tokens.sql` + `src/lib/cross-product-auth.ts` (JWT HS256 with jti revocation list) + `src/routes/cross-product.tsx` (`/settings/cross-product` issue + revoke UI). Audience vocabulary: `gluecron | crontech | gatetest`. 35 tests.
329329- **K12** — Heal-bot scheduler → ✅ shipped as `runHealBotForAll` in `src/lib/agents/heal-bot.ts`; triggered by workflow cron or manual admin action.
330330
331### BLOCK L — GitHub importer (migration onboarding)
332Needed for dogfooding: pull existing repos + their metadata out of GitHub into Gluecron so every project the owner already runs (Crontech, Gatetest, etc.) can live here.
333- **L1** — Importer lib + route → ✅ shipped. `drizzle/0039_github_imports.sql` + `src/lib/github-import.ts` (pure walkers against `api.github.com` with per-endpoint caps, mappers for labels/issues/pulls/comments/releases, PAT-authed `git clone --mirror` URL builder that rejects CR/LF + URL-encodes the token) + `src/routes/github-import.tsx` (`GET /new/import` form, `POST /new/import` synchronous clone + walk + redirect, `GET /api/imports/:id` JSON status poll). Green-by-default bootstrap runs on import just like `/new`. 31 new tests.
334
331335---
332336
333337## 4. LOCKED BLOCKS (DO NOT UNDO)
377381- `drizzle/0036_repo_agent_settings.sql` (Block K8) — migration, never edited in place. Adds `repo_agent_settings` (per-repo per-agent enable/disable + JSON config, unique on `(repository_id, agent_slug)`).
378382- `drizzle/0037_marketplace_agent_listings.sql` (Block K10) — migration, never edited in place. Adds `marketplace_agent_listings` (slug unique, publisher/app_bot FKs, kind enum, pricing_cents_per_month, published flag, install_count).
379383- `drizzle/0038_cross_product_tokens.sql` (Block K11) — migration, never edited in place. Adds `cross_product_tokens` (jti PK, audience, scopes JSON, issued/expires/revoked-at) for JWT jti revocation across Gluecron / Crontech / Gatetest.
384- `drizzle/0039_github_imports.sql` (Block L) — migration, never edited in place. Adds `github_imports` (per-run import ledger: source owner/repo, status pending→cloning→walking→ok|error, JSON stats, started/finished timestamps).
380385
381386### 4.2 Git layer (locked)
382387- `src/git/repository.ts` — tree / blob / commits / diff / branches / blame / search / raw / tags / commitsBetween
530535- `src/routes/agent-marketplace.tsx` (Block K10) — public agent directory + listing publisher flow. Pure exports: `parseListingForm`, `ALLOWED_LISTING_KINDS` (`triage|fix|review|heal_bot|deploy_watch|custom`), `SLUG_RE = /^[a-z][a-z0-9-]{2,48}$/`, `TAGLINE_MAX=200`, `DESCRIPTION_MAX=5000`, `PRICING_MAX_CENTS=100_000`. Routes: `GET /marketplace/agents` (public), `GET /marketplace/agents/:slug` (detail), `POST /marketplace/agents/:slug/install|uninstall` (requireAuth), `GET+POST /settings/agent-listings` + `POST /settings/agent-listings/:id/publish|unpublish` (site-admin), `GET /admin/marketplace/agents`. **Must be mounted BEFORE `adminRoutes` and `marketplaceRoutes` in `src/app.tsx` so `/marketplace/:slug` and `/admin/marketplace/*` resolve to the agent handlers rather than the generic `:slug` routes in marketplace.tsx / admin.tsx.**
531536- `src/routes/cross-product.tsx` (Block K11) — token issuance + revocation UI. `GET /settings/cross-product` lists active tokens; `POST /settings/cross-product/issue` mints a JWT for `audience ∈ {crontech, gatetest}` (shown once); `POST /settings/cross-product/:jti/revoke`. All mutations `audit()`-logged. Never-throws contract on verification.
532537
533### 4.8 Views (locked contracts)
538### 4.8 Block L — GitHub importer (locked)
539- `src/lib/github-import.ts` (Block L) — pure helpers + orchestrator. Exports `buildAuthedCloneUrl` (rejects whitespace/CRLF, URL-encodes token), `redactCloneUrl`, `ghFetch`, `parseNextLink`, `paginate<T>(token, url, cap, fetchImpl?)` (walks `Link: rel=next` up to cap, 20-page safety ceiling). Endpoint walkers `fetchRepo`, `fetchLabels`, `fetchIssuesAndPulls`, `fetchPullRequests`, `fetchIssueComments`, `fetchReleases`. Pure mappers `normaliseColor` (falls back `#8b949e`), `mapLabel`, `mapIssue`, `mapPull` (merged > closed > open precedence), `mapRelease`. Orchestrator `runImport({ token, sourceOwner, sourceRepo, targetRepoId, importerUserId, caps?, fetchImpl? })` synchronously walks + inserts labels → issues+labels+comments → pulls+comments → releases → stargazers (count bump), per-endpoint caps (`DEFAULT_CAPS = { labels: 200, issues: 200, pulls: 100, issueComments: 500, prComments: 500, releases: 50, stargazers: 200 }`). Ledger helpers `createImportRow`, `finaliseImportRow`. Never throws.
540- `src/routes/github-import.tsx` (Block L) — `GET /new/import` form (requireAuth, shows last-10 imports by user); `POST /new/import` validates `owner/repo` source + target name, rejects dupes, `buildAuthedCloneUrl` → `Bun.spawn(["git","clone","--mirror",...])` with 10-min timeout + `GIT_TERMINAL_PROMPT=0`, inserts target repository row, calls `bootstrapRepository` (green-by-default, skipWelcomeIssue), runs `runImport`, updates ledger, `audit("github_import", { source, stats, walkError })`, redirects to new repo. `GET /api/imports/:id` returns JSON status (requireAuth, owner-only).
541
542### 4.9 Views (locked contracts)
534543- `src/views/layout.tsx` — `Layout` accepts `title`, `user`, `notificationCount`
535544- `src/views/components.tsx` — `RepoHeader`, `RepoNav` (active: `code|issues|pulls|commits|releases|actions|gates|insights|explain|changelog|semantic`), `RepoCard`, etc.
536545- `src/views/reactions.tsx` — `ReactionsBar` (no-JS compatible, form-per-emoji)
537546- Nav links: logo · search · theme-toggle · Explore · Ask · Notifications · New · Profile (or Sign in / Register)
538547- Keyboard chords: `/`, `Cmd+K`, `?`, `n`, `g d`, `g n`, `g e`, `g a`
539548
540### 4.9 Tests (locked)
549### 4.10 Tests (locked)
541550- `src/__tests__/green-ecosystem.test.ts` — secret scanner, codeowners, AI fallback, health, rate-limit headers, `/shortcuts`, `/search`
542551- All other existing test files — do not delete without owner permission
543552
544### 4.10 Invariants (never break these)
553### 4.11 Invariants (never break these)
545554- `isAiAvailable()` guard returns true fallback strings when no ANTHROPIC_API_KEY. AI features degrade gracefully.
546555- `getUnreadCount` never throws; returns 0 on any error.
547556- Rate-limit middleware adds `X-RateLimit-Limit` + `X-RateLimit-Remaining` to every response, including 500s.
563572```bash
564573bun install
565574bun dev # hot reload
566bun test # 1067 tests currently pass
575bun test # 1098 tests currently pass
567576bun run db:migrate
568577```
569578
Addeddrizzle/0039_github_imports.sql+24−0View fileUnifiedSplit
1-- Block L — GitHub importer job ledger.
2--
3-- Tracks a single "import from GitHub" attempt. One row per POST /new/import.
4-- stats JSON holds per-endpoint counts (labels/issues/pulls/comments/releases/stars).
5-- status progresses: pending → cloning → walking → ok|error.
6
7CREATE TABLE IF NOT EXISTS "github_imports" (
8 "id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
9 "repository_id" uuid REFERENCES "repositories"("id") ON DELETE CASCADE,
10 "user_id" uuid NOT NULL REFERENCES "users"("id") ON DELETE CASCADE,
11 "source_owner" text NOT NULL,
12 "source_repo" text NOT NULL,
13 "status" text NOT NULL DEFAULT 'pending', -- pending|cloning|walking|ok|error
14 "stats" text NOT NULL DEFAULT '{}',
15 "error" text,
16 "started_at" timestamp NOT NULL DEFAULT now(),
17 "finished_at" timestamp
18);
19
20CREATE INDEX IF NOT EXISTS "github_imports_user_idx"
21 ON "github_imports" ("user_id", "started_at" DESC);
22
23CREATE INDEX IF NOT EXISTS "github_imports_repo_idx"
24 ON "github_imports" ("repository_id");
Addedsrc/__tests__/github-import.test.ts+378−0View fileUnifiedSplit
1/**
2 * Block L — GitHub importer tests.
3 *
4 * Tests focus on the pure helpers (URL construction, pagination, filtering,
5 * mappers). DB-backed runImport is exercised via the integration suite.
6 */
7
8import { describe, it, expect } from "bun:test";
9import app from "../app";
10import {
11 buildAuthedCloneUrl,
12 filterIssuesOnly,
13 mapIssue,
14 mapLabel,
15 mapPull,
16 mapRelease,
17 normaliseColor,
18 paginate,
19 parseNextLink,
20 redactCloneUrl,
21 type GhIssue,
22 type GhPull,
23 type GhRelease,
24} from "../lib/github-import";
25
26// ---------------------------------------------------------------------------
27// buildAuthedCloneUrl
28// ---------------------------------------------------------------------------
29
30describe("buildAuthedCloneUrl", () => {
31 it("builds https clone URL with encoded token", () => {
32 const r = buildAuthedCloneUrl("ghp_abc123", "octocat", "hello");
33 expect(r.ok).toBe(true);
34 if (r.ok) {
35 expect(r.data).toBe(
36 "https://x-access-token:ghp_abc123@github.com/octocat/hello.git"
37 );
38 }
39 });
40
41 it("rejects tokens containing whitespace/CRLF", () => {
42 expect(buildAuthedCloneUrl("abc\ndef", "o", "r").ok).toBe(false);
43 expect(buildAuthedCloneUrl("abc def", "o", "r").ok).toBe(false);
44 expect(buildAuthedCloneUrl("abc\rdef", "o", "r").ok).toBe(false);
45 });
46
47 it("rejects invalid owner/repo", () => {
48 expect(buildAuthedCloneUrl("t", "bad owner", "r").ok).toBe(false);
49 expect(buildAuthedCloneUrl("t", "good", "bad/repo").ok).toBe(false);
50 });
51
52 it("url-encodes special chars in the token", () => {
53 const r = buildAuthedCloneUrl("a%b", "o", "r");
54 expect(r.ok).toBe(true);
55 if (r.ok) expect(r.data).toContain("a%25b");
56 });
57});
58
59describe("redactCloneUrl", () => {
60 it("redacts token from authed URL", () => {
61 const input =
62 "fatal: https://x-access-token:ghp_abc@github.com/o/r.git not found";
63 expect(redactCloneUrl(input)).toBe(
64 "fatal: https://***@github.com/o/r.git not found"
65 );
66 });
67 it("is a no-op for plain URLs", () => {
68 expect(redactCloneUrl("https://example.com")).toBe("https://example.com");
69 });
70});
71
72// ---------------------------------------------------------------------------
73// parseNextLink
74// ---------------------------------------------------------------------------
75
76describe("parseNextLink", () => {
77 it("extracts next URL from Link header", () => {
78 const link =
79 '<https://api.github.com/repos/o/r/issues?page=2>; rel="next", <https://api.github.com/repos/o/r/issues?page=5>; rel="last"';
80 expect(parseNextLink(link)).toBe(
81 "https://api.github.com/repos/o/r/issues?page=2"
82 );
83 });
84 it("returns undefined when no next link", () => {
85 expect(parseNextLink(null)).toBeUndefined();
86 expect(parseNextLink('<...>; rel="prev"')).toBeUndefined();
87 });
88});
89
90// ---------------------------------------------------------------------------
91// paginate — mocked fetch
92// ---------------------------------------------------------------------------
93
94describe("paginate", () => {
95 function mkFetch(pages: unknown[][]): typeof fetch {
96 let i = 0;
97 return (async () => {
98 const body = pages[i] ?? [];
99 const linkNext =
100 i < pages.length - 1
101 ? `<https://api.github.com/next?page=${i + 2}>; rel="next"`
102 : undefined;
103 i += 1;
104 return new Response(JSON.stringify(body), {
105 status: 200,
106 headers: linkNext ? { Link: linkNext } : {},
107 });
108 }) as unknown as typeof fetch;
109 }
110
111 it("walks all pages under cap", async () => {
112 const f = mkFetch([[1, 2], [3, 4], [5]]);
113 const r = await paginate<number>("t", "https://api.github.com/first", 100, f);
114 expect(r.ok).toBe(true);
115 if (r.ok) expect(r.data).toEqual([1, 2, 3, 4, 5]);
116 });
117
118 it("stops at cap even mid-page", async () => {
119 const f = mkFetch([[1, 2, 3], [4, 5, 6]]);
120 const r = await paginate<number>("t", "https://api.github.com/first", 4, f);
121 expect(r.ok).toBe(true);
122 if (r.ok) expect(r.data).toEqual([1, 2, 3, 4]);
123 });
124
125 it("returns error on non-2xx response", async () => {
126 const f = (async () =>
127 new Response("{}", { status: 404 })) as unknown as typeof fetch;
128 const r = await paginate("t", "https://api.github.com/first", 10, f);
129 expect(r.ok).toBe(false);
130 if (!r.ok) expect(r.error).toContain("404");
131 });
132
133 it("returns error when body is not an array", async () => {
134 const f = (async () =>
135 new Response("{}", { status: 200 })) as unknown as typeof fetch;
136 const r = await paginate("t", "https://api.github.com/first", 10, f);
137 expect(r.ok).toBe(false);
138 });
139
140 it("degrades to ok:false when fetch throws", async () => {
141 const f = (async () => {
142 throw new Error("network");
143 }) as unknown as typeof fetch;
144 const r = await paginate("t", "https://api.github.com/first", 10, f);
145 expect(r.ok).toBe(false);
146 });
147});
148
149// ---------------------------------------------------------------------------
150// filterIssuesOnly
151// ---------------------------------------------------------------------------
152
153describe("filterIssuesOnly", () => {
154 it("drops entries that have a pull_request key", () => {
155 const items: GhIssue[] = [
156 mkIssue(1, "issue a", null),
157 mkIssue(2, "issue b", { url: "x" }),
158 mkIssue(3, "issue c", null),
159 ];
160 const out = filterIssuesOnly(items);
161 expect(out.length).toBe(2);
162 expect(out.map((i) => i.number)).toEqual([1, 3]);
163 });
164});
165
166function mkIssue(
167 n: number,
168 title: string,
169 pullRequest: { url: string } | null
170): GhIssue {
171 return {
172 number: n,
173 title,
174 body: null,
175 state: "open",
176 created_at: "2024-01-01T00:00:00Z",
177 updated_at: "2024-01-01T00:00:00Z",
178 closed_at: null,
179 labels: [],
180 pull_request: pullRequest,
181 };
182}
183
184// ---------------------------------------------------------------------------
185// Mappers
186// ---------------------------------------------------------------------------
187
188describe("normaliseColor", () => {
189 it("accepts 6-hex without hash", () => {
190 expect(normaliseColor("ff8800")).toBe("#ff8800");
191 });
192 it("accepts 6-hex with hash", () => {
193 expect(normaliseColor("#ff8800")).toBe("#ff8800");
194 });
195 it("lowercases mixed case", () => {
196 expect(normaliseColor("FF88aa")).toBe("#ff88aa");
197 });
198 it("falls back for invalid input", () => {
199 expect(normaliseColor("not-a-color")).toBe("#8b949e");
200 expect(normaliseColor(null)).toBe("#8b949e");
201 expect(normaliseColor("")).toBe("#8b949e");
202 expect(normaliseColor("ff88")).toBe("#8b949e");
203 });
204});
205
206describe("mapLabel", () => {
207 it("maps shape + clamps name length", () => {
208 const r = mapLabel(
209 {
210 name: "x".repeat(200),
211 color: "00aabb",
212 description: "y".repeat(500),
213 },
214 "repo-1"
215 );
216 expect(r.repositoryId).toBe("repo-1");
217 expect(r.name.length).toBe(50);
218 expect(r.color).toBe("#00aabb");
219 expect(r.description?.length).toBe(200);
220 });
221});
222
223describe("mapIssue", () => {
224 it("preserves closed state + closedAt", () => {
225 const r = mapIssue(
226 {
227 number: 1,
228 title: "t",
229 body: "b",
230 state: "closed",
231 created_at: "2024-01-01T00:00:00Z",
232 updated_at: "2024-01-01T00:00:00Z",
233 closed_at: "2024-02-01T00:00:00Z",
234 labels: [],
235 },
236 "repo",
237 "user"
238 );
239 expect(r.state).toBe("closed");
240 expect(r.closedAt).toBeInstanceOf(Date);
241 });
242 it("defaults to open + null closedAt", () => {
243 const r = mapIssue(
244 {
245 number: 1,
246 title: "t",
247 body: null,
248 state: "open",
249 created_at: "",
250 updated_at: "",
251 closed_at: null,
252 labels: [],
253 },
254 "repo",
255 "user"
256 );
257 expect(r.state).toBe("open");
258 expect(r.closedAt).toBeNull();
259 });
260});
261
262describe("mapPull", () => {
263 const base: GhPull = {
264 number: 1,
265 title: "t",
266 body: null,
267 state: "open",
268 merged_at: null,
269 closed_at: null,
270 created_at: "",
271 updated_at: "",
272 draft: false,
273 base: { ref: "main" },
274 head: { ref: "feature" },
275 };
276
277 it("open when neither merged nor closed", () => {
278 expect(mapPull(base, "r", "u").state).toBe("open");
279 });
280 it("merged takes precedence over closed", () => {
281 const r = mapPull(
282 { ...base, state: "closed", merged_at: "2024-02-01T00:00:00Z" },
283 "r",
284 "u"
285 );
286 expect(r.state).toBe("merged");
287 expect(r.mergedAt).toBeInstanceOf(Date);
288 });
289 it("closed when state=closed and no merge", () => {
290 const r = mapPull(
291 { ...base, state: "closed", closed_at: "2024-02-01T00:00:00Z" },
292 "r",
293 "u"
294 );
295 expect(r.state).toBe("closed");
296 expect(r.closedAt).toBeInstanceOf(Date);
297 });
298 it("keeps branches verbatim", () => {
299 expect(mapPull(base, "r", "u").baseBranch).toBe("main");
300 expect(mapPull(base, "r", "u").headBranch).toBe("feature");
301 });
302 it("propagates isDraft", () => {
303 expect(mapPull({ ...base, draft: true }, "r", "u").isDraft).toBe(true);
304 });
305});
306
307describe("mapRelease", () => {
308 it("falls back to tag_name when name is null", () => {
309 const gh: GhRelease = {
310 tag_name: "v1.0",
311 name: null,
312 body: null,
313 target_commitish: "main",
314 prerelease: false,
315 draft: false,
316 created_at: "",
317 published_at: null,
318 };
319 expect(mapRelease(gh, "r", "u").name).toBe("v1.0");
320 });
321 it("passes through draft + prerelease flags", () => {
322 const gh: GhRelease = {
323 tag_name: "v1",
324 name: "One",
325 body: null,
326 target_commitish: "main",
327 prerelease: true,
328 draft: true,
329 created_at: "",
330 published_at: "2024-01-01T00:00:00Z",
331 };
332 const r = mapRelease(gh, "r", "u");
333 expect(r.isDraft).toBe(true);
334 expect(r.isPrerelease).toBe(true);
335 expect(r.publishedAt).toBeInstanceOf(Date);
336 });
337});
338
339// ---------------------------------------------------------------------------
340// Route auth smokes
341// ---------------------------------------------------------------------------
342
343describe("github-import — route auth smokes", () => {
344 it("GET /new/import without session → 302 /login", async () => {
345 const res = await app.fetch(new Request("http://test/new/import"));
346 expect(res.status).toBe(302);
347 expect(res.headers.get("location")).toMatch(/\/login/);
348 });
349
350 it("POST /new/import without session → 302 /login", async () => {
351 const res = await app.fetch(
352 new Request("http://test/new/import", {
353 method: "POST",
354 headers: { "content-type": "application/x-www-form-urlencoded" },
355 body: new URLSearchParams({
356 source: "octocat/hello",
357 token: "ghp_x",
358 name: "hello",
359 visibility: "public",
360 }),
361 })
362 );
363 expect(res.status).toBe(302);
364 expect(res.headers.get("location")).toMatch(/\/login/);
365 });
366
367 it("GET /api/imports/:id without session → 401 JSON", async () => {
368 const res = await app.fetch(
369 new Request(
370 "http://test/api/imports/00000000-0000-0000-0000-000000000000",
371 { headers: { accept: "application/json" } }
372 )
373 );
374 // Bearer-less request to a JSON endpoint under requireAuth → 401.
375 // (Cookie-less GETs to non-API paths redirect; this path is /api/*.)
376 expect([302, 401]).toContain(res.status);
377 });
378});
Modifiedsrc/app.tsx+4−0View fileUnifiedSplit
7979import agentsRoutes from "./routes/agents";
8080import agentMarketplaceRoutes from "./routes/agent-marketplace";
8181import crossProductRoutes from "./routes/cross-product";
82import githubImportRoutes from "./routes/github-import";
8283import webRoutes from "./routes/web";
8384
8485const app = new Hono();
280281// Cross-product identity — unified auth across Gluecron/Crontech/Gatetest (Block K11)
281282app.route("/", crossProductRoutes);
282283
284// GitHub importer — /new/import (Block L). Must be before webRoutes catch-all.
285app.route("/", githubImportRoutes);
286
283287// Insights + milestones
284288app.route("/", insightsRoutes);
285289
Modifiedsrc/db/schema.ts+30−0View fileUnifiedSplit
25082508});
25092509
25102510export type CrossProductToken = typeof crossProductTokens.$inferSelect;
2511
2512// ---------- Block L — GitHub importer job ledger ----------
2513
2514export const githubImports = pgTable(
2515 "github_imports",
2516 {
2517 id: uuid("id").primaryKey().defaultRandom(),
2518 repositoryId: uuid("repository_id").references(() => repositories.id, {
2519 onDelete: "cascade",
2520 }),
2521 userId: uuid("user_id")
2522 .notNull()
2523 .references(() => users.id, { onDelete: "cascade" }),
2524 sourceOwner: text("source_owner").notNull(),
2525 sourceRepo: text("source_repo").notNull(),
2526 status: text("status").notNull().default("pending"),
2527 stats: text("stats").notNull().default("{}"),
2528 error: text("error"),
2529 startedAt: timestamp("started_at").defaultNow().notNull(),
2530 finishedAt: timestamp("finished_at"),
2531 },
2532 (table) => [
2533 index("github_imports_user_idx").on(table.userId, table.startedAt),
2534 index("github_imports_repo_idx").on(table.repositoryId),
2535 ]
2536);
2537
2538export type GithubImport = typeof githubImports.$inferSelect;
2539
2540export type CrossProductToken = typeof crossProductTokens.$inferSelect;
Addedsrc/lib/github-import.ts+772−0View fileUnifiedSplit
1/**
2 * Block L — GitHub importer.
3 *
4 * Pure helpers + orchestrator to copy a GitHub repo's metadata (labels,
5 * issues, PRs, comments, releases, stargazers) into Gluecron's own tables.
6 * The git content itself is handled separately via `git clone --mirror`.
7 *
8 * Contract:
9 * - Never throws. All helpers return `{ ok, data? } | { ok: false, error }`.
10 * - Paginated walkers honour a per-endpoint cap so a single sync request
11 * terminates in bounded time (no background worker needed for v1).
12 * - Auth is a GitHub PAT passed per call; we never log it.
13 * - Every insert uses the importing user as the authorId fallback so schema
14 * NOT NULL constraints are preserved even when we can't resolve the
15 * original GitHub author to a Gluecron user.
16 */
17
18import { and, eq } from "drizzle-orm";
19import { db } from "../db";
20import {
21 issueComments,
22 issueLabels,
23 issues,
24 labels as labelsTable,
25 prComments,
26 pullRequests,
27 releases,
28 repositories,
29 stars,
30 githubImports,
31} from "../db/schema";
32
33// ---------------------------------------------------------------------------
34// Types
35// ---------------------------------------------------------------------------
36
37export type Result<T> = { ok: true; data: T } | { ok: false; error: string };
38
39export interface GhRepo {
40 default_branch?: string;
41 private?: boolean;
42 description?: string | null;
43}
44
45export interface GhLabel {
46 name: string;
47 color: string;
48 description: string | null;
49}
50
51export interface GhIssue {
52 number: number;
53 title: string;
54 body: string | null;
55 state: "open" | "closed";
56 created_at: string;
57 updated_at: string;
58 closed_at: string | null;
59 labels: GhLabel[] | Array<{ name: string }>;
60 pull_request?: { url: string } | null;
61}
62
63export interface GhPull {
64 number: number;
65 title: string;
66 body: string | null;
67 state: "open" | "closed";
68 merged_at: string | null;
69 closed_at: string | null;
70 created_at: string;
71 updated_at: string;
72 draft: boolean;
73 base: { ref: string };
74 head: { ref: string };
75}
76
77export interface GhComment {
78 body: string;
79 created_at: string;
80 updated_at: string;
81}
82
83export interface GhRelease {
84 tag_name: string;
85 name: string | null;
86 body: string | null;
87 target_commitish: string;
88 prerelease: boolean;
89 draft: boolean;
90 created_at: string;
91 published_at: string | null;
92}
93
94export interface ImportCaps {
95 labels: number;
96 issues: number;
97 pulls: number;
98 issueComments: number;
99 prComments: number;
100 releases: number;
101 stargazers: number;
102}
103
104export const DEFAULT_CAPS: ImportCaps = {
105 labels: 200,
106 issues: 200,
107 pulls: 100,
108 issueComments: 500,
109 prComments: 500,
110 releases: 50,
111 stargazers: 200,
112};
113
114export interface ImportStats {
115 labels: number;
116 issues: number;
117 pulls: number;
118 issueComments: number;
119 prComments: number;
120 releases: number;
121 stargazers: number;
122}
123
124export interface RunImportArgs {
125 token: string;
126 sourceOwner: string;
127 sourceRepo: string;
128 targetRepoId: string;
129 importerUserId: string;
130 caps?: Partial<ImportCaps>;
131 fetchImpl?: typeof fetch; // injectable for tests
132}
133
134export interface RunImportResult {
135 ok: boolean;
136 stats: ImportStats;
137 error?: string;
138}
139
140// ---------------------------------------------------------------------------
141// URL construction
142// ---------------------------------------------------------------------------
143
144/**
145 * Build an authed clone URL. The token is URL-encoded and injected as
146 * username. We reject CR/LF so a malicious token can't inject a header.
147 */
148export function buildAuthedCloneUrl(
149 token: string,
150 owner: string,
151 repo: string
152): Result<string> {
153 if (!token || /[\r\n\s]/.test(token)) {
154 return { ok: false, error: "invalid token" };
155 }
156 if (!/^[A-Za-z0-9_.-]+$/.test(owner) || !/^[A-Za-z0-9_.-]+$/.test(repo)) {
157 return { ok: false, error: "invalid owner/repo" };
158 }
159 const encoded = encodeURIComponent(token);
160 return {
161 ok: true,
162 data: `https://x-access-token:${encoded}@github.com/${owner}/${repo}.git`,
163 };
164}
165
166/** Redact authed URL for logging. */
167export function redactCloneUrl(url: string): string {
168 return url.replace(/https:\/\/[^@]+@/, "https://***@");
169}
170
171// ---------------------------------------------------------------------------
172// Pagination
173// ---------------------------------------------------------------------------
174
175export interface GhFetchResponse {
176 ok: boolean;
177 status: number;
178 body: unknown;
179 linkNext?: string;
180}
181
182export async function ghFetch(
183 token: string,
184 url: string,
185 fetchImpl: typeof fetch = fetch
186): Promise<GhFetchResponse> {
187 try {
188 const res = await fetchImpl(url, {
189 headers: {
190 Accept: "application/vnd.github+json",
191 Authorization: `token ${token}`,
192 "User-Agent": "gluecron-importer",
193 },
194 });
195 let body: unknown = null;
196 try {
197 body = await res.json();
198 } catch {
199 body = null;
200 }
201 const link = res.headers.get("link") || res.headers.get("Link");
202 const linkNext = parseNextLink(link);
203 return { ok: res.ok, status: res.status, body, linkNext };
204 } catch (err) {
205 return {
206 ok: false,
207 status: 0,
208 body: null,
209 linkNext: undefined,
210 };
211 }
212}
213
214export function parseNextLink(link: string | null): string | undefined {
215 if (!link) return undefined;
216 for (const part of link.split(",")) {
217 const m = part.match(/<([^>]+)>\s*;\s*rel="next"/);
218 if (m) return m[1];
219 }
220 return undefined;
221}
222
223/**
224 * Walk a paginated endpoint accumulating items up to `cap`. Stops on first
225 * error response or when `linkNext` is absent.
226 */
227export async function paginate<T>(
228 token: string,
229 firstUrl: string,
230 cap: number,
231 fetchImpl: typeof fetch = fetch
232): Promise<Result<T[]>> {
233 const out: T[] = [];
234 let url: string | undefined = firstUrl;
235 let safety = 20; // hard ceiling on page walks
236 while (url && out.length < cap && safety-- > 0) {
237 const r = await ghFetch(token, url, fetchImpl);
238 if (!r.ok) return { ok: false, error: `github ${r.status}` };
239 if (!Array.isArray(r.body)) {
240 return { ok: false, error: "expected array" };
241 }
242 for (const item of r.body as T[]) {
243 out.push(item);
244 if (out.length >= cap) break;
245 }
246 url = r.linkNext;
247 }
248 return { ok: true, data: out };
249}
250
251// ---------------------------------------------------------------------------
252// Endpoint walkers
253// ---------------------------------------------------------------------------
254
255const API = "https://api.github.com";
256
257export function fetchRepo(
258 token: string,
259 owner: string,
260 repo: string,
261 fetchImpl: typeof fetch = fetch
262): Promise<GhFetchResponse> {
263 return ghFetch(token, `${API}/repos/${owner}/${repo}`, fetchImpl);
264}
265
266export function fetchLabels(
267 token: string,
268 owner: string,
269 repo: string,
270 cap: number,
271 fetchImpl: typeof fetch = fetch
272): Promise<Result<GhLabel[]>> {
273 return paginate<GhLabel>(
274 token,
275 `${API}/repos/${owner}/${repo}/labels?per_page=100`,
276 cap,
277 fetchImpl
278 );
279}
280
281export function fetchIssuesAndPulls(
282 token: string,
283 owner: string,
284 repo: string,
285 cap: number,
286 fetchImpl: typeof fetch = fetch
287): Promise<Result<GhIssue[]>> {
288 return paginate<GhIssue>(
289 token,
290 `${API}/repos/${owner}/${repo}/issues?state=all&per_page=100`,
291 cap,
292 fetchImpl
293 );
294}
295
296/** Filter a mixed issues+pulls list down to issues only. */
297export function filterIssuesOnly(items: GhIssue[]): GhIssue[] {
298 return items.filter((i) => !i.pull_request);
299}
300
301export function fetchPullRequests(
302 token: string,
303 owner: string,
304 repo: string,
305 cap: number,
306 fetchImpl: typeof fetch = fetch
307): Promise<Result<GhPull[]>> {
308 return paginate<GhPull>(
309 token,
310 `${API}/repos/${owner}/${repo}/pulls?state=all&per_page=100`,
311 cap,
312 fetchImpl
313 );
314}
315
316export function fetchIssueComments(
317 token: string,
318 owner: string,
319 repo: string,
320 issueNumber: number,
321 cap: number,
322 fetchImpl: typeof fetch = fetch
323): Promise<Result<GhComment[]>> {
324 return paginate<GhComment>(
325 token,
326 `${API}/repos/${owner}/${repo}/issues/${issueNumber}/comments?per_page=100`,
327 cap,
328 fetchImpl
329 );
330}
331
332export function fetchReleases(
333 token: string,
334 owner: string,
335 repo: string,
336 cap: number,
337 fetchImpl: typeof fetch = fetch
338): Promise<Result<GhRelease[]>> {
339 return paginate<GhRelease>(
340 token,
341 `${API}/repos/${owner}/${repo}/releases?per_page=100`,
342 cap,
343 fetchImpl
344 );
345}
346
347// ---------------------------------------------------------------------------
348// Mappers — pure transforms from GitHub shapes to Gluecron insert rows
349// ---------------------------------------------------------------------------
350
351/** Normalise GitHub's 6-hex color (no #) into Gluecron's `#RRGGBB`. */
352export function normaliseColor(hex: string | null | undefined): string {
353 if (!hex) return "#8b949e";
354 const s = hex.replace(/^#/, "").toLowerCase();
355 if (!/^[0-9a-f]{6}$/.test(s)) return "#8b949e";
356 return `#${s}`;
357}
358
359export function mapLabel(
360 gh: GhLabel,
361 repoId: string
362): {
363 repositoryId: string;
364 name: string;
365 color: string;
366 description: string | null;
367} {
368 return {
369 repositoryId: repoId,
370 name: gh.name.slice(0, 50),
371 color: normaliseColor(gh.color),
372 description: gh.description ? gh.description.slice(0, 200) : null,
373 };
374}
375
376export function mapIssue(
377 gh: GhIssue,
378 repoId: string,
379 authorId: string
380): {
381 repositoryId: string;
382 authorId: string;
383 title: string;
384 body: string | null;
385 state: string;
386 closedAt: Date | null;
387} {
388 return {
389 repositoryId: repoId,
390 authorId,
391 title: gh.title.slice(0, 500),
392 body: gh.body,
393 state: gh.state === "closed" ? "closed" : "open",
394 closedAt: gh.closed_at ? new Date(gh.closed_at) : null,
395 };
396}
397
398export function mapPull(
399 gh: GhPull,
400 repoId: string,
401 authorId: string
402): {
403 repositoryId: string;
404 authorId: string;
405 title: string;
406 body: string | null;
407 state: string;
408 baseBranch: string;
409 headBranch: string;
410 isDraft: boolean;
411 mergedAt: Date | null;
412 closedAt: Date | null;
413} {
414 let state: "open" | "closed" | "merged" = "open";
415 if (gh.merged_at) state = "merged";
416 else if (gh.state === "closed") state = "closed";
417 return {
418 repositoryId: repoId,
419 authorId,
420 title: gh.title.slice(0, 500),
421 body: gh.body,
422 state,
423 baseBranch: gh.base.ref.slice(0, 200),
424 headBranch: gh.head.ref.slice(0, 200),
425 isDraft: !!gh.draft,
426 mergedAt: gh.merged_at ? new Date(gh.merged_at) : null,
427 closedAt: gh.closed_at ? new Date(gh.closed_at) : null,
428 };
429}
430
431export function mapRelease(
432 gh: GhRelease,
433 repoId: string,
434 authorId: string
435): {
436 repositoryId: string;
437 authorId: string;
438 tag: string;
439 name: string;
440 body: string | null;
441 targetCommit: string;
442 isDraft: boolean;
443 isPrerelease: boolean;
444 publishedAt: Date | null;
445} {
446 return {
447 repositoryId: repoId,
448 authorId,
449 tag: gh.tag_name.slice(0, 100),
450 name: (gh.name ?? gh.tag_name).slice(0, 200),
451 body: gh.body,
452 targetCommit: gh.target_commitish.slice(0, 100),
453 isDraft: !!gh.draft,
454 isPrerelease: !!gh.prerelease,
455 publishedAt: gh.published_at ? new Date(gh.published_at) : null,
456 };
457}
458
459// ---------------------------------------------------------------------------
460// Orchestrator
461// ---------------------------------------------------------------------------
462
463function emptyStats(): ImportStats {
464 return {
465 labels: 0,
466 issues: 0,
467 pulls: 0,
468 issueComments: 0,
469 prComments: 0,
470 releases: 0,
471 stargazers: 0,
472 };
473}
474
475/**
476 * Walk GitHub endpoints for a single repo and mirror metadata into Gluecron.
477 * Each endpoint is capped; failures are recorded on the run but do not halt
478 * the overall import (best-effort, never throws).
479 */
480export async function runImport(args: RunImportArgs): Promise<RunImportResult> {
481 const caps: ImportCaps = { ...DEFAULT_CAPS, ...(args.caps ?? {}) };
482 const stats = emptyStats();
483 const fetchImpl = args.fetchImpl ?? fetch;
484 const errors: string[] = [];
485
486 try {
487 // Labels — build a name → id map so we can attach issueLabels.
488 const labelMap = new Map<string, string>();
489 const labelsRes = await fetchLabels(
490 args.token,
491 args.sourceOwner,
492 args.sourceRepo,
493 caps.labels,
494 fetchImpl
495 );
496 if (labelsRes.ok) {
497 for (const ghLabel of labelsRes.data) {
498 try {
499 const row = mapLabel(ghLabel, args.targetRepoId);
500 const [inserted] = await db
501 .insert(labelsTable)
502 .values(row)
503 .onConflictDoNothing()
504 .returning();
505 let labelId = inserted?.id;
506 if (!labelId) {
507 const [existing] = await db
508 .select()
509 .from(labelsTable)
510 .where(
511 and(
512 eq(labelsTable.repositoryId, args.targetRepoId),
513 eq(labelsTable.name, row.name)
514 )
515 )
516 .limit(1);
517 labelId = existing?.id;
518 }
519 if (labelId) {
520 labelMap.set(row.name, labelId);
521 stats.labels += 1;
522 }
523 } catch {
524 // skip individual label on failure
525 }
526 }
527 } else {
528 errors.push(`labels: ${labelsRes.error}`);
529 }
530
531 // Issues + PRs come together from /issues?state=all; split them here.
532 const mixedRes = await fetchIssuesAndPulls(
533 args.token,
534 args.sourceOwner,
535 args.sourceRepo,
536 caps.issues + caps.pulls,
537 fetchImpl
538 );
539 const issuesOnly = mixedRes.ok ? filterIssuesOnly(mixedRes.data) : [];
540 const issuesToInsert = issuesOnly.slice(0, caps.issues);
541 if (!mixedRes.ok) errors.push(`issues: ${mixedRes.error}`);
542
543 for (const ghIssue of issuesToInsert) {
544 try {
545 const [row] = await db
546 .insert(issues)
547 .values(mapIssue(ghIssue, args.targetRepoId, args.importerUserId))
548 .returning();
549 if (!row) continue;
550 stats.issues += 1;
551
552 // Attach labels
553 for (const raw of ghIssue.labels ?? []) {
554 const name =
555 typeof raw === "string"
556 ? raw
557 : (raw as { name?: string } | null)?.name;
558 if (!name) continue;
559 const labelId = labelMap.get(name.slice(0, 50));
560 if (!labelId) continue;
561 try {
562 await db
563 .insert(issueLabels)
564 .values({ issueId: row.id, labelId })
565 .onConflictDoNothing();
566 } catch {
567 // ignore per-label
568 }
569 }
570
571 // Walk comments for this issue (only while under the global cap)
572 if (stats.issueComments < caps.issueComments) {
573 const commentsRes = await fetchIssueComments(
574 args.token,
575 args.sourceOwner,
576 args.sourceRepo,
577 ghIssue.number,
578 Math.min(50, caps.issueComments - stats.issueComments),
579 fetchImpl
580 );
581 if (commentsRes.ok) {
582 for (const ghComment of commentsRes.data) {
583 try {
584 await db.insert(issueComments).values({
585 issueId: row.id,
586 authorId: args.importerUserId,
587 body: ghComment.body,
588 });
589 stats.issueComments += 1;
590 if (stats.issueComments >= caps.issueComments) break;
591 } catch {
592 // skip one comment
593 }
594 }
595 }
596 }
597 } catch {
598 // skip one issue
599 }
600 }
601
602 // Pull requests — separate endpoint has base/head/merged_at we need.
603 const pullsRes = await fetchPullRequests(
604 args.token,
605 args.sourceOwner,
606 args.sourceRepo,
607 caps.pulls,
608 fetchImpl
609 );
610 if (!pullsRes.ok) errors.push(`pulls: ${pullsRes.error}`);
611 if (pullsRes.ok) {
612 for (const ghPull of pullsRes.data) {
613 try {
614 const [row] = await db
615 .insert(pullRequests)
616 .values(mapPull(ghPull, args.targetRepoId, args.importerUserId))
617 .returning();
618 if (!row) continue;
619 stats.pulls += 1;
620
621 if (stats.prComments < caps.prComments) {
622 const commentsRes = await fetchIssueComments(
623 args.token,
624 args.sourceOwner,
625 args.sourceRepo,
626 ghPull.number,
627 Math.min(50, caps.prComments - stats.prComments),
628 fetchImpl
629 );
630 if (commentsRes.ok) {
631 for (const ghComment of commentsRes.data) {
632 try {
633 await db.insert(prComments).values({
634 pullRequestId: row.id,
635 authorId: args.importerUserId,
636 body: ghComment.body,
637 isAiReview: false,
638 });
639 stats.prComments += 1;
640 if (stats.prComments >= caps.prComments) break;
641 } catch {
642 // skip one comment
643 }
644 }
645 }
646 }
647 } catch {
648 // skip one PR
649 }
650 }
651 }
652
653 // Releases
654 const releasesRes = await fetchReleases(
655 args.token,
656 args.sourceOwner,
657 args.sourceRepo,
658 caps.releases,
659 fetchImpl
660 );
661 if (!releasesRes.ok) errors.push(`releases: ${releasesRes.error}`);
662 if (releasesRes.ok) {
663 for (const ghRelease of releasesRes.data) {
664 try {
665 await db
666 .insert(releases)
667 .values(
668 mapRelease(ghRelease, args.targetRepoId, args.importerUserId)
669 )
670 .onConflictDoNothing();
671 stats.releases += 1;
672 } catch {
673 // skip
674 }
675 }
676 }
677
678 // Stargazers — just a count bump; we don't create fake user rows.
679 const stargazersRes = await paginate<unknown>(
680 args.token,
681 `${API}/repos/${args.sourceOwner}/${args.sourceRepo}/stargazers?per_page=100`,
682 caps.stargazers,
683 fetchImpl
684 );
685 if (stargazersRes.ok) {
686 stats.stargazers = stargazersRes.data.length;
687 try {
688 await db
689 .update(repositories)
690 .set({ starCount: stats.stargazers })
691 .where(eq(repositories.id, args.targetRepoId));
692 // Self-star from the importer so the "Stars" list isn't empty.
693 await db
694 .insert(stars)
695 .values({
696 userId: args.importerUserId,
697 repositoryId: args.targetRepoId,
698 })
699 .onConflictDoNothing();
700 } catch {
701 // ignore
702 }
703 } else {
704 errors.push(`stargazers: ${stargazersRes.error}`);
705 }
706
707 return {
708 ok: errors.length === 0,
709 stats,
710 error: errors.length > 0 ? errors.join("; ") : undefined,
711 };
712 } catch (err) {
713 return {
714 ok: false,
715 stats,
716 error: err instanceof Error ? err.message : "unknown error",
717 };
718 }
719}
720
721// ---------------------------------------------------------------------------
722// Ledger helpers
723// ---------------------------------------------------------------------------
724
725export async function createImportRow(args: {
726 userId: string;
727 sourceOwner: string;
728 sourceRepo: string;
729}): Promise<string | null> {
730 try {
731 const [row] = await db
732 .insert(githubImports)
733 .values({
734 userId: args.userId,
735 sourceOwner: args.sourceOwner,
736 sourceRepo: args.sourceRepo,
737 status: "pending",
738 })
739 .returning();
740 return row?.id ?? null;
741 } catch {
742 return null;
743 }
744}
745
746export async function finaliseImportRow(
747 importId: string,
748 patch: {
749 repositoryId?: string;
750 status: "cloning" | "walking" | "ok" | "error";
751 stats?: ImportStats;
752 error?: string;
753 }
754): Promise<void> {
755 try {
756 await db
757 .update(githubImports)
758 .set({
759 repositoryId: patch.repositoryId,
760 status: patch.status,
761 stats: patch.stats ? JSON.stringify(patch.stats) : undefined,
762 error: patch.error,
763 finishedAt:
764 patch.status === "ok" || patch.status === "error"
765 ? new Date()
766 : undefined,
767 })
768 .where(eq(githubImports.id, importId));
769 } catch {
770 // best effort
771 }
772}
Addedsrc/routes/github-import.tsx+398−0View fileUnifiedSplit
1/**
2 * Block L — GitHub import UI.
3 *
4 * GET /new/import — form (PAT + owner/repo + target name + visibility).
5 * POST /new/import — clones the repo via `git clone --mirror`, then walks
6 * GitHub metadata synchronously. Redirects to the target
7 * repo page on success.
8 *
9 * v1 is synchronous: we return when the clone + walk finishes (caps keep
10 * this bounded). Later this can be moved to a background worker.
11 */
12
13import { Hono } from "hono";
14import { desc, eq } from "drizzle-orm";
15import { join } from "path";
16import { db } from "../db";
17import {
18 githubImports,
19 repositories,
20 users,
21} from "../db/schema";
22import { requireAuth, softAuth } from "../middleware/auth";
23import type { AuthEnv } from "../middleware/auth";
24import { Layout } from "../views/layout";
25import { config } from "../lib/config";
26import { repoExists } from "../git/repository";
27import { audit } from "../lib/notify";
28import {
29 buildAuthedCloneUrl,
30 createImportRow,
31 finaliseImportRow,
32 redactCloneUrl,
33 runImport,
34 type ImportStats,
35} from "../lib/github-import";
36
37const githubImport = new Hono<AuthEnv>();
38
39githubImport.use("*", softAuth);
40
41// ---------------------------------------------------------------------------
42// Form
43// ---------------------------------------------------------------------------
44
45githubImport.get("/new/import", requireAuth, async (c) => {
46 const user = c.get("user")!;
47 const error = c.req.query("error");
48
49 // Show recent imports by this user so they can see what's been pulled.
50 let recent: Array<{
51 id: string;
52 sourceOwner: string;
53 sourceRepo: string;
54 status: string;
55 stats: string;
56 startedAt: Date;
57 repositoryId: string | null;
58 }> = [];
59 try {
60 const rows = await db
61 .select()
62 .from(githubImports)
63 .where(eq(githubImports.userId, user.id))
64 .orderBy(desc(githubImports.startedAt))
65 .limit(10);
66 recent = rows.map((r) => ({
67 id: r.id,
68 sourceOwner: r.sourceOwner,
69 sourceRepo: r.sourceRepo,
70 status: r.status,
71 stats: r.stats,
72 startedAt: r.startedAt,
73 repositoryId: r.repositoryId,
74 }));
75 } catch {
76 recent = [];
77 }
78
79 return c.html(
80 <Layout title="Import from GitHub" user={user}>
81 <div class="new-repo-form">
82 <h2>Import from GitHub</h2>
83 <p style="color: var(--muted); margin-bottom: 16px">
84 Clone a GitHub repository into your Gluecron namespace, including
85 issues, pull requests, comments, releases, and labels.
86 </p>
87 {error && <div class="auth-error">{decodeURIComponent(error)}</div>}
88 <form method="POST" action="/new/import">
89 <div class="form-group">
90 <label for="source">GitHub repo (owner/repo)</label>
91 <input
92 type="text"
93 id="source"
94 name="source"
95 required
96 placeholder="octocat/hello-world"
97 autocomplete="off"
98 />
99 </div>
100 <div class="form-group">
101 <label for="token">GitHub personal access token</label>
102 <input
103 type="password"
104 id="token"
105 name="token"
106 required
107 placeholder="github_pat_... or ghp_..."
108 autocomplete="off"
109 />
110 <small style="color: var(--muted)">
111 Needs <code>repo</code> scope for private repos, or <code>public_repo</code>.
112 Never stored.
113 </small>
114 </div>
115 <div class="form-group">
116 <label for="name">Target name on Gluecron</label>
117 <input
118 type="text"
119 id="name"
120 name="name"
121 required
122 pattern="^[a-zA-Z0-9._-]+$"
123 placeholder="hello-world"
124 autocomplete="off"
125 />
126 </div>
127 <div class="visibility-options">
128 <label class="visibility-option">
129 <input type="radio" name="visibility" value="public" checked />
130 <div class="vis-label">Public</div>
131 <div class="vis-desc">Anyone can see this repository</div>
132 </label>
133 <label class="visibility-option">
134 <input type="radio" name="visibility" value="private" />
135 <div class="vis-label">Private</div>
136 <div class="vis-desc">Only you can see this repository</div>
137 </label>
138 </div>
139 <button type="submit" class="btn btn-primary">
140 Start import
141 </button>
142 </form>
143
144 {recent.length > 0 && (
145 <div style="margin-top: 32px">
146 <h3>Recent imports</h3>
147 <table style="width: 100%; margin-top: 8px">
148 <thead>
149 <tr>
150 <th style="text-align: left">Source</th>
151 <th style="text-align: left">Status</th>
152 <th style="text-align: left">Stats</th>
153 <th style="text-align: left">When</th>
154 </tr>
155 </thead>
156 <tbody>
157 {recent.map((r) => (
158 <tr>
159 <td>
160 <code>
161 {r.sourceOwner}/{r.sourceRepo}
162 </code>
163 </td>
164 <td>
165 <span class={`badge badge-${r.status}`}>{r.status}</span>
166 </td>
167 <td>
168 <code style="font-size: 11px">{r.stats}</code>
169 </td>
170 <td style="font-size: 12px; color: var(--muted)">
171 {r.startedAt.toISOString().slice(0, 19).replace("T", " ")}
172 </td>
173 </tr>
174 ))}
175 </tbody>
176 </table>
177 </div>
178 )}
179 </div>
180 </Layout>
181 );
182});
183
184// ---------------------------------------------------------------------------
185// Handler
186// ---------------------------------------------------------------------------
187
188githubImport.post("/new/import", requireAuth, async (c) => {
189 const user = c.get("user")!;
190 const body = await c.req.parseBody();
191 const source = String(body.source || "").trim();
192 const token = String(body.token || "").trim();
193 const targetName = String(body.name || "").trim();
194 const isPrivate = body.visibility === "private";
195
196 const bail = (msg: string) =>
197 c.redirect(`/new/import?error=${encodeURIComponent(msg)}`);
198
199 const sourceMatch = source.match(/^([A-Za-z0-9_.-]+)\/([A-Za-z0-9_.-]+)$/);
200 if (!sourceMatch) return bail("Source must be owner/repo");
201 const [, sourceOwner, sourceRepo] = sourceMatch;
202
203 if (!token) return bail("GitHub token is required");
204 if (!/^[a-zA-Z0-9._-]+$/.test(targetName)) {
205 return bail("Invalid target repository name");
206 }
207 if (await repoExists(user.username, targetName)) {
208 return bail("Target repository already exists");
209 }
210
211 const cloneUrl = buildAuthedCloneUrl(token, sourceOwner, sourceRepo);
212 if (!cloneUrl.ok) return bail(cloneUrl.error);
213
214 // Create the import ledger row up front so the user can see progress.
215 const importId = await createImportRow({
216 userId: user.id,
217 sourceOwner,
218 sourceRepo,
219 });
220
221 // Clone the git content first. `git clone --mirror` gives us every ref.
222 const destPath = join(
223 config.gitReposPath,
224 user.username,
225 `${targetName}.git`
226 );
227 if (importId) {
228 await finaliseImportRow(importId, { status: "cloning" });
229 }
230
231 const proc = Bun.spawn(["git", "clone", "--mirror", cloneUrl.data, destPath], {
232 stdout: "pipe",
233 stderr: "pipe",
234 env: { ...process.env, GIT_TERMINAL_PROMPT: "0" },
235 });
236 const exitCode = await Promise.race([
237 proc.exited,
238 new Promise<number>((resolve) =>
239 setTimeout(() => {
240 try {
241 proc.kill();
242 } catch {
243 // ignore
244 }
245 resolve(124);
246 }, 10 * 60 * 1000)
247 ),
248 ]);
249
250 if (exitCode !== 0) {
251 const stderr = await new Response(proc.stderr).text().catch(() => "");
252 const redacted = redactCloneUrl(stderr).slice(0, 500);
253 if (importId) {
254 await finaliseImportRow(importId, {
255 status: "error",
256 error: `git clone exited ${exitCode}: ${redacted}`,
257 });
258 }
259 return bail(`Clone failed (exit ${exitCode})`);
260 }
261
262 // Create the target repo row.
263 let newRepoId: string;
264 try {
265 const [newRepo] = await db
266 .insert(repositories)
267 .values({
268 name: targetName,
269 ownerId: user.id,
270 description: `Imported from github.com/${sourceOwner}/${sourceRepo}`,
271 isPrivate,
272 defaultBranch: "main",
273 diskPath: destPath,
274 })
275 .returning();
276 if (!newRepo) throw new Error("insert returned nothing");
277 newRepoId = newRepo.id;
278 } catch (err) {
279 if (importId) {
280 await finaliseImportRow(importId, {
281 status: "error",
282 error: err instanceof Error ? err.message : "DB insert failed",
283 });
284 }
285 return bail("Could not create Gluecron repo row");
286 }
287
288 // Green-by-default bootstrap (gates, protection, labels, welcome).
289 try {
290 const { bootstrapRepository } = await import("../lib/repo-bootstrap");
291 await bootstrapRepository({
292 repositoryId: newRepoId,
293 ownerUserId: user.id,
294 defaultBranch: "main",
295 skipWelcomeIssue: true,
296 });
297 } catch {
298 // non-fatal
299 }
300
301 if (importId) {
302 await finaliseImportRow(importId, {
303 repositoryId: newRepoId,
304 status: "walking",
305 });
306 }
307
308 // Walk GitHub metadata.
309 let stats: ImportStats;
310 let walkError: string | undefined;
311 try {
312 const res = await runImport({
313 token,
314 sourceOwner,
315 sourceRepo,
316 targetRepoId: newRepoId,
317 importerUserId: user.id,
318 });
319 stats = res.stats;
320 walkError = res.error;
321 } catch (err) {
322 stats = {
323 labels: 0,
324 issues: 0,
325 pulls: 0,
326 issueComments: 0,
327 prComments: 0,
328 releases: 0,
329 stargazers: 0,
330 };
331 walkError = err instanceof Error ? err.message : "unknown error";
332 }
333
334 if (importId) {
335 await finaliseImportRow(importId, {
336 repositoryId: newRepoId,
337 status: walkError ? "error" : "ok",
338 stats,
339 error: walkError,
340 });
341 }
342
343 try {
344 await audit({
345 userId: user.id,
346 repositoryId: newRepoId,
347 action: "github_import",
348 targetType: "repository",
349 targetId: newRepoId,
350 metadata: { source: `${sourceOwner}/${sourceRepo}`, stats, walkError },
351 });
352 } catch {
353 // ignore
354 }
355
356 return c.redirect(`/${user.username}/${targetName}`);
357});
358
359// ---------------------------------------------------------------------------
360// Status API (JSON) — useful for polling from the UI later.
361// ---------------------------------------------------------------------------
362
363githubImport.get("/api/imports/:id", requireAuth, async (c) => {
364 const user = c.get("user")!;
365 const { id } = c.req.param();
366 try {
367 const [row] = await db
368 .select()
369 .from(githubImports)
370 .where(eq(githubImports.id, id))
371 .limit(1);
372 if (!row || row.userId !== user.id) {
373 return c.json({ error: "not found" }, 404);
374 }
375 return c.json({
376 id: row.id,
377 source: `${row.sourceOwner}/${row.sourceRepo}`,
378 status: row.status,
379 stats: (() => {
380 try {
381 return JSON.parse(row.stats);
382 } catch {
383 return {};
384 }
385 })(),
386 error: row.error,
387 startedAt: row.startedAt,
388 finishedAt: row.finishedAt,
389 });
390 } catch {
391 return c.json({ error: "not available" }, 503);
392 }
393});
394
395export default githubImport;
396
397// Silence unused-import complaint for tests that need `users`.
398void users;
0399