feat: add wiki publishing tooling

This commit is contained in:
2026-05-28 11:32:36 +08:00
parent fee9b13ada
commit cda0db06db
10 changed files with 1558 additions and 0 deletions
+7
View File
@@ -6,6 +6,7 @@ PGPORT=5432
PGUSER=example_user
PGPASSWORD=example_password
PGDATABASE=example_db
WIKIDATABASE=example_wiki_db
# 额外示例变量:
APP_ENV=development
@@ -14,4 +15,10 @@ APP_SECRET_KEY=change-me
WORKBOOK_PATH=KSP Engine Tweak Chart.xlsx
SQLALCHEMY_ECHO=false
# 登录到wiki的账号信息
wiki_useremail=example@email.com
wiki_username=example_user
wiki_password=example_password
# 说明:此文件为示例,不包含真实凭据。将真实凭据写入本地的 .env 文件并确保其被 .gitignore 忽略。
@@ -0,0 +1,80 @@
---
name: wiki-backend-update
description: "Use when: updating Wiki.js pages from local Markdown/HTML, writing Wiki.js pages through backend/database workflows, fixing stale Wiki.js frontend cache after direct DB edits, refreshing pages.update/pages.render/pages.flushCache, or publishing KSP wiki articles. Requires scripts/wiki_update_page.py and final frontend verification."
argument-hint: "path=<wiki path> content-file=<local file> title=<title>"
---
# Wiki Backend Update
## 使用场景
当任务涉及 Wiki.js 页面后台更新、直接写库后的页面刷新、把本地 Markdown/HTML 发布到 Wiki.js、或处理“编辑/历史记录已更新但前台仍显示旧内容”的问题时,必须使用此流程。
## 必走流程
1. 先确认本地内容文件是最终版本,并运行适合该文件的基本检查。
2. 使用 `.env` 读取 Wiki.js PostgreSQL 与登录信息;不要打印密码、JWT 或完整账号信息。
3. 不要只改 `pages.content/render/toc` 就结束。直接 DB 更新后必须走官方 GraphQL 刷新链路。
4. 确认登录账号所在组的命中 Page Rule 包含 `write:pages``manage:pages`。组级权限有这两个值但 Page Rule 没有时,`pages.update` 仍会返回 `PageUpdateForbidden`
5. 调用官方 `pages.update` 时必须传 `tags`,即使页面没有标签也传 `tags: []`,否则 Wiki.js resolver 可能报 `Cannot read properties of undefined (reading 'map')`
6. `pages.render``pages.flushCache` 需要 `manage:system`。使用前必须 ask user;若获准,临时授予,完成后立即移除。
7. 前台验证必须检查普通页面 URL 已包含新内容且不包含旧占位文本。若工具返回疑似缓存结果,用终端无缓存抓取复核。
8. 上传图片等资产后,必须直接验证资源 URL 返回 200 和正确 Content-Type;不要只依赖 `assets` / `assetData` 数据库记录。
9. 清理所有 `_tmp_` 临时脚本或输出文件。
10. 任务收尾必须使用 ask user,确认页面状态或下一步。
## 固化脚本
使用仓库脚本 [scripts/wiki_update_page.py](../../../scripts/wiki_update_page.py)。它会:
- 读取 `.env` 中的 `PGHOST``PGUSER``PGPASSWORD``WIKIDATABASE`/`PGDATABASE` 和 Wiki 登录信息。
- 解析目标页面并保留现有 tags。
- 检查命中 Page Rule 是否具备 `write:pages` / `manage:pages`
- 通过官方 GraphQL 执行 `pages.update``pages.render``pages.flushCache`
- 对 HTML 页面使用 `--extract-style-to-script-css`,把本地 `<style>` 发布到 Wiki.js 页面 CSS 字段,避免 Wiki.js 过滤正文中的样式。
- 如果 HTML 页面包含交互脚本,使用 `--extract-script-to-script-js`,把第一个 `<script>` 发布到 Wiki.js 页面 JavaScript 字段,避免内联脚本留在正文中。
- 可在用户授权后临时添加 `manage:system`,并在 `finally` 中移除。
- 抓取前台页面并验证 `--must-contain` / `--must-not-contain`
上传 Wiki.js 资产使用 [scripts/wiki_upload_assets.py](../../../scripts/wiki_upload_assets.py)。它会通过 `/u` 上传端点写入既有资源文件夹,并验证 `assets` / `assetData` 记录。上传后仍需单独抓取资源 URL;当前实例存在部分 DB-backed assets 直连 500 的情况,页面交付前必须确认图片在前台实际可见。若资源端未修好,可临时使用压缩图片 `data:` URI,并保留 `data-wiki-asset` 标记以便之后切回正式资源 URL。
## 示例命令
```powershell
python scripts\wiki_update_page.py `
--path home/Space_Shuttles/Vulture_Shuttle/html_test `
--content-file vulture_shuttle_html_test.html `
--title "HTML version" `
--editor code `
--extract-style-to-script-css `
--extract-script-to-script-js `
--temporary-manage-system `
--must-contain "秃鹫航天飞机" `
--must-not-contain "# Title" `
--must-not-contain "Some text here"
```
如果命中 Page Rule 缺少写权限,先 ask user,再使用:
```powershell
python scripts\wiki_update_page.py `
--path home/Space_Shuttles/Vulture_Shuttle/html_test `
--content-file vulture_shuttle_html_test.html `
--title "HTML version" `
--editor code `
--extract-style-to-script-css `
--extract-script-to-script-js `
--patch-page-rule `
--temporary-manage-system `
--must-contain "秃鹫航天飞机" `
--must-not-contain "# Title" `
--must-not-contain "Some text here"
```
## 失败判断
- `PageUpdateForbidden`: 命中 Page Rule 仍缺少 `write:pages` 或路径规则没匹配。
- `Cannot read properties of undefined (reading 'map')`: `pages.update` 没有传 `tags`
- `pages.render``pages.flushCache``Forbidden`: 缺少 `manage:system`,或权限缓存未刷新。
- 上传资产后数据库记录存在但 URL 返回 500:资源服务/cache 层异常,DB 验证不够;先直接验证图片 URL,必要时临时使用 `data:` URI 保证页面可见。
- 编辑页/历史记录正确但前台旧内容:必须执行 render/flushCache,必要时重启 Wiki.js 容器。
+26
View File
@@ -97,6 +97,32 @@ docker compose up --build
- 根目录已经补上 .gitignore,避免后续再次提交 .env
- 如果 .env 已经在 Git 索引里,.gitignore 不会自动取消跟踪,需要你本地手动把它移出索引后再提交
## Wiki.js 资产与页面发布记录
Wiki.js 页面发布使用 `scripts/wiki_update_page.py`,图片上传使用 `scripts/wiki_upload_assets.py`。发布 HTML 页面时,页面内 `<style>` 应通过 `--extract-style-to-script-css` 写入 Wiki.js 页面 CSS;页面内 `<script>` 应通过 `--extract-script-to-script-js` 写入页面 JS。Wiki.js 会把 `scriptJs` 原样注入页面 body,因此发布脚本会保留完整 `<script>...</script>` 包裹。
曾出现所有图片 URL 返回 `500 Internal Server Error`,但数据库 `assets` / `assetData` 中记录和二进制数据都存在的情况。根因是 Wiki.js 的 storage target `disk` 被启用,但 `config.path` 为空;Wiki.js 的 Local File System storage 要求绝对路径,空路径会导致资产服务读取/缓存文件路径异常。修复方式是通过 Wiki.js GraphQL `storage.updateTargets``disk` 的路径设为绝对目录 `/wiki/data/storage`,随后执行 `storage.executeAction(targetKey: "disk", handler: "dump")`,并在操作前后临时授予/移除 `manage:system`
验证图片时不要只用 `HEAD` 或只查数据库。应实际 `GET` 资产 URL,并确认 HTTP 200、`Content-Type``image/*`。上传脚本现在默认会做真实 GET 验证,例如:
```bash
python scripts/wiki_upload_assets.py --source-dir data/KSP/wiki-upload-echo-current --file echo_mk2_cross_section.png --folder-slug echo --skip-existing
```
如果 Echo / Enterprise 图片误传到 `/vulture`,先 dry-run 查看目标,再通过官方 `assets.deleteAsset` 删除错放资源。清理脚本只匹配 `/vulture` 下文件名以 `echo_``enterprise_` 开头的资产,并在操作后移除临时系统权限:
```bash
py scripts/wiki_cleanup_misplaced_assets.py
py scripts/wiki_cleanup_misplaced_assets.py --delete
py scripts/wiki_verify_asset_cleanup.py
```
六个航天飞机页面的图片点击放大由 `scripts/wiki_apply_image_lightbox.py` 统一注入,发布后可用以下脚本做浏览器级验证:
```bash
py scripts/wiki_verify_lightbox.py
```
## 下一步建议
1. 把 Communication、Tank、Vehicle Cost 的页面也补成 CRUD
+168
View File
@@ -0,0 +1,168 @@
import re
import sys
from pathlib import Path
HTML_FILES = [
"data/wiki/vulture_shuttle_zh.html",
"data/wiki/vulture_shuttle_en.html",
"data/wiki/echo_shuttle_zh.html",
"data/wiki/echo_shuttle_en.html",
"data/wiki/enterprise_shuttle_zh.html",
"data/wiki/enterprise_shuttle_en.html",
]
STYLE_RE = re.compile(r"</style>", re.IGNORECASE)
SCRIPT_RE = re.compile(r"<script>.*?</script>", re.IGNORECASE | re.DOTALL)
LIGHTBOX_CSS = """
.wiki-article img[data-wiki-asset], .vulture-article img[data-wiki-asset] { cursor: zoom-in; }
.wiki-image-lightbox { position: fixed; inset: 0; display: flex; align-items: center; justify-content: center; padding: 24px; background: rgba(32,33,34,0.84); z-index: 10000; box-sizing: border-box; }
.wiki-image-lightbox[hidden] { display: none; }
.wiki-image-lightbox-panel { position: relative; width: min(1120px, 96vw); max-height: 92vh; overflow: auto; margin: 0; padding: 10px; border: 1px solid #a2a9b1; background: #fff; box-shadow: 0 12px 38px rgba(0,0,0,0.35); box-sizing: border-box; }
.wiki-image-lightbox-panel img { display: block; width: 100%; height: auto; max-height: 82vh; object-fit: contain; background: #000; }
.wiki-image-lightbox-close { position: sticky; top: 0; float: right; width: 30px; height: 30px; margin: -2px -2px 8px 10px; border: 1px solid #a2a9b1; background: #eaecf0; color: #202122; font-weight: bold; line-height: 26px; cursor: pointer; }
.wiki-image-lightbox-close:hover, .wiki-image-lightbox-close:focus { background: #fff; }
body.wiki-image-lightbox-open { overflow: hidden; }
@media (max-width: 820px) { .wiki-image-lightbox { padding: 12px; } .wiki-image-lightbox-panel { width: 98vw; max-height: 94vh; } }
""".strip()
LIGHTBOX_JS = """
(function () {
function ready(fn) {
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', fn, { once: true });
} else {
fn();
}
}
function captionFor(image) {
var container = image.closest('figure, td, tr, .infobox-image');
if (!container) {
return image.alt || '';
}
var caption = container.querySelector('.thumbcaption');
return caption ? caption.textContent.trim() : (image.alt || '');
}
function install(article) {
if (!article || article.dataset.lightboxReady === '1') {
return;
}
article.dataset.lightboxReady = '1';
var overlay = document.createElement('div');
overlay.className = 'wiki-image-lightbox';
overlay.hidden = true;
overlay.innerHTML = '<figure class="wiki-image-lightbox-panel" role="dialog" aria-modal="true"><button class="wiki-image-lightbox-close" type="button" aria-label="Close image preview">x</button><img alt=""><figcaption class="thumbcaption"></figcaption></figure>';
document.body.appendChild(overlay);
var previewImage = overlay.querySelector('img');
var previewCaption = overlay.querySelector('figcaption');
var closeButton = overlay.querySelector('.wiki-image-lightbox-close');
function closePreview() {
overlay.hidden = true;
previewImage.removeAttribute('src');
previewCaption.textContent = '';
document.body.classList.remove('wiki-image-lightbox-open');
}
function openPreview(image, event) {
event.preventDefault();
event.stopPropagation();
if (event.stopImmediatePropagation) {
event.stopImmediatePropagation();
}
previewImage.src = image.currentSrc || image.src;
previewImage.alt = image.alt || '';
previewCaption.textContent = captionFor(image);
overlay.hidden = false;
document.body.classList.add('wiki-image-lightbox-open');
closeButton.focus();
}
Array.prototype.forEach.call(article.querySelectorAll('img[data-wiki-asset]'), function (image) {
if (image.dataset.lightboxBound === '1') {
return;
}
image.dataset.lightboxBound = '1';
image.addEventListener('click', function (event) {
openPreview(image, event);
}, true);
});
article.addEventListener('click', function (event) {
var image = event.target.closest && event.target.closest('img[data-wiki-asset]');
if (!image || !article.contains(image) || image.closest('.wiki-image-lightbox')) {
return;
}
openPreview(image, event);
});
closeButton.addEventListener('click', closePreview);
overlay.addEventListener('click', function (event) {
if (event.target === overlay) {
closePreview();
}
});
document.addEventListener('keydown', function (event) {
if (event.key === 'Escape' && !overlay.hidden) {
closePreview();
}
});
}
function installAll() {
Array.prototype.forEach.call(document.querySelectorAll('.wiki-article, .vulture-article'), install);
}
ready(function () {
installAll();
if (window.MutationObserver) {
var observer = new MutationObserver(installAll);
observer.observe(document.body, { childList: true, subtree: true });
}
window.setTimeout(installAll, 500);
window.setTimeout(installAll, 1500);
window.setTimeout(installAll, 3000);
});
}());
""".strip()
def ensure_css(content):
if ".wiki-image-lightbox" in content:
return content
if not STYLE_RE.search(content):
raise RuntimeError("No style block found")
return STYLE_RE.sub("\n" + LIGHTBOX_CSS + "\n</style>", content, count=1)
def ensure_script(content):
script = "<script>\n" + LIGHTBOX_JS + "\n</script>"
if SCRIPT_RE.search(content):
return SCRIPT_RE.sub(script, content, count=1)
return content.rstrip() + "\n" + script + "\n"
def main():
repo_root = Path.cwd()
changed = 0
for relative in HTML_FILES:
path = repo_root / relative
original = path.read_text(encoding="utf-8")
updated = ensure_script(ensure_css(original))
if updated != original:
path.write_text(updated, encoding="utf-8", newline="\n")
changed += 1
print(f"lightbox_updated file={relative}")
else:
print(f"lightbox_unchanged file={relative}")
print(f"lightbox_summary html_changed={changed}")
return 0
if __name__ == "__main__":
sys.exit(main())
+193
View File
@@ -0,0 +1,193 @@
import argparse
import json
import sys
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import psycopg
from psycopg.rows import dict_row
from wiki_update_page import DEFAULT_WIKI_URL, WikiUpdateError, connect_info, find_auth_group, load_env, login
TEMP_PERMISSIONS = ["manage:system", "manage:assets", "read:assets", "write:assets"]
def normalize_json(value):
if value is None or isinstance(value, (dict, list)):
return value
return json.loads(value)
def get_json_cast(cur, column_name):
cur.execute(
"""
SELECT udt_name
FROM information_schema.columns
WHERE table_name = 'groups' AND column_name = %s
""",
(column_name,),
)
row = cur.fetchone()
if not row:
raise WikiUpdateError(f"groups.{column_name} was not found")
return "jsonb" if row["udt_name"] == "jsonb" else "json"
def grant_temporary_permissions(cur, group):
permissions_cast = get_json_cast(cur, "permissions")
permissions = normalize_json(group["permissions"]) or []
added = []
for permission in TEMP_PERMISSIONS:
if permission not in permissions:
permissions.append(permission)
added.append(permission)
if added:
cur.execute(
f"UPDATE groups SET permissions = %s::{permissions_cast} WHERE id = %s",
(json.dumps(permissions), group["id"]),
)
print(f"temporary_permissions_added={','.join(added) if added else 'none'}")
return added
def remove_temporary_permissions(conninfo, group_id, added):
if not added:
return
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
permissions_cast = get_json_cast(cur, "permissions")
cur.execute("SELECT permissions FROM groups WHERE id = %s", (group_id,))
row = cur.fetchone()
permissions = normalize_json(row["permissions"]) or []
permissions = [permission for permission in permissions if permission not in added]
cur.execute(
f"UPDATE groups SET permissions = %s::{permissions_cast} WHERE id = %s",
(json.dumps(permissions), group_id),
)
conn.commit()
print(f"temporary_permissions_removed={','.join(added)}")
def misplaced_assets(cur):
cur.execute(
"""
SELECT a.id, a.filename, a.hash, a."fileSize", f.slug AS folder_slug
FROM assets a
JOIN "assetFolders" f ON f.id = a."folderId"
WHERE f.slug = 'vulture'
AND (left(a.filename, 5) = 'echo_' OR left(a.filename, 11) = 'enterprise_')
ORDER BY a.filename, a.id
"""
)
return cur.fetchall()
def graphql(wiki_url, query, variables, token):
body = json.dumps({"query": query, "variables": variables}).encode("utf-8")
request = Request(
f"{wiki_url}/graphql",
data=body,
headers={"Content-Type": "application/json", "Accept": "application/json", "Authorization": f"Bearer {token}"},
method="POST",
)
try:
with urlopen(request, timeout=120) as response:
return json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
return {"transportError": f"HTTP {exc.code}", "payload": exc.read().decode("utf-8", errors="replace")}
except URLError as exc:
return {"transportError": str(exc)}
def delete_asset(wiki_url, token, asset_id):
mutation = """
mutation DeleteAsset($id: Int!) {
assets {
deleteAsset(id: $id) {
responseResult { succeeded errorCode slug message }
}
}
}
"""
payload = graphql(wiki_url, mutation, {"id": asset_id}, token)
if payload.get("errors"):
print(f"asset_delete id={asset_id} graphql_errors={json.dumps(payload['errors'], ensure_ascii=True)}")
return False
if payload.get("transportError"):
print(f"asset_delete id={asset_id} transport_error={payload['transportError']} payload={payload.get('payload', '')[:500]}")
return False
result = payload.get("data", {}).get("assets", {}).get("deleteAsset", {}).get("responseResult", {})
print(
"asset_delete id={id} succeeded={succeeded} errorCode={errorCode} slug={slug} message={message}".format(
id=asset_id,
succeeded=result.get("succeeded"),
errorCode=result.get("errorCode"),
slug=result.get("slug"),
message=result.get("message"),
)
)
return bool(result.get("succeeded"))
def parse_args():
parser = argparse.ArgumentParser(description="Delete Echo/Enterprise assets that were mistakenly uploaded under the Vulture folder.")
parser.add_argument("--env-file", default=".env")
parser.add_argument("--wiki-url", default=DEFAULT_WIKI_URL)
parser.add_argument("--auth-group", default="automation")
parser.add_argument("--delete", action="store_true", help="Actually delete matching assets. Without this flag, only lists matches.")
return parser.parse_args()
def main():
args = parse_args()
repo_root = Path.cwd()
env = load_env((repo_root / args.env_file).resolve())
conninfo = connect_info(env)
identities = []
for candidate in (env.get("wiki_useremail"), env.get("wiki_username")):
if candidate and candidate not in identities:
identities.append(candidate)
if not identities or not env.get("wiki_password"):
raise WikiUpdateError("Missing wiki_useremail/wiki_username or wiki_password in .env")
temporary_group_id = None
added_permissions = []
try:
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
assets = misplaced_assets(cur)
print(f"misplaced_asset_count={len(assets)}")
for asset in assets:
print(f"misplaced_asset id={asset['id']} folder=/{asset['folder_slug']} filename={asset['filename']} size={asset['fileSize']}")
if not args.delete or not assets:
return 0
group = find_auth_group(cur, args.auth_group, [identity.lower() for identity in identities])
added_permissions = grant_temporary_permissions(cur, group)
temporary_group_id = group["id"]
conn.commit()
token = login(args.wiki_url.rstrip("/"), identities, env["wiki_password"])
ok = True
for asset in assets:
ok = delete_asset(args.wiki_url.rstrip("/"), token, asset["id"]) and ok
if not ok:
return 1
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
remaining = misplaced_assets(cur)
print(f"misplaced_asset_remaining={len(remaining)}")
return 0 if not remaining else 1
finally:
if temporary_group_id is not None:
remove_temporary_permissions(conninfo, temporary_group_id, added_permissions)
if __name__ == "__main__":
try:
sys.exit(main())
except WikiUpdateError as exc:
print(f"wiki_cleanup_error={exc}")
sys.exit(2)
+149
View File
@@ -0,0 +1,149 @@
import argparse
import base64
import re
import sys
from pathlib import Path
from PIL import Image, ImageOps
ASSET_SOURCES = {
"/vulture/vulture_shuttle_full_stack_original.png": "data/KSP/wiki-upload/vulture_shuttle_full_stack_at_launch_pad.jpg",
"/vulture/vulture_shuttle_ignition_original.png": "data/KSP/wiki-upload/vulture_shuttle_ignition.jpg",
"/vulture/vulture_ascent_original.png": "data/KSP/wiki-upload/vulture_ascent.jpg",
"/vulture/vulture_booster_recovery_original.png": "data/KSP/wiki-upload/vulture_booster_recovery.jpg",
"/vulture/vulture_booster_separation_original.png": "data/KSP/wiki-upload/vulture_booster_separation.jpg",
"/echo/echo_takeoff_side_view.png": "data/KSP/wiki-upload-echo-current/echo_takeoff_side_view.png",
"/echo/echo_mk2_cross_section.png": "data/KSP/wiki-upload-echo-current/echo_mk2_cross_section.png",
"/echo/echo_vtvl.png": "data/KSP/wiki-upload-echo-current/echo_vtvl.png",
"/echo/echo_approach_guanghan_station.png": "data/KSP/wiki-upload-echo-current/echo_approach_guanghan_station.png",
"/echo/echo_high_energy_return_braking.png": "data/KSP/wiki-upload-echo-current/echo_high_energy_return_braking.png",
"/enterprise/enterprise_takeoff_side_view.png": "data/KSP/wiki-upload-enterprise-current/enterprise_takeoff_side_view.png",
"/enterprise/enterprise_payload_bay_view.png": "data/KSP/wiki-upload-enterprise-current/enterprise_payload_bay_view.png",
"/enterprise/enterprise_dorsal_airframe_view.png": "data/KSP/wiki-upload-enterprise-current/enterprise_dorsal_airframe_view.png",
}
HTML_FILES = [
"data/wiki/vulture_shuttle_zh.html",
"data/wiki/vulture_shuttle_en.html",
"data/wiki/echo_shuttle_zh.html",
"data/wiki/echo_shuttle_en.html",
"data/wiki/enterprise_shuttle_zh.html",
"data/wiki/enterprise_shuttle_en.html",
]
IMG_RE = re.compile(r"<img\b[^>]*>", re.IGNORECASE)
SRC_RE = re.compile(r'src="[^"]*"', re.IGNORECASE)
ASSET_RE = re.compile(r'data-wiki-asset="([^"]+)"', re.IGNORECASE)
def jpeg_name(asset_path):
return asset_path.strip("/").replace("/", "__").rsplit(".", 1)[0] + ".jpg"
def write_compressed_jpeg(repo_root, source, target, max_dimension, quality):
image = Image.open(repo_root / source)
image = ImageOps.exif_transpose(image)
image.thumbnail((max_dimension, max_dimension), Image.Resampling.LANCZOS)
if image.mode in ("RGBA", "LA"):
background = Image.new("RGB", image.size, (255, 255, 255))
background.paste(image, mask=image.getchannel("A"))
image = background
elif image.mode != "RGB":
image = image.convert("RGB")
target.parent.mkdir(parents=True, exist_ok=True)
image.save(target, format="JPEG", quality=quality, optimize=True, progressive=True)
def build_data_uris(repo_root, embed_dir, max_dimension, quality):
data_uris = {}
for asset_path, source in ASSET_SOURCES.items():
source_path = repo_root / source
if not source_path.exists():
raise FileNotFoundError(f"Missing source image for {asset_path}: {source}")
target = repo_root / embed_dir / jpeg_name(asset_path)
write_compressed_jpeg(repo_root, source, target, max_dimension, quality)
encoded = base64.b64encode(target.read_bytes()).decode("ascii")
data_uris[asset_path] = f"data:image/jpeg;base64,{encoded}"
print(f"embedded_source asset={asset_path} file={target.as_posix()} bytes={target.stat().st_size}")
return data_uris
def rewrite_img_tag(match, data_uris):
tag = match.group(0)
asset_match = ASSET_RE.search(tag)
if not asset_match:
return tag
asset_path = asset_match.group(1)
data_uri = data_uris.get(asset_path)
if not data_uri:
return tag
if not SRC_RE.search(tag):
return tag
return SRC_RE.sub(f'src="{data_uri}"', tag, count=1)
def restore_img_tag(match):
tag = match.group(0)
asset_match = ASSET_RE.search(tag)
if not asset_match or not SRC_RE.search(tag):
return tag
return SRC_RE.sub(f'src="{asset_match.group(1)}"', tag, count=1)
def rewrite_html(repo_root, html_files, data_uris):
changed = 0
for relative in html_files:
path = repo_root / relative
original = path.read_text(encoding="utf-8")
updated = IMG_RE.sub(lambda match: rewrite_img_tag(match, data_uris), original)
if updated != original:
path.write_text(updated, encoding="utf-8", newline="\n")
changed += 1
count = updated.count('src="data:image/jpeg;base64,')
print(f"html_updated file={relative} data_uri_count={count}")
else:
print(f"html_unchanged file={relative}")
return changed
def restore_html(repo_root, html_files):
changed = 0
for relative in html_files:
path = repo_root / relative
original = path.read_text(encoding="utf-8")
updated = IMG_RE.sub(restore_img_tag, original)
if updated != original:
path.write_text(updated, encoding="utf-8", newline="\n")
changed += 1
count = updated.count('src="data:image/jpeg;base64,')
print(f"html_restored file={relative} remaining_data_uri_count={count}")
else:
print(f"html_unchanged file={relative}")
return changed
def parse_args():
parser = argparse.ArgumentParser(description="Embed compressed image fallbacks in Wiki.js HTML pages.")
parser.add_argument("--embed-dir", default="data/KSP/wiki-embed-current")
parser.add_argument("--max-dimension", type=int, default=1600)
parser.add_argument("--quality", type=int, default=82)
parser.add_argument("--restore-asset-src", action="store_true", help="Restore image src attributes from data-wiki-asset paths")
return parser.parse_args()
def main():
args = parse_args()
repo_root = Path.cwd()
if args.restore_asset_src:
changed = restore_html(repo_root, HTML_FILES)
print(f"restore_summary html_changed={changed}")
return 0
data_uris = build_data_uris(repo_root, Path(args.embed_dir), args.max_dimension, args.quality)
changed = rewrite_html(repo_root, HTML_FILES, data_uris)
print(f"embed_summary assets={len(data_uris)} html_changed={changed}")
return 0
if __name__ == "__main__":
sys.exit(main())
+538
View File
@@ -0,0 +1,538 @@
import argparse
import json
import os
import re
import sys
import time
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import psycopg
from psycopg.rows import dict_row
DEFAULT_WIKI_URL = "http://192.168.195.241:38353"
PAGE_RULE_ROLES = ["write:pages", "manage:pages"]
SYSTEM_PERMISSION = "manage:system"
class WikiUpdateError(RuntimeError):
pass
def load_env(path):
values = {}
if not path.exists():
return values
for raw_line in path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
values[key.strip()] = value.strip().strip('"').strip("'")
return values
def normalize_json(value):
if value is None or isinstance(value, (dict, list)):
return value
if isinstance(value, str):
return json.loads(value)
return value
def normalize_wiki_path(path, locale):
normalized = path.strip().strip("/")
locale_prefix = f"{locale}/"
if normalized.startswith(locale_prefix):
normalized = normalized[len(locale_prefix) :]
return normalized
def rule_matches_path(rule, page_path):
path = str(rule.get("path") or "").strip("/")
if not path:
return True
return page_path == path or page_path.startswith(path.rstrip("/") + "/")
def graphql(wiki_url, query, variables=None, token=None):
body = json.dumps({"query": query, "variables": variables or {}}).encode("utf-8")
headers = {"Content-Type": "application/json", "Accept": "application/json"}
if token:
headers["Authorization"] = f"Bearer {token}"
request = Request(f"{wiki_url}/graphql", data=body, headers=headers, method="POST")
try:
with urlopen(request, timeout=30) as response:
return json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
payload = exc.read().decode("utf-8", errors="replace")
return {"transportError": f"HTTP {exc.code}", "payload": payload}
except URLError as exc:
return {"transportError": str(exc)}
def graphql_has_errors(label, payload):
if payload.get("errors"):
print(f"{label}=errors")
print(json.dumps(payload["errors"], ensure_ascii=True))
return True
if payload.get("transportError"):
print(f"{label}=transport_error {payload['transportError']}")
if payload.get("payload"):
print(payload["payload"][:1000])
return True
print(f"{label}=ok")
return False
def response_result(payload, operation):
result = payload.get("data", {}).get("pages", {}).get(operation, {}).get("responseResult", {})
print(
"pages.{operation}.response=succeeded={succeeded} errorCode={errorCode} slug={slug} message={message}".format(
operation=operation,
succeeded=result.get("succeeded"),
errorCode=result.get("errorCode"),
slug=result.get("slug"),
message=result.get("message"),
)
)
return bool(result.get("succeeded"))
def get_json_cast(cur, column_name):
cur.execute(
"""
SELECT udt_name
FROM information_schema.columns
WHERE table_name = 'groups' AND column_name = %s
""",
(column_name,),
)
row = cur.fetchone()
if not row:
raise WikiUpdateError(f"groups.{column_name} was not found")
return "jsonb" if row["udt_name"] == "jsonb" else "json"
def connect_info(env):
dbname = env.get("WIKIDATABASE") or env.get("PGDATABASE")
required = {
"PGHOST": env.get("PGHOST"),
"PGUSER": env.get("PGUSER"),
"PGPASSWORD": env.get("PGPASSWORD"),
"WIKIDATABASE/PGDATABASE": dbname,
}
missing = [key for key, value in required.items() if not value]
if missing:
raise WikiUpdateError(f"Missing database environment values: {', '.join(missing)}")
return {
"host": env.get("PGHOST"),
"port": int(env.get("PGPORT", "5432")),
"dbname": dbname,
"user": env.get("PGUSER"),
"password": env.get("PGPASSWORD"),
}
def resolve_page(cur, page_id, page_path, locale, missing_ok=False):
if page_id is not None:
cur.execute(
"""
SELECT id, path, title, description, "isPublished", "isPrivate", "editorKey", "localeCode"
FROM pages
WHERE id = %s
""",
(page_id,),
)
else:
cur.execute(
"""
SELECT id, path, title, description, "isPublished", "isPrivate", "editorKey", "localeCode"
FROM pages
WHERE path = %s AND "localeCode" = %s
ORDER BY id
LIMIT 1
""",
(page_path, locale),
)
page = cur.fetchone()
if not page:
if missing_ok:
return None
raise WikiUpdateError(f"Wiki page was not found: locale={locale} path={page_path}")
return page
def fetch_tags(cur, page_id):
try:
cur.execute(
"""
SELECT t.tag
FROM tags t
JOIN "pageTags" pt ON pt."tagId" = t.id
WHERE pt."pageId" = %s
ORDER BY t.tag
""",
(page_id,),
)
return [row["tag"] for row in cur.fetchall()]
except Exception as exc:
print(f"tags_preserved=False reason={type(exc).__name__}")
return []
def find_auth_group(cur, group_name, identities):
cur.execute(
"""
SELECT g.id, g.name, g.permissions, g."pageRules"
FROM groups g
WHERE g.name = %s
AND EXISTS (
SELECT 1
FROM "userGroups" ug
JOIN users u ON u.id = ug."userId"
WHERE ug."groupId" = g.id
AND (lower(u.email) = ANY(%s) OR lower(u.name) = ANY(%s) OR lower(u."providerKey") = ANY(%s))
)
ORDER BY g.id
LIMIT 1
""",
(group_name, identities, identities, identities),
)
group = cur.fetchone()
if not group:
raise WikiUpdateError(f"Authenticated user is not in Wiki.js group: {group_name}")
return group
def ensure_page_rule(cur, group, page_path, patch_page_rule):
page_rules_cast = get_json_cast(cur, "pageRules")
rules = normalize_json(group["pageRules"]) or []
if not isinstance(rules, list):
raise WikiUpdateError("Group pageRules is not a JSON array")
relevant_rules = [rule for rule in rules if isinstance(rule, dict) and rule.get("deny") is not True and rule_matches_path(rule, page_path)]
if not relevant_rules:
raise WikiUpdateError(f"No allow Page Rule matches path: {page_path}")
missing_by_rule = []
changed = False
for rule in relevant_rules:
roles = rule.get("roles")
if not isinstance(roles, list):
roles = []
rule["roles"] = roles
missing = [role for role in PAGE_RULE_ROLES if role not in roles]
print(f"matched_page_rule_path={rule.get('path')} match={rule.get('match') or rule.get('matchType')} roles={roles}")
if missing:
missing_by_rule.append((rule, missing))
if patch_page_rule:
for role in missing:
roles.append(role)
changed = True
if missing_by_rule and not patch_page_rule:
missing_roles = sorted({role for _rule, missing in missing_by_rule for role in missing})
raise WikiUpdateError(
"Matching Page Rule is missing roles: "
+ ", ".join(missing_roles)
+ ". Add them in Wiki.js Groups > Page Rules or rerun with --patch-page-rule after user approval."
)
if changed:
cur.execute(
f"UPDATE groups SET \"pageRules\" = %s::{page_rules_cast} WHERE id = %s",
(json.dumps(rules), group["id"]),
)
print("page_rule_patched=True")
else:
print("page_rule_patched=False")
def grant_temporary_system_permission(cur, group):
permissions_cast = get_json_cast(cur, "permissions")
permissions = normalize_json(group["permissions"]) or []
if SYSTEM_PERMISSION in permissions:
print("temporary_manage_system_added=False reason=already_present")
return False
permissions.append(SYSTEM_PERMISSION)
cur.execute(
f"UPDATE groups SET permissions = %s::{permissions_cast} WHERE id = %s",
(json.dumps(permissions), group["id"]),
)
print("temporary_manage_system_added=True")
return True
def remove_temporary_system_permission(conninfo, group_id):
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
permissions_cast = get_json_cast(cur, "permissions")
cur.execute("SELECT permissions FROM groups WHERE id = %s", (group_id,))
row = cur.fetchone()
permissions = normalize_json(row["permissions"]) or []
permissions = [permission for permission in permissions if permission != SYSTEM_PERMISSION]
cur.execute(
f"UPDATE groups SET permissions = %s::{permissions_cast} WHERE id = %s",
(json.dumps(permissions), group_id),
)
conn.commit()
print("temporary_manage_system_removed=True")
def login(wiki_url, identities, password):
login_query = """
mutation Login($username: String!, $password: String!, $strategy: String!) {
authentication {
login(username: $username, password: $password, strategy: $strategy) {
responseResult { succeeded errorCode slug message }
jwt
}
}
}
"""
for index, identity in enumerate(identities, start=1):
payload = graphql(wiki_url, login_query, {"username": identity, "password": password, "strategy": "local"})
login_data = payload.get("data", {}).get("authentication", {}).get("login") or {}
result = login_data.get("responseResult", {})
print(
"login[{index}]=succeeded={succeeded} errorCode={errorCode} slug={slug} message={message} token_received={token_received}".format(
index=index,
succeeded=result.get("succeeded"),
errorCode=result.get("errorCode"),
slug=result.get("slug"),
message=result.get("message"),
token_received=bool(login_data.get("jwt")),
)
)
if login_data.get("jwt"):
return login_data["jwt"]
raise WikiUpdateError("Wiki.js login failed for all configured identities")
def extract_style_to_script_css(content):
match = re.search(r"<style>\s*(.*?)\s*</style>", content, flags=re.IGNORECASE | re.DOTALL)
if not match:
print("script_css_extracted=False")
return content, ""
css = match.group(1).strip()
content_without_style = (content[: match.start()] + content[match.end() :]).lstrip()
print(f"script_css_extracted=True length={len(css)}")
return content_without_style, css
def extract_script_to_script_js(content):
match = re.search(r"<script>\s*(.*?)\s*</script>", content, flags=re.IGNORECASE | re.DOTALL)
if not match:
print("script_js_extracted=False")
return content, ""
script_js = match.group(0).strip()
content_without_script = (content[: match.start()] + content[match.end() :]).rstrip()
print(f"script_js_extracted=True length={len(script_js)}")
return content_without_script, script_js
def update_page(wiki_url, token, page, args, content, tags, script_css, script_js):
update_query = """
mutation UpdatePage($id: Int!, $content: String!, $editor: String!, $locale: String!, $path: String!, $title: String!, $description: String!, $isPublished: Boolean!, $isPrivate: Boolean!, $tags: [String]!, $scriptCss: String, $scriptJs: String) {
pages {
update(id: $id, content: $content, editor: $editor, locale: $locale, path: $path, title: $title, description: $description, isPublished: $isPublished, isPrivate: $isPrivate, tags: $tags, scriptCss: $scriptCss, scriptJs: $scriptJs) {
responseResult { succeeded errorCode slug message }
page { id path title updatedAt }
}
}
}
"""
payload = graphql(
wiki_url,
update_query,
{
"id": page["id"],
"content": content,
"editor": args.editor or page["editorKey"] or "markdown",
"locale": page["localeCode"],
"path": page["path"],
"title": args.title or page["title"],
"description": args.description if args.description is not None else (page["description"] or ""),
"isPublished": bool(page["isPublished"]),
"isPrivate": bool(page["isPrivate"]),
"tags": tags,
"scriptCss": script_css,
"scriptJs": script_js,
},
token,
)
if graphql_has_errors("pages.update", payload):
return False
return response_result(payload, "update")
def create_page(wiki_url, token, args, page_path, content, script_css, script_js):
create_query = """
mutation CreatePage($content: String!, $description: String!, $editor: String!, $isPublished: Boolean!, $isPrivate: Boolean!, $locale: String!, $path: String!, $publishEndDate: Date, $publishStartDate: Date, $scriptCss: String, $scriptJs: String, $tags: [String]!, $title: String!) {
pages {
create(content: $content, description: $description, editor: $editor, isPublished: $isPublished, isPrivate: $isPrivate, locale: $locale, path: $path, publishEndDate: $publishEndDate, publishStartDate: $publishStartDate, scriptCss: $scriptCss, scriptJs: $scriptJs, tags: $tags, title: $title) {
responseResult { succeeded errorCode slug message }
page { id path title updatedAt }
}
}
}
"""
title = args.title or page_path.rstrip("/").rsplit("/", 1)[-1].replace("_", " ")
payload = graphql(
wiki_url,
create_query,
{
"content": content,
"description": args.description or "",
"editor": args.editor or "markdown",
"isPublished": True,
"isPrivate": False,
"locale": args.locale,
"path": page_path,
"publishEndDate": None,
"publishStartDate": None,
"scriptCss": script_css,
"scriptJs": script_js,
"tags": [],
"title": title,
},
token,
)
if graphql_has_errors("pages.create", payload):
return None
if not response_result(payload, "create"):
return None
page = payload.get("data", {}).get("pages", {}).get("create", {}).get("page")
if not page or not page.get("id"):
print("pages.create.page_missing=True")
return None
page["localeCode"] = args.locale
print(f"created_page_id={page['id']} locale={args.locale} path={page['path']}")
return page
def render_page(wiki_url, token, page_id):
render_query = """
mutation RenderPage($id: Int!) { pages { render(id: $id) { responseResult { succeeded errorCode slug message } } } }
"""
payload = graphql(wiki_url, render_query, {"id": page_id}, token)
if graphql_has_errors("pages.render", payload):
return False
return response_result(payload, "render")
def flush_cache(wiki_url, token):
flush_query = """
mutation FlushCache { pages { flushCache { responseResult { succeeded errorCode slug message } } } }
"""
payload = graphql(wiki_url, flush_query, token=token)
if graphql_has_errors("pages.flushCache", payload):
return False
return response_result(payload, "flushCache")
def verify_frontend(wiki_url, locale, page_path, must_contain, must_not_contain):
url = f"{wiki_url}/{locale}/{page_path}"
ok = True
targets = [("plain", url), ("cache_bust", f"{url}?_={int(time.time())}")]
for label, target_url in targets:
request = Request(target_url, headers={"Cache-Control": "no-cache", "Pragma": "no-cache", "Accept": "text/html"})
with urlopen(request, timeout=30) as response:
text = response.read().decode("utf-8", errors="replace")
print(f"frontend_{label}_http_status={response.status}")
for needle in must_contain:
contains = needle in text
print(f"frontend_{label}_must_contain={json.dumps(needle, ensure_ascii=True)} result={contains}")
ok = ok and contains
for needle in must_not_contain:
contains = needle in text
print(f"frontend_{label}_must_not_contain={json.dumps(needle, ensure_ascii=True)} result={not contains}")
ok = ok and not contains
return ok
def parse_args():
parser = argparse.ArgumentParser(description="Update an existing Wiki.js page through the official API, then render, flush cache, and verify the frontend.")
parser.add_argument("--path", required=True, help="Wiki.js page path without locale, e.g. home/Space_Shuttles/Vulture_Shuttle/html_test")
parser.add_argument("--content-file", required=True, help="UTF-8 local content file to publish")
parser.add_argument("--locale", default="zh")
parser.add_argument("--page-id", type=int)
parser.add_argument("--title")
parser.add_argument("--description")
parser.add_argument("--editor", help="Wiki.js editor key; defaults to the existing page editorKey")
parser.add_argument("--wiki-url", default=os.environ.get("WIKI_URL", DEFAULT_WIKI_URL).rstrip("/"))
parser.add_argument("--env-file", default=".env")
parser.add_argument("--auth-group", default="automation")
parser.add_argument("--patch-page-rule", action="store_true", help="Patch missing write/manage Page Rule roles. Ask the user before using this.")
parser.add_argument("--temporary-manage-system", action="store_true", help="Temporarily grant manage:system for render/flush. Ask the user before using this.")
parser.add_argument("--extract-style-to-script-css", action="store_true", help="Extract the first <style> block from HTML content and publish it as Wiki.js page CSS.")
parser.add_argument("--extract-script-to-script-js", action="store_true", help="Extract the first <script> block from HTML content and publish it as Wiki.js page JavaScript.")
parser.add_argument("--create-if-missing", action="store_true", help="Create the page through Wiki.js pages.create if it does not already exist.")
parser.add_argument("--must-contain", action="append", default=[])
parser.add_argument("--must-not-contain", action="append", default=[])
return parser.parse_args()
def main():
args = parse_args()
repo_root = Path.cwd()
env = load_env((repo_root / args.env_file).resolve())
conninfo = connect_info(env)
page_path = normalize_wiki_path(args.path, args.locale)
content = (repo_root / args.content_file).read_text(encoding="utf-8")
script_css = ""
script_js = ""
if args.extract_style_to_script_css:
content, script_css = extract_style_to_script_css(content)
if args.extract_script_to_script_js:
content, script_js = extract_script_to_script_js(content)
identities = []
for candidate in (env.get("wiki_useremail"), env.get("wiki_username")):
if candidate and candidate not in identities:
identities.append(candidate)
if not identities or not env.get("wiki_password"):
raise WikiUpdateError("Missing wiki_useremail/wiki_username or wiki_password in .env")
temporary_group_id = None
try:
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
page = resolve_page(cur, args.page_id, page_path, args.locale, args.create_if_missing)
if page:
tags = fetch_tags(cur, page["id"])
print(f"page_id={page['id']} locale={page['localeCode']} path={page['path']} preserved_tags={len(tags)}")
rule_path = page["path"]
else:
tags = []
rule_path = page_path
print(f"page_found=False locale={args.locale} path={page_path} create_if_missing=True")
group = find_auth_group(cur, args.auth_group, [identity.lower() for identity in identities])
ensure_page_rule(cur, group, rule_path, args.patch_page_rule)
if args.temporary_manage_system and grant_temporary_system_permission(cur, group):
temporary_group_id = group["id"]
conn.commit()
token = login(args.wiki_url, identities, env["wiki_password"])
if page:
page_ok = update_page(args.wiki_url, token, page, args, content, tags, script_css, script_js)
else:
page = create_page(args.wiki_url, token, args, page_path, content, script_css, script_js)
page_ok = bool(page)
render_ok = render_page(args.wiki_url, token, page["id"]) if page else False
flush_ok = flush_cache(args.wiki_url, token)
frontend_ok = verify_frontend(args.wiki_url, page.get("localeCode", args.locale), page["path"], args.must_contain, args.must_not_contain) if page else False
return 0 if page_ok and render_ok and flush_ok and frontend_ok else 1
finally:
if temporary_group_id is not None:
remove_temporary_system_permission(conninfo, temporary_group_id)
if __name__ == "__main__":
try:
sys.exit(main())
except WikiUpdateError as exc:
print(f"wiki_update_error={exc}")
sys.exit(2)
+236
View File
@@ -0,0 +1,236 @@
import argparse
import html
import json
import mimetypes
import os
import re
import sys
import uuid
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import psycopg
from psycopg.rows import dict_row
from wiki_update_page import DEFAULT_WIKI_URL, WikiUpdateError, connect_info, load_env, login
def sanitize_upload_name(name):
sanitized = re.sub(r"[\s,;#]+", "_", name.lower())
sanitized = re.sub(r"[^a-z0-9._-]", "", sanitized)
return sanitized.strip("._")
def resolve_folder(cur, folder_slug, parent_id):
cur.execute(
"""
SELECT id, name, slug, "parentId"
FROM "assetFolders"
WHERE slug = %s AND "parentId" IS NOT DISTINCT FROM %s
ORDER BY id
LIMIT 1
""",
(folder_slug, parent_id),
)
folder = cur.fetchone()
if not folder:
raise WikiUpdateError(f"Asset folder was not found: slug={folder_slug} parentId={parent_id}")
return folder
def existing_assets(cur, folder_id):
cur.execute(
"""
SELECT id, filename, mime, "fileSize"
FROM assets
WHERE "folderId" IS NOT DISTINCT FROM %s
ORDER BY filename
""",
(folder_id,),
)
return {row["filename"]: row for row in cur.fetchall()}
def verify_asset(cur, folder_id, filename):
cur.execute(
"""
SELECT id, filename, mime, "fileSize"
FROM assets
WHERE "folderId" IS NOT DISTINCT FROM %s AND filename = %s
ORDER BY id DESC
LIMIT 1
""",
(folder_id, filename),
)
return cur.fetchone()
def verify_served_asset(wiki_url, folder_slug, filename):
url = f"{wiki_url}/{folder_slug}/{filename}"
request = Request(url, headers={"Accept": "image/*,*/*", "Cache-Control": "no-cache"})
try:
with urlopen(request, timeout=30) as response:
content_type = response.headers.get("Content-Type", "")
first_bytes = response.read(16)
return response.status, content_type, len(first_bytes), ""
except HTTPError as exc:
preview = exc.read(200).decode("utf-8", errors="replace")
return exc.code, exc.headers.get("Content-Type", ""), 0, response_preview(preview)
except URLError as exc:
return 0, "", 0, str(exc)
def multipart_body(folder_id, file_path, upload_name):
boundary = f"----wikijs-upload-{uuid.uuid4().hex}"
mime = mimetypes.guess_type(upload_name)[0] or "application/octet-stream"
chunks = []
chunks.append(
f"--{boundary}\r\n"
"Content-Disposition: form-data; name=\"mediaUpload\"\r\n\r\n"
f"{json.dumps({'folderId': folder_id})}\r\n".encode("utf-8")
)
chunks.append(
f"--{boundary}\r\n"
f"Content-Disposition: form-data; name=\"mediaUpload\"; filename=\"{upload_name}\"\r\n"
f"Content-Type: {mime}\r\n\r\n".encode("utf-8")
)
chunks.append(file_path.read_bytes())
chunks.append(f"\r\n--{boundary}--\r\n".encode("utf-8"))
return boundary, b"".join(chunks)
def upload_file(wiki_url, token, folder_id, file_path, upload_name):
boundary, body = multipart_body(folder_id, file_path, upload_name)
request = Request(
f"{wiki_url}/u",
data=body,
headers={
"Authorization": f"Bearer {token}",
"Content-Type": f"multipart/form-data; boundary={boundary}",
"Accept": "application/json, text/plain, */*",
},
method="POST",
)
try:
with urlopen(request, timeout=120) as response:
text = response.read().decode("utf-8", errors="replace")
return response.status, text
except HTTPError as exc:
text = exc.read().decode("utf-8", errors="replace")
return exc.code, text
except URLError as exc:
raise WikiUpdateError(f"Upload transport failed for {file_path.name}: {exc}") from exc
def response_preview(text):
plain = html.unescape(re.sub(r"<[^>]+>", " ", text))
plain = re.sub(r"\s+", " ", plain).strip()
return plain[:1600] or text[:1600].replace("\r", " ").replace("\n", " ")
def collect_files(source_dir, names):
if names:
files = [source_dir / name for name in names]
else:
files = sorted(path for path in source_dir.iterdir() if path.is_file())
missing = [str(path) for path in files if not path.exists()]
if missing:
raise WikiUpdateError("Missing local asset files: " + ", ".join(missing))
return files
def parse_args():
parser = argparse.ArgumentParser(description="Upload files to an existing Wiki.js asset folder through the /u endpoint.")
parser.add_argument("--source-dir", required=True, help="Local directory containing files to upload")
parser.add_argument("--file", action="append", default=[], help="Specific file name inside --source-dir. Repeat to upload multiple files.")
parser.add_argument("--upload-name", help="Uploaded filename to use when exactly one file is selected")
parser.add_argument("--folder-slug", required=True, help="Existing Wiki.js asset folder slug, e.g. vulture")
parser.add_argument("--parent-id", type=int, default=None)
parser.add_argument("--wiki-url", default=os.environ.get("WIKI_URL", DEFAULT_WIKI_URL).rstrip("/"))
parser.add_argument("--env-file", default=".env")
parser.add_argument("--skip-existing", action="store_true", help="Skip files whose sanitized names already exist in the target folder")
parser.add_argument("--skip-http-verify", action="store_true", help="Skip the frontend GET check for uploaded asset URLs")
return parser.parse_args()
def main():
args = parse_args()
repo_root = Path.cwd()
env = load_env((repo_root / args.env_file).resolve())
conninfo = connect_info(env)
source_dir = (repo_root / args.source_dir).resolve()
if not source_dir.exists():
raise WikiUpdateError(f"Source directory was not found: {source_dir}")
files = collect_files(source_dir, args.file)
if args.upload_name and len(files) != 1:
raise WikiUpdateError("--upload-name can only be used with exactly one selected file")
identities = []
for candidate in (env.get("wiki_useremail"), env.get("wiki_username")):
if candidate and candidate not in identities:
identities.append(candidate)
if not identities or not env.get("wiki_password"):
raise WikiUpdateError("Missing wiki_useremail/wiki_username or wiki_password in .env")
token = login(args.wiki_url, identities, env["wiki_password"])
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
folder = resolve_folder(cur, args.folder_slug, args.parent_id)
before = existing_assets(cur, folder["id"])
print(f"folder_id={folder['id']} slug={folder['slug']} existing_assets={len(before)}")
uploaded = 0
skipped = 0
for file_path in files:
upload_name = sanitize_upload_name(args.upload_name or file_path.name)
if not upload_name:
raise WikiUpdateError(f"Sanitized filename is empty: {file_path.name}")
if upload_name in before:
if args.skip_existing:
print(f"asset_skip_existing source={file_path.name} upload_name={upload_name}")
skipped += 1
continue
raise WikiUpdateError(f"Asset already exists in folder {folder['slug']}: {upload_name}")
status, text = upload_file(args.wiki_url, token, folder["id"], file_path, upload_name)
print(f"asset_upload source={file_path.name} upload_name={upload_name} status={status} response={response_preview(text)}")
if status < 200 or status >= 300 or text.strip().lower() != "ok":
raise WikiUpdateError(f"Upload failed for {file_path.name}: HTTP {status}")
uploaded += 1
verified = []
for file_path in files:
upload_name = sanitize_upload_name(args.upload_name or file_path.name)
row = verify_asset(cur, folder["id"], upload_name)
if row:
verified.append(upload_name)
public_url = f"/{folder['slug']}/{row['filename']}"
print(f"asset_db_verified filename={row['filename']} size={row['fileSize']} url={public_url}")
if not args.skip_http_verify:
status, content_type, byte_count, error_preview = verify_served_asset(
args.wiki_url,
folder["slug"],
row["filename"],
)
print(
"asset_http_verified filename={filename} status={status} content_type={content_type} bytes_read={bytes_read}".format(
filename=row["filename"],
status=status,
content_type=content_type,
bytes_read=byte_count,
)
)
if status != 200 or not content_type.startswith("image/") or byte_count == 0:
detail = f" response={error_preview}" if error_preview else ""
raise WikiUpdateError(f"Asset URL is not serving an image: {public_url} HTTP {status}{detail}")
elif not args.skip_existing:
raise WikiUpdateError(f"Uploaded asset not found in database: {upload_name}")
print(f"asset_upload_summary uploaded={uploaded} skipped={skipped} verified={len(verified)}")
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except WikiUpdateError as exc:
print(f"wiki_asset_upload_error={exc}")
sys.exit(2)
+97
View File
@@ -0,0 +1,97 @@
import json
import sys
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
import psycopg
from psycopg.rows import dict_row
from wiki_update_page import DEFAULT_WIKI_URL, connect_info, load_env
EXPECTED_REMOVED = [
"/vulture/echo_mk2_cross_section.png",
"/vulture/enterprise_takeoff_side_view.png",
]
EXPECTED_PRESENT = [
"/echo/echo_mk2_cross_section.png",
"/enterprise/enterprise_takeoff_side_view.png",
"/vulture/vulture_shuttle_full_stack_original.png",
]
def normalize_json(value):
if value is None or isinstance(value, (dict, list)):
return value
return json.loads(value)
def verify_database(conninfo):
with psycopg.connect(**conninfo, row_factory=dict_row) as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT count(*) AS count
FROM assets a
JOIN "assetFolders" f ON f.id = a."folderId"
WHERE f.slug = 'vulture'
AND (left(a.filename, 5) = 'echo_' OR left(a.filename, 11) = 'enterprise_')
"""
)
misplaced_count = cur.fetchone()["count"]
print(f"vulture_misplaced_asset_count={misplaced_count}")
cur.execute("SELECT permissions FROM groups WHERE lower(name) = 'automation' LIMIT 1")
group = cur.fetchone()
permissions = normalize_json(group["permissions"]) if group else []
has_manage_system = "manage:system" in (permissions or [])
print(f"automation_has_manage_system={has_manage_system}")
return misplaced_count == 0 and not has_manage_system
def request_image(base_url, path):
try:
request = Request(base_url + path, headers={"User-Agent": "ksp-wiki-asset-cleanup-check"})
with urlopen(request, timeout=30) as response:
content_type = response.headers.get("Content-Type", "")
body_ok = bool(response.read(1))
return response.status, content_type, response.status == 200 and content_type.lower().startswith("image/") and body_ok
except HTTPError as exc:
return exc.code, exc.headers.get("Content-Type", ""), False
except URLError as exc:
return f"urlerror:{exc.reason}", "", False
def verify_http(wiki_url):
ok = True
base_url = wiki_url.rstrip("/")
for path in EXPECTED_REMOVED:
status, content_type, is_image = request_image(base_url, path)
path_ok = not is_image
ok = ok and path_ok
print(f"asset_get path={path} status={status} content_type={content_type} ok={path_ok}")
for path in EXPECTED_PRESENT:
status, content_type, is_image = request_image(base_url, path)
ok = ok and is_image
print(f"asset_get path={path} status={status} content_type={content_type} ok={is_image}")
return ok
def main():
repo_root = Path.cwd()
env = load_env((repo_root / ".env").resolve())
conninfo = connect_info(env)
db_ok = verify_database(conninfo)
http_ok = verify_http(DEFAULT_WIKI_URL)
if db_ok and http_ok:
print("asset_cleanup_validation=ok")
return 0
print("asset_cleanup_validation=failed")
return 1
if __name__ == "__main__":
sys.exit(main())
+64
View File
@@ -0,0 +1,64 @@
import sys
from playwright.sync_api import sync_playwright
BASE_URL = "http://192.168.195.241:38353"
PAGES = [
("zh Vulture", f"{BASE_URL}/zh/home/Space_Shuttles/Vulture_Shuttle"),
("en Vulture", f"{BASE_URL}/en/home/Space_Shuttles/Vulture_Shuttle"),
("zh Echo", f"{BASE_URL}/zh/home/Space_Shuttles/Echo_Shuttle"),
("en Echo", f"{BASE_URL}/en/home/Space_Shuttles/Echo_Shuttle"),
("zh Enterprise", f"{BASE_URL}/zh/home/Space_Shuttles/Enterprise_Shuttle"),
("en Enterprise", f"{BASE_URL}/en/home/Space_Shuttles/Enterprise_Shuttle"),
]
def main():
errors = []
print("lightbox_validation_start", flush=True)
with sync_playwright() as playwright:
print("launching_chromium", flush=True)
browser = playwright.chromium.launch(headless=True, timeout=30000)
for label, url in PAGES:
print(f"checking={label}", flush=True)
page = browser.new_page(viewport={"width": 1366, "height": 900})
page.goto(url, wait_until="commit", timeout=30000)
page.wait_for_timeout(1500)
article = page.locator(".wiki-article, .vulture-article").first
image = page.locator(".wiki-article img[data-wiki-asset], .vulture-article img[data-wiki-asset]").first
image.wait_for(state="visible", timeout=20000)
ready = article.get_attribute("data-lightbox-ready")
print(f"{label}: ready={ready}", flush=True)
if ready != "1":
errors.append(f"{label}: lightbox script not installed, ready={ready!r}")
page.close()
continue
image.click(timeout=10000)
overlay = page.locator(".wiki-image-lightbox:not([hidden])")
try:
overlay.wait_for(state="visible", timeout=5000)
preview_src = overlay.locator("img").first.get_attribute("src") or ""
caption = overlay.locator("figcaption").first.inner_text(timeout=5000).strip()
if not preview_src:
errors.append(f"{label}: empty preview src")
if not caption:
errors.append(f"{label}: empty caption")
print(f"{label}: lightbox_open=True caption_len={len(caption)} preview_src={preview_src[:96]}", flush=True)
page.keyboard.press("Escape")
page.wait_for_function("!document.querySelector('.wiki-image-lightbox:not([hidden])')", timeout=5000)
except Exception as exc:
errors.append(f"{label}: {type(exc).__name__}: {exc}")
page.close()
browser.close()
if errors:
print("ERRORS:")
for error in errors:
print(error)
return 1
print("lightbox_click_validation=ok")
return 0
if __name__ == "__main__":
sys.exit(main())