carol/soma-parser
Sign in
main
maindefault
soma-parser/README.md
124 lines · 6.0 KBRaw

soma_chat_history_parser

Rust core that replaces the duplicated chat-import pipelines in Soma_ios_app (Swift) and Soma_cap_app (TypeScript) with one shared, schema-conformant implementation. Takes a platform chat export and produces a <base>.chat_history.zip (meta.json + part.1.txt JSONL), with on-device PII desensitization applied before packaging.

The Rust core is exposed to mobile via UniFFI (proc-macro mode), producing native packages built by GitHub Actions:

Package Dir Consumers Artifact
iOS library swift/ SwiftPM wrappers and Capacitor XCFramework zip + generated bindings
Android (Kotlin) android/ any Android app; future Capacitor plugin soma-chat-history-parser-android.aar
npm / Capacitor plugin capacitor/ Soma_cap_app @soma/capacitor-chat-history-parser (npm tarball; wraps the .aar + .xcframework)

Platform support

The two-phase import/export flow supports whatsapp, instagram, imessage, telegram, google, facebook, twitter, tiktok, outlook, wechat, qq, doubao, and other. iMessage requires pdfium at runtime.

Repository layout

soma_chat_history_parser/
  src/                      # Rust core + UniFFI FFI layer (src/ffi.rs)
    ffi.rs                  # UniFFI proc-macro surface (import/get/export/release)
    bin/uniffi-bindgen.rs   # in-tree bindgen CLI (uniffi/cli feature) - regenerates bindings
  uniffi.toml               # Swift module / Kotlin package names
  swift/                    # Generated Swift bindings + local iOS build outputs
  android/                  # Standalone Android Gradle library (.aar)
  capacitor/                # Capacitor plugin (npm) wrapping the .aar + .xcframework
  scripts/                  # binding, native-package, and Pdfium download scripts
  .github/workflows/        # ci.yml, build-ios.yml, build-android.yml, build-capacitor.yml
  tests/                    # e2e (imessage, instagram) + schema checks

FFI surface

src/ffi.rs mirrors the core types, flattening what UniFFI can't cross: PathBuf → String, usize → u64, BTreeMap → HashMap, and the Io/Json error variants → Other(String). Five functions are exported:

  • import_file(input_path, platform) -> ImportStub
  • import_text(text, original_filename, platform) -> ImportStub
  • get_stub(stub_id) -> ImportStub
  • export_stub(stub_id, selection, output_dir) -> ImportGenerationResult
  • release_stub(stub_id) -> bool

Import parses the raw input exactly once and retains all normalized messages in a process-local intermediate store. ImportStub is an opaque id plus session, participant, and count summaries. It remains valid until release_stub, or until the hosting process exits.

ExportSelection.session_ids chooses conversations (empty means all). ExportSelection.participants is the complete export allow-list and supplies each sender's display name. Exactly one entry must set is_target_persona; all omitted participants are excluded. export_stub reads no original input and produces the canonical transcript, desensitized transcript, and .chat_history.zip. Instagram exports preserve supported image attachments in the V1 archive and connect them to messages through media_path.

Errors surface as SomaParserError (Swift: SomaParserError → wrapped by SomaChatParserError; Kotlin: SomaParserException sealed class).

Building locally

# Regenerate Swift + Kotlin bindings from the host cdylib (run after FFI changes):
./scripts/gen-bindings.sh swift  /tmp/sw
./scripts/gen-bindings.sh kotlin /tmp/kt

# Full native packages (need cross toolchains):
./scripts/build-swift-package.sh      # macOS + Xcode -> Swift bindings + xcframework
./scripts/download-pdfium.sh          # Android ABIs + current macOS host -> lib/
./scripts/build-android-package.sh    # Android NDK + cargo-ndk + gradle -> PDF-capable AAR

The in-tree uniffi-bindgen binary uses the uniffi crate's cli feature (there is no separately-published uniffi-bindgen crate in 0.32). Run it via cargo run --features bindgen --bin uniffi-bindgen -- ….

CI (.github/workflows/)

  • ci.yml — on every push/PR: cargo fmt --check, cargo test, cargo build, and a binding-generation smoke (FFI stays valid). Clippy runs advisory (pre-existing lints in ported parsers don't block).
  • Swift job — cross-compiles aarch64-apple-ios + universal simulator (aarch64-apple-ios-sim + x86_64-apple-ios, lipo'd), assembles the SomaChatHistoryParserFFI.xcframework, regenerates Swift bindings, and publishes an XCFramework-only zip to the daily GitHub Release.
  • Android job — downloads Pdfium, cross-compiles 2 ABIs (arm64-v8a/x86_64), regenerates Kotlin bindings, verifies all native libraries, and publishes the PDF-capable AAR.
  • Capacitor job — builds the iOS artifacts (macOS) + Android .aar (Linux), stages both into capacitor/, then npm run build + npm pack. On a v* tag, attaches the npm tarball to the release.

PDF parsing

imessage classifies bubbles by PDF text-line geometry:

  • Rust host uses pdfium-render and the host dylib prepared under lib/macos/<arch>/ by scripts/download-pdfium.sh.
  • Android uses pdfium-render; scripts/build-android-package.sh stages the two 64-bit libpdfium.so files from lib/android/ into the AAR.
  • iOS uses the system PDFKit framework to extract positioned lines, then passes those lines to the same Rust iMessage normalization and stub logic. No unsupported iOS Pdfium binary is required.

The downloader uses gh-proxy.com by default and records SHA-256 checksums in lib/pdfium-sha256.txt.

Status

  • ✅ Rust core (7/12 platforms) + UniFFI FFI.
  • ✅ Swift SPM package + Android/Kotlin .aar package.
  • ✅ Capacitor plugin (capacitor/) wrapping the .aar + .xcframework.
  • ✅ CI workflows (iOS xcframework, Android .aar, Capacitor npm, PR checks).
  • ⏳ Port remaining 5 platforms (google/facebook/twitter/tiktok/outlook).