So you have a rich text editor. You have an AI. You want the AI to suggest changes in the rich text editor. You probably have some image in your head of the AI suggesting changes as you type, live, as if it was another collaborator.

We had that idea, too. It’s much harder than it looks. We couldn’t find anyone who actually does it.

ChatGPT Canvas simply halts the editor while the AI is at work, and they gave an excellent talk about how hard the general area is. Cursor live-edits the file, but it works exclusively for plaintext. Notion AI live-edits your workspace, but edits seem to rewrite the entire block, and blow away your changes if you were typing at the same time. Google Docs and Coda AI both write a suggestion and then ask you to click a button to insert that suggestion.

So: is live collaborative editing with an agent possible? And how ill-advised is it, actually? Come, learn from our pain.

Caveat emptor. I'm pretty sure no one has this completely figured out. If you're working on something like this and want to trade notes, we'd love to hear from you! Via Twitter (I am @hausdorff_space, DMs open), or in the text editor-focused Discord server we co-admin, Text Editors Hate You Too.

What we did

Our goal for the “AI agents” feature in Moment is to allow the user to edit rich text documents while the AI is at work, as if it was just another live collaborator among potentially many other human live collaborators. You can roughly see how the feature works here:

That's the vision, anyway. But, while it is not 100% complete, we’ve seen enough to advocate for it. Roughly the approach is thus:

We believe these decisions make it possible to have live collaborative editing with an agent. The rest of this post is about the details required to make them work.

1. We serialize the document to plain-old Markdown

Ok, so, look, this is controversial, but you want to represent your document data as Markdown text. You do. LLMs are string-in, string-out machines, and they are trained extensively on Markdown. You can try to fine-tune the models, or add a bunch of scaffolding to allow it to output a potentially-invalid JSON string that can be hydrated to a ProseMirror Transaction object (or whatever). OpenAI tried this kind of thing and then say they backed away from it.

So: yes, you can do that. OR, you can have the winds of the entire RL budget of opus-4.5 blowing in your sails, and just use a text format like Markdown.

Yes, I know all the retorts, and they are all wrong.

Recommendations

So, every document is stored (e.g., in database, in .md files) as Markdown text. For the in-browser rich text editor, we use ProseMirror. (More on ProseMirror later; you should definitely use it.) To transform a .md file into a ProseMirror EditorState object, we use remark, and in particular, @handlewithcare/remark-prosemirror.

We use a handful of community-standard Markdown extensions, mostly related to GitHub-Flavored Markdown, e.g., micromark-extension-gfm-task-list-item, which produces checklist TODO items. We have a bunch of customization for features specific to us.

Seriously, not .patch files or CRDT ops. Maybe someday the models will get really good at things like producing accuate line counts in diffs, but for now, they are not, and they are sure to regularly generate nonsense that fails to apply to your document, or even basically-corrupts it. claude, amp, and copilot mostly work by having the models generate regex-replace operations, which they are far better at.

2. The user edits a document via the in-browser rich text editor

In order for users to see suggested changes from the AI, they have to edit using the Moment Desktop App. These suggested changes look like this:

Users can edit the .md files manually, but of course, they won’t see any of this.

Recommendations

We’ll talk about how we actually generate the “suggested changes” transactions in the next couple sections.

For the actual editor view, we use React and @handlewithcare/react-prosemirror. It avoids the pervasive state-tearing issues that are obviously noticeable in hand-rolled React/ProseMirror integrations, as well as in TipTap.

Yes, ProseMirror. I know there is a lot of ProseMirror baggage out there. I've got my own. It is also the only real option to do something this complex. If you have used ProseMirror with React before it seemed like it simply wasn’t working, seriously consider giving react-prosemirror a shot. If you attempt a raw integration or use TipTap, I (unfortunately) pretty much guarantee that for nontrivial usage, you will run into state-tearing issues and they will drive you to the very edge of your sanity.

3. The AI edits the Markdown itself, directly

In our model, users can open an embedded terminal and run claude, amp, or copilot directly. These TUIs edit the .md files directly, and when the Moment Desktop App detects the change, it projects the change “up” into the EditorState. In the next section, we’ll talk about how, but for now you can see copilot at work to in a Moment document (I've entasked it with making the Gettysburg Address more exciting):

Recommendations

This is perhaps peculiar to our product, but if you’re using an embedded terminal to run claude (especially; amp and copilot do better), as of January 2025, it is VERY IMPORTANT that you run xterm.js v6 (NOT v5), and NOT ghostty-web:

4. Construct “AI-suggested changes” by inspecting the diff

In our product, we have to watch the filesystem for changes .md files. This has many complications, so let’s set that aside and talk only about the problem at hand: when the Markdown text changes, we need to compute the differences, and project them “up” into the EditorState as suggested changes.

So you have your ProseMirror EditorState object, and the AI’s ProseMirror EditorState object. The overall idea is to find a list of transactional Steps that transforms yours into theirs. The simplest solution is to do something like the following.

The Extremely Naïve Solution

Our first cut at this will produce some pretty wacky diffs, but it actually works pretty well like 60% of the time.

Our prototype code looked like this (nb., there are probably bugs here!):

Our production implementation of createStepForBlockChange is reasonably complicated, but the simplest thing to do here is to just always replace the block, which is what we do when the types differ:

The last thing we need to do—turning the Steps into a suggested change—looks roughly like this. We add a nice meta tag to indicate it was an “external” change, for bookkeeping!

The Less Naïve Solution

The previous regimen does work, but has two deep flaws (as well as a bunch of more minor flaws we'll just ignore):

This is certainly fixable, with more effort.

Unfortunately, the last step is a “draw the rest of the owl” moment. For us, this is deeply entwined in our collab layer, so we don’t have helpful code to share here.

The idea of prosemirror-collab-commit is to keep track the editor, and then “automatically rebase” incoming changes. You’d have to use their sendableCommit(aiEditorState) to generate the changes, and then receiveCommitTransaction to apply them to userEditorState.

Another approach is to just store the user’s changes, and apply the incoming AI changes “through” the StepMap.

Other considerations

The main one is performance. We have previously written about the lengths we’ve gone through to ensure the app runs at 60fps, even if you have many collaborators.

There is not an easy way to move the “compute the EditorState diff” work off the render thread. We could have introduced significant complexity by adding another JavaScript process to manage these events, but we judged this to be not worth it. So, the diffing does block the render thread. Empirically the user seems not to be typing enough to notice this, though.

An interlude about filesystems

This is more of an aside, but in Moment’s case, claude, amp, and copilot will deal exclusively with .md files, on your local disk. The Moment Desktop App’s job is to watch for changes to those files and “propagate them up” into the EditorState, as described above. But, hidden in this problem is a nice, miniature Byzantine generals problem.

Consider:

And you can’t solve this, fundamentally, without having the AI coordinate with the desktop app.

Our solution to this is kind of lame: we check whether the user is running claude, amp, or copilot in the embedded terminal, and halt write_text_file until it’s done and resolved.

Conclusions

There are a lot of details missing. We do co-operate a community discord called Text Editors Hate You Too if you want to discuss, or have questions.