obsidian

303.

303 posts manually backfilled one by one from Obsidian Publish to Ghost, with great effort and mental + physical expense. But why?

The Goal of a Zettelkasten

For the uninitiated, a zettelkasten is a form of a personal knowledge management system (PKM). The Zettelkasten method prioritizes small pieces of information (i.e. the information that could fit on one flashcard) linked together in a web like format by using IDs.

These two simple features together create emergence, as different notes influence the system in unique ways (A is linked to B, B is linked to C, meaning it possible to get to C from A despite no direct link).

"Wholly new forms of encyclopedias will appear, ready-made with a mesh of associative trails running through them, ready to be dropped into the memex and there amplified."

https://en.wikipedia.org/wiki/As_We_May_Think

In the 2020s, the Zettelkasten has come back into vogue thanks to software like Obsidian and Roam Research. Everyone from academics to business professionals drowning in information deluges have sought the refuge of systems that can scale flexibly without breaking under their own weight.

The New Native

The Zettelkasten, in my belief, is a pivotal piece in the future of information creation and consumption. The system allows for healthy consumption (easy to jump into the graph and easy to jump out), updating of stale information when it is revisited, and most importantly implicit dense forests of linking.

But there is a problem. How do we share this information on the web, the closest technology we have to a global communication layer, a platform that works on a Hypertext layer, but displays information linearly? (think of a Twitter feed, information is delivered in LTR and top to bottom) How can we create a graph like experience when information is hidden behind links?

Obsidian + Publish

This is where Obsidian comes in.

Obsidian is beautiful software. In fact, I like it so much, I wrote a 10,000+ word guide on how to use it to its full potential.

Obsidian offers an official plugin named Publish. Publish functions exactly how you might imagine. It publishes markdown notes from Obsidian onto the web.

Publish, much like Obsidian, is a solid piece of software. For a lightweight hosting solution for Obsidian notes, it is fast, reliable, and intuitive. It functionally operates as a CRUD uploader with options to change custom domains, upload custom JS and CSS files, etc.

Unfortunately, the current version of Publish didn't fit the full laundry list of features I needed to make the Zettelkasten in my head a reality. But what are these features anyway? And why can't I live without them?

The Deal Breakers

I had a list of non negotiable elements I wanted to achieve in this build. I have some pretty strong opinions about the logos, ethos, and pathos of a working & published Zettelkasten, and I wasn't really willing to compromise on anything from this list.

It HAS To Be Fun

First and foremost, my cardinal belief is that writing and reading the Zettelkasten should be fun. I as the writer and you as the reader should feel ideally feel no friction moving from note to note.

Topics should merely be a slight change of context instead of a drastic pivot from one field of expertise to another. The non linear path should be a sandbox for the mind to discover concepts that are surprising -- akin to a pleasant midday walk through the garden of the mind.
"Oh, the hydrangeas look nice today! Or perhaps I'll spend time amongst the cherry blossoms. I think I'll take this tulip home with me."

If it is not fun to garden, if it is not fun to walk the garden -- then the whole enterprise should be abandoned from the outset.

Lazy Utils

In service of making writing an enjoyable and fun process, the work around publishing a piece should be minimized. My goals were as follows:

• Spend minimal time on tedious dev work: this includes maintenance, required features, annoying work like targeting different browsers/optimizing for mobile, SEO, etc.
• Do 100% of the writing process within Obsidian
• One Button Sharing

All three of these allow for me to focus on the act of writing, while being to maintain a high quality visual and tactile experience for the reader as well.

Reader Engagement

Speaking of the reader, I wanted a blog that could engage with the audience on a level past just reading. For better or worse, social media has become the human bazaar of communication. This causes a lot of problems, since most people have a lot of emotional baggage attached to their social profiles, not to mention the patterns of addiction and harm caused from subtraction of context + hostile territory.

For years, I've been looking for a way to solve this problem. I think there are four reliable, solid tools to manage reader-author interactions on the web: self-hosting, email newsletters, RSS, and in-place comments. All of these technologies are tried and true, last years, and many have protocols built into the very fabric of the web.

Having access to all four greatly increases the longevity of a website, as well as the utility the owner can provide to the visitors.

Customizability

Being able to get my hands dirty with the source code allows me to build spirit into the garden. As I evolve as a writer and developer, and as I learn new skills, customizability is paramount to creating a fun experience for the reader.

In fact, the need to have customization is what initially chased me away from Substack, I didn't like how limited I felt by the tools Substack offered (they don't even offer an API!).

Why (Not) Ghost?

Of the dealbreakers listed above Ghost offers a lot, but also misses a few things that I had to try my hand at building manually. From the above list, Ghost provides built-in:

• Minimal dev work for features like fast mobile loading, image optimization, routing, etc. built-in
• Comments built in
• Full text search built in
• RSS built in
• Email newsletters built in
• Admin dashboards built in
• Admin and Content API built in
• Mainstream CSS themes to start from built in
• An official way to make money through Stripe built in (this is uber important for people who want to go full time with their work!)
• Option to self host or host through Ghost's platform

Thanks to Ghost being open source, I don't (probably) have to worry about situations like the below happening (which it did, and because I didn't see the email, I was blindsided and lost access to all the newsletters I wrote on Revue -- like totally locked out)

The deal breakers that had yet to be built:

• One button publish from Obsidian
• Writing in Obsidian and converting Markdown to a published post -- seamless CRUD workflow
• Elements of fun and exploration
• Internal Hover Links

The Obsidian Ghost Plugin

The first and most important extension to Ghost's base offerings was to build an Obsidian plugin that would automatically interface with Ghost without leaving Obsidian.

One button publish was built off a great open source plugin by jaynguyens, which converts Obsidian markdown into HTML and uploads it directly to Ghost using the Ghost API. I forked it and added a number of features including:

• automatically checks if slug exists; if so updates instead of creating a new post
• converts [[ links into a.href and maintains canonical name
• inserts image placeholder cards that match names of local files for easy discovery while uploading
• removes title H1 from notes to prevent duplicate titles
• auto sizes YouTube embeds and auto centers them in the middle of the page
• opens automatically in preferred browser after uploading with the API

For example, here's the wikilink to a.href converter feature code:

// converting <a href="https://bram-adams.ghost.io/link">link</a> to <a href="BASE_URL/id" class="link-previews">Internal Micro</a>for Ghost
	const content = data.content.replace(
		/!*\[\[(.*?)\]\]/g,
		(match: any, p1: string) => {
			if (p1.toLowerCase().includes(".png") || p1.toLowerCase().includes(".jpg") || p1.toLowerCase().includes(".jpeg") || p1.toLowerCase().includes(".gif")) {
				try {
					// get the year
					const year = new Date().getFullYear();
					// get the month
					const monthNum = new Date().getMonth() + 1;
					let month = monthNum.toString();
					if (monthNum < 10) {
						month = `0${monthNum}`;
					}

					return `<figure class="kg-card kg-image-card"><img src="${BASE_URL}/content/images/${year}/${month}/${p1.replace(/ /g, "-").replace(/%20/g, "-")}" alt="${BASE_URL}/content/images/${year}/${month}/${p1.replace(/ /g, "-").replace(/%20/g, "-")}"></img><figcaption>${p1}</figcaption></figure>`;
				} catch (err) {
					console.log(err);
				}
			}

			const [link, text] = p1.split("|");
			const [id, slug] = link.split("#");
			const url = `${BASE_URL}/${id}`;
			const linkText = text || slug || link;
			const linkHTML = `<a href="${url}">${linkText}</a>`;
			return linkHTML;
		}
	);

Fun and Exploration

To make the process of traversing the published Zettelkasten fun, I thought about what I liked about note discovery in Obsidian. The first thoughts that came to mind were tags and internal hover links.

All topics are available at the bottom of each post in addition to read later from items within the same tag. This gives the reader an option to laterally move within a tag or to jump back to the top level of another tag and explore there instead.

jump to a top level search -- a new tag with completely new ideas to explore

or explore within the same tag

To implement these, I edited the code in my Ghost theme to show the posts I wanted to surface for users:

{{!-- https://github.com/TryGhost/Themes/blob/6f71fe0f1b10e9b81678bb7e0fde12489dfc995f/packages/journal/index.hbs#L111-L126 --}}

{{#get "tags" include="count.posts" limit="all" as |topic|}}
    {{!-- <section class="gh-section"> --}}
    <div class="gh-read-next gh-canvas">
        <section class="gh-pagehead">
            <h4 class="gh-pagehead-title">Topics</h4>
        </section>

        <div class="gh-topic">
            {{#foreach topic}}
                <a class="gh-topic-item" href="{{url}}">
                    <h3 class="gh-topic-name">{{name}}</h3>
                    <span class="gh-topic-count">
                        {{plural count.posts empty="0 issues" singular="% issue" plural="% issues"}}
                    </span>
                </a>
            {{/foreach}}
        </div>
    </div>
{{/get}}

Show Me What You GOT

For internal hover links, I used the Microlink API and code injection to create a block in the footer of my Ghost pages that would give all internal links on my site the ability to be hovered (if you are on desktop -- hover this link for an example may take a second to load! 202212181947)

<!-- https://microlink.io/docs/sdk/getting-started/styling --> <style> :root { --microlink-background-color: #1A1A1A; --microlink-color: white; --microlink-border-color: #666; --microlink-hover-border-color: #999; } </style>  <!-- load @microlink/hover-vanilla --> <script src="https://cdn.jsdelivr.net/combine/npm/react@16/umd/react.production.min.js,npm/react-dom@16/umd/react-dom.production.min.js,npm/react-is@16/umd/react-is.production.min.js,npm/styled-components@5/dist/styled-components.min.js,npm/@microlink/mql@latest/dist/mql.min.js,npm/@microlink/hover-vanilla@latest/dist/microlink.min.js"></script>  <!-- intialize `microlinkHover` --> <script> document.addEventListener('DOMContentLoaded', function (event) { microlinkHover('.link-previews') }) </script>
<script defer>
    const BASE_PATH = "https://bram-adams.ghost.io"
	const EXCLUDED_PATHS = ['/', '/about/']
    const EXCLUDED_PATTERNS = ['#/portal/', '/author/', '/tag/', '#fn']
	document.querySelectorAll("a").forEach((a) => {
    	if(
            a.href.includes("https://bram-adams.ghost.io") // only match internal links
           && EXCLUDED_PATHS.filter(path => a.href !== BASE_PATH + path).length == EXCLUDED_PATHS.length // skip exact paths
           && EXCLUDED_PATTERNS.filter(epattern => !a.href.includes(epattern)).length == EXCLUDED_PATTERNS.length // skip paths including a pattern
           && (!a.classList.contains("gh-card-link") && !a.classList.contains("gh-topic-item") && !a.classList.contains("gh-navigation-link"))) // homepage bug
        {
       		a.classList.add("link-previews");
    	}
	});
</script>

Was it Worth It?

I had to add many features to Ghost to "catch up" with Publish, but the stuff I would've had to add to Publish (full text search, mainstream CSS, newsletter, comments, admin pages, Stripe integrations, etc.) would be nigh impossible, so I'm glad overall that I made the move.

The overall UX of Ghost is much more polished than Obsidian Publish currently (though that may one day change!).

If you want to have a blog that could pass as a high quality standalone blog, check out Ghost. The ease of connecting to Obsidian really sealed the deal for me, as I would be able to maintain my writing habits while gaining a new venue to publish in.

What's Next?

I want to continue to make this site more fun as I come across new ideas and techniques along my journey. I'm excited to play with new themes and new visual ways to tell stories.

Concretely, I'd like to add some form of Obsidian block support for Ghost. Being able to link within specific blocks within Obsidian pages is such a treat, it's hard to not have that on my website.

More pressingly, I'm using this time to write. At the top of this post I mentioned I manually uploaded 300+ posts one by one. This is true. What I failed to mention is that my #to-process tag (where all my zettels are sourced from) has over 4000 (oof) items to go through. Some of these will be quick, others will take days to fully process.

In other words, I have a lot of writing to do.

*me right after I rebuild my site from scratch*

If you read this far into this post, I'm gonna assume you're into Zettelkasten stuff. If you want to learn about the process of building a Zettelkasten by watching one be built from the bottom up, do be sure to subscribe!

Thank you for reading, and I hope to walk past you in the garden one day!

Edit 2023-02-03

A comment on Reddit from /u/johnlunney mentioned that it would be helpful to call out the features Publish is missing, so I'd like to carve a little space to do that here.

Here are some things that caused me to move away from Publish.

Limited Control Over Full Feel

Publish gives an option to upload a custom publish.css file, but unfortunately it is up to the user to find the correct CSS classes and reverse engineer them. For example, I added this feature on mobile to be able to swipe to open the sidebar, but I had to inspect the element itself and take a leap of faith that the CSS class I was toggling would control what I hoped. It worked, but the whole thing was honestly a shot in the dark since there is very minimal documentation floating about the web.

// swipe left on mobile to open menu
let touchstartX = 0;
let touchendX = 0;

function checkDirection() {
  // check if greater than 100px
  // if swipe right close menu
    if (touchendX < touchstartX && touchstartX - touchendX > 100) {
    document
        .querySelector(".published-container")
        .classList.remove("is-left-column-open");
    }

  if (touchendX > touchstartX && touchendX - touchstartX > 100) {
    document
      .querySelector(".published-container")
      .classList.add("is-left-column-open");
  }
}

document.addEventListener("touchstart", (e) => {
  touchstartX = e.changedTouches[0].screenX;
});

document.addEventListener("touchend", (e) => {
  touchendX = e.changedTouches[0].screenX;
  checkDirection();
});

There are many other examples like this, but off the top of my head:

• No control over the way the sidebar displays file order
• No way to add onClick() events
• Adding comments and other features is hard due to loading order (onLoad() didn't work since I think Publish is a SPA app?)
• Hard to anticipate how publish.css/js changes will propagate to mobile
• Unnecessarily large gutters for content
• Full Text Search
• Limited Space (4GB) for everything

Unfortunately all this adds up to a novel (read: kinda weird) UX for both the developer and readers to learn.

I think Zettelkastens across the web would be a lot more popular if SEO was better and they adhered to more mainstream principles.

Gitignore

The "gitignore" feature in Publish is awesome. It makes uploading changed and new files a breeze, as well as deleting them. Unfortunately, this also causes all the files in Obsidian to be tracked, which is pretty annoying.

To combat this, the Obsidian devs added an option to ignore folders/files. This works great…until you locally move the files. Then the whole game starts over again. This is pretty annoying if you uploaded something, changed its path and then need to do the whole upload process from scratch.

Limited Communication Around Updates

With the core team hard at work on the app of Obsidian, I felt at times Publish was passed over as a side offering instead of a main project. As I said earlier in the essay: this is fine if you need an immediate readonly hosted solution. But the second you want to scale the public offer, Publish can no longer provide utility, imo.

You can find more of my thoughts on whether or not Obsidian Publish makes for a remarkable venue here: 202301240135