programming

With the announcement of ChatGPT Plugins using the OpenAPI spec, developers the world over simultaneously thought: "damn, maybe I should've made those Swagger docs after all, lmao".

For NextJS users (at least), the I wrote a program that automatically introspects the /api folder and finds endpoints within the handler method and uses GPT to infer an OpenAPI spec. Meta!!

How does this work?

First, we define two helper functions: getApiRoutes that uses glob to find all TypeScript files in the pages/api directory, and getHandlerFunction that extracts the handler function from each file using Babel to parse the code and returns an array of {name, content} objects with the name being the filename, and content being a string slice containing only the handler function code.

Then, we define a function called inferOpenApiSpec, which calls both previously defined helper functions to obtain an array of handlers with their respective contents, then loops over each item in this array.
For each handler function found, it sends a message as input to GPT-3 along with some preamble text describing how to convert TS functions into JSDoc YAML.

Once it receives a response from GPT-3 it writes resulting contents into ./openapi/{handlerFunction.name}.js.

/* 
1. get the handler function from files
2. use gpt to infer the openapi yaml from the spec
3. return the result as a string
*/
import { glob } from "glob";
import { readFileWrapper, writeFileWrapper } from "./helpers";
import { loadEnvConfig } from "@next/env";
loadEnvConfig(process.cwd());

import { Configuration, OpenAIApi } from "openai";
const configuration = new Configuration({
  organization: "…",
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

const SKIP_FILES = []; // files to skip from api folder

import * as babel from "@babel/core";
// needs to be required instead of imported to work
const parser = require("@babel/parser");

const getApiRoutes = async () => {
  const apiRoutes = await glob("pages/api/**/*.ts");
  return apiRoutes;
};

const getHandlerFunction = async (apiRoutes: string[]) => {
  const handlerFunctions = [];
  for (const apiRoute of apiRoutes) {
    const file = await readFileWrapper(apiRoute);

    const ast = parser.parse(file, {
      sourceType: "module",
      plugins: ["typescript"],
    });

    babel.traverse(ast, {
      ExportDefaultDeclaration(path) {
        const declaration = path.node.declaration;
        if (
          declaration.type === "FunctionDeclaration" &&
          declaration.id.name === "handler"
        ) {

          handlerFunctions.push({ content: file.slice(declaration.start, declaration.end), name: apiRoute.slice(10, -3) });
        }
      },
    });
  }

  return handlerFunctions.filter((handlerFunction) => !SKIP_FILES.includes(handlerFunction.name));
};

const inferOpenApiSpec = async () => {
  const apiRoutes = await getApiRoutes();
  const handlerFunctions = await getHandlerFunction(apiRoutes);

  for (const handlerFunction of handlerFunctions) {
    console.log(`Reading ${handlerFunction.name}`);
    const completion = await openai.createChatCompletion({
      model: "gpt-3.5-turbo",
      messages: [
        {
          role: "system",
          content:
            "I am a bot that converts handler functions into OpenAPI specs (the path only). I only return YAML in JSDoc format.",
        },
        {
          role: "user",
          content:
            `I want to convert this handler function with path /api/${handlerFunction.name} into an OpenAPI spec: ${handlerFunction.content}`
        },
      ],
    });
    const res = completion.data.choices[0].message.content;
    const openApiSpec = res.slice(res.indexOf("```yaml") + 7, res.indexOf("```", res.indexOf("```yaml") + 7));
    let extractedJSDoc = openApiSpec.slice(openApiSpec.indexOf("/**") + 3, openApiSpec.indexOf("*/"))
    .replace("openapi", "swagger")
    
    extractedJSDoc = "/**\n" + extractedJSDoc + "\n*/\n";
    console.log("Full response:");
    console.log(res)
    console.log(`Writing ${handlerFunction.name}`);
    await writeFileWrapper(`./openapi/${handlerFunction.name}.js`, extractedJSDoc);
  }
};

inferOpenApiSpec();

bramadams.dev is a reader-supported published Zettelkasten. Both free and paid subscriptions are available. If you want to support my work, the best way is by taking out a paid subscription.

In this code block, we will learn how to remove frontmatter from a message and parse it using two different functions. These functions are useful for Obsidian plugin developers who need to process the body of a note without its metadata.

The first function removeYMLFromMessage removes the YAML front matter from a given string by replacing it with an empty string. The second function getFrontmatter uses the gray-matter library to extract the front matter object from a MarkdownView instance in Obsidian. Let's dive into each of these functions:

Removing Front Matter

removeYMLFromMessage(message: string) {
	try {
		const YAMLFrontMatter = /---\s*[\s\S]*?\s*---/g;
		const newMessage = message.replace(YAMLFrontMatter, "");
		return newMessage;
	} catch (err) {
		throw new Error("Error removing YML from message" + err);
	}
}

This function takes in a message parameter which is expected to be a string containing YAML front matter at the beginning of it. It then creates a regular expression pattern that matches any text between two sets of three dashes (---). This pattern is stored in YAMLFrontMatter. Finally, it replaces all instances of this pattern with an empty string using JavaScript's built-in replace method.

Parsing Front Matter

import matter from "gray-matter";

getFrontmatter(view: MarkdownView): Chat_MD_FrontMatter {
	try {
        const metaMatter =
            app.metadataCache.getFileCache(noteFile)?.frontmatter;
        const data = matter(view.getViewData());

        const frontmatter = {
	        title: metaMatter?.title || view.file.basename,
	        ...,
	        ...data.data
        };

        return frontmatter;
    } catch (err) {
        throw new Error("Error getting frontmatter");
    }
}

This function takes in an instance of MarkdownView, which represents the currently active view within Obsidian's editor window. It then retrieves information about the current file being viewed through Obsidian's API and extracts its cached metadata using .getFileCache().

Next, it uses gray-matter library to extract both content and metadata objects as separate properties on returned object called 'data'. Then, we create our own custom object called 'frontmatter' where we can add additional properties or override existing ones if needed.

Finally, this parsed data is returned as an object containing various fields such as title, tags etc., which can be used by plugins for further processing.

bramadams.dev is a reader-supported published Zettelkasten. Both free and paid subscriptions are available. If you want to support my work, the best way is by taking out a paid subscription.

The following code block is a TypeScript function that calls the OpenAI API to generate text based on given parameters.

Motivation

Calling OpenAI's API from Obsidian requires a lot of forethought on what variables to include or exclude.

This particular tutorial also highlights streaming, which OpenAI passes back as an entire array and then timers are artificially added to show the user the text coming in a character at a time^[1] Finally, these characters are written into the Obsidian editor in real time.

Tutorial

Let's go through each parameter and what it does:

The function uses request library internally with async/await syntax. If streaming is enabled, it splits up the response into individual lines using newlines (\n\n) as delimiters, removes any metadata at beginning of line ("data: "), parses JSON from each line, extracts delta content from choices field within JSON object returned by API call, appends delta content onto editor instance passed into method call while respecting typing speed settings set elsewhere in codebase.

If streaming is disabled then it simply waits until entire response has been received before parsing JSON data out of stringified HTTP body returned by request library call.

In case there's any error during API call process then it throws an error with details logged into console along with displaying notice about issue calling OpenAI API.

bramadams.dev is a reader-supported published Zettelkasten. Both free and paid subscriptions are available. If you want to support my work, the best way is by taking out a paid subscription.

async callOpenAIAPI(
		editor: Editor,
		messages: { role: string; content: string }[],
		model = "gpt-3.5-turbo",
		max_tokens = 250,
		temperature = 0.3,
		top_p = 1,
		presence_penalty = 0.5,
		frequency_penalty = 0.5,
		stream = true,
		stop: string[] | null = null,
		n = 1,
		logit_bias: any | null = null,
		user: string | null = null
	) {
		try {
			console.log("calling openai api");
			

			const response = await request({
				url: `https://api.openai.com/v1/chat/completions`,
				method: "POST",
				headers: {
					Authorization: `Bearer ${this.settings.apiKey}`,
					"Content-Type": "application/json",
				},
				contentType: "application/json",
				body: JSON.stringify({
					model: model,
					messages: messages,
					max_tokens: max_tokens,
					temperature: temperature,
					top_p: top_p,
					presence_penalty: presence_penalty,
					frequency_penalty: frequency_penalty,
					stream: stream,
					stop: stop,
					n: n,
					// logit_bias: logit_bias, // not yet supported
					// user: user, // not yet supported
				}),
			});

			if (stream) {
				// split response by new line
				const responseLines = response.split("\n\n");

				// remove data: from each line
				for (let i = 0; i < responseLines.length; i++) {
					responseLines[i] = responseLines[i].split("data: ")[1];
				}

				const newLine = ``; // hr for assistant
				editor.replaceRange(newLine, editor.getCursor());

				// move cursor to end of line
				const cursor = editor.getCursor();
				const newCursor = {
					line: cursor.line,
					ch: cursor.ch + newLine.length,
				};
				editor.setCursor(newCursor);

				let fullstr = "";

				// loop through response lines
				for (const responseLine of responseLines) {
					// if response line is not [DONE] then parse json and append delta to file
					if (responseLine && !responseLine.includes("[DONE]")) {
						const responseJSON = JSON.parse(responseLine);
						const delta = responseJSON.choices[0].delta.content;

						// if delta is not undefined then append delta to file
						if (delta) {
							const cursor = editor.getCursor();
							if (delta === "`") {
								editor.replaceRange(delta, cursor);
								await new Promise((r) => setTimeout(r, 82)); // what in the actual fuck -- why does this work lol
							} else {
								editor.replaceRange(delta, cursor);
								await new Promise((r) =>
									setTimeout(r, this.settings.streamSpeed)
								);
							}

							const newCursor = {
								line: cursor.line,
								ch: cursor.ch + delta.length,
							};
							editor.setCursor(newCursor);

							fullstr += delta;
						}
					}
				}

				console.log(fullstr);

				return "streaming";
			} else {
				const responseJSON = JSON.parse(response);
				return responseJSON.choices[0].message.content;
			}
		} catch (err) {
			new Notice(
				"issue calling OpenAI API, see console for more details"
			);
			throw new Error(
				"issue calling OpenAI API, see error for more details: " + err
			);
		}
	}

The following code block defines a TypeScript class called ChatTemplates that helps users create new chat templates in Obsidian. It extends the built-in SuggestModal class which provides an interface for displaying and selecting suggestions based on user input.

Motivation

To offer insight from an Obsidian plugin developer using the Obsidian API, which is pretty poorly documented/hard to parse.

Constructor Method

The constructor is like the "setup" function when you're building something new. It takes two things: an instance of the main app (App) and some settings for chat templates (ChatGPT_MDSettings). When we create a new instance of this class, we pass these things into it so that it can use them later on.

getFilesInChatFolder Method

This method gets all files from a specific folder path (which is stored as part of our settings). We only want to show chat templates to users, so we filter out any files that aren't in this folder.

getSuggestions Method

When someone types into the search box while using our plugin, this method will be called with whatever they typed as its argument. This method looks at all available chat template files and returns only those whose names match what was typed by the user (case-insensitive).

renderSuggestion Method

Once we have filtered down our list of possible matches based on what was typed by the user, each remaining suggestion needs to be displayed visually somehow. This method creates HTML elements representing each suggestion item one-by-one and adds them to the UI.

bramadams.dev is a reader-supported published Zettelkasten. Both free and paid subscriptions are available. If you want to support my work, the best way is by taking out a paid subscription.

onChooseSuggestion Method

Finally, when someone clicks or selects one of these suggestions from our list, this function will run! First it displays a message saying which template was selected by using Obsidian's built-in Notice API. Then it reads contents from selected template file using Vault API's .read(), creates new markdown file using Vault API's .create(), writes contents read earlier to newly created markdown file and opens newly created markdown note using Workspace API's .openLinkText().

export class ChatTemplates extends SuggestModal<ChatTemplate> {
	settings: ChatGPT_MDSettings;

	constructor(app: App, settings: ChatGPT_MDSettings) {
		super(app);
		this.settings = settings;
	}

	getFilesInChatFolder(): TFile[] {
		return this.app.vault
			.getFiles()
			.filter(
				(file) => file.parent.path === this.settings.chatTemplateFolder
			);
	}

	// Returns all available suggestions.
	getSuggestions(query: string): ChatTemplate[] {
		const chatTemplateFiles = this.getFilesInChatFolder();

		if (query == "") {
			return chatTemplateFiles.map((file) => {
				return {
					title: file.basename,
					file: file,
				};
			});
		}

		return chatTemplateFiles
			.filter((file) => {
				return file.basename.toLowerCase().includes(query.toLowerCase());
			})
			.map((file) => {
				return {
					title: file.basename,
					file: file,
				};
			});
	}

	// Renders each suggestion item.
	renderSuggestion(template: ChatTemplate, el: HTMLElement) {
		el.createEl("div", { text: template.title });
	}

	// Perform action on the selected suggestion.
	async onChooseSuggestion(
		template: ChatTemplate,
		evt: MouseEvent | KeyboardEvent
	) {
		new Notice(`Selected ${template.title}`);
		const templateText = await this.app.vault.read(template.file);
		// use template text to create new file in chat folder
		const file = await this.app.vault.create(
			`${this.settings.chatFolder}/${getDate()}.md`,
			templateText
		);

		// open new file
		this.app.workspace.openLinkText(file.basename, "", true);
	}
}