symbex: search Python code for functions and classes, then pipe them into a LLM

Plus notes on the new Functions mechanism for the OpenAI APIs

In this newsletter:

• symbex: search Python code for functions and classes, then pipe them into a LLM

Plus 7 links and 1 quotation and 4 TILs

symbex: search Python code for functions and classes, then pipe them into a LLM - 2023-06-18

I just released a new Python CLI tool called symbex. It's a search tool, loosely inspired by ripgrep, which lets you search Python code for functions and classes by name or wildcard, then see just the source code of those matching entities.

Searching for functions and classes

Here's an example of what it can do. Running in my datasette/ folder:

symbex inspect_hash            

Output:

# File: datasette/inspect.py Line: 17
def inspect_hash(path):
    """Calculate the hash of a database, efficiently."""
    m = hashlib.sha256()
    with path.open("rb") as fp:
        while True:
            data = fp.read(HASH_BLOCK_SIZE)
            if not data:
                break
            m.update(data)

    return m.hexdigest()

I gave it the name of a function (classes work too) and it searched all subfolders of the current directory, found that function and output it to my terminal.

Why is this more useful than ripgrep or any of the many other tools that can do this?

I partly built this to have fun learning Python's ast module, but it's mainly designed to complement my LLM CLI tool for running large language model prompts.

Code explanations with a large language model

Check this out:

symbex inspect_hash | llm --system 'explain succinctly'

Output:

This function calculates the hash of a database file efficiently by reading the file in blocks and updating the hash object using SHA256 algorithm from the hashlib module. The resulting hash value is returned as a hexadecimal string.

This is pretty cool!

llm --system 'explain succinctly' runs a prompt against gpt-3.5-turbo using "explain succinctly" as the system prompt.

The system prompt says what to do, then the content piped to the tool is treated as the data that should be processed.

Using wildcards and guessing what a tool does from tests

That was a pretty basic example. Here's something more fun:

symbex 'test*csv*' | llm --system 'based on these tests guess what this tool does'

I'm using the test*csv* wildcard here to find all of my test functions that mention CSV in the name. There are 18 of those, with names like test_table_csv_with_labels and test_table_csv_cors_headers.

I can count the number of tokens in the test output like this (see llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs for an intro to my ttok tool):

symbex 'test_*csv*' | ttok

2682

That's significantly below the 4000 token limit for gpt-3.5-turbo.

Here's what I get when I ask it "based on these tests guess what this tool does":

This tool seems to be a web application that provides a user interface to query databases and export query results in various formats, including HTML, JSON, and CSV. The tests verify that the tool correctly generates HTML pages with links to export query results in JSON or CSV format, exports tables to CSV format with custom options (such as whether to include labels and nullable labels), and performs streaming of CSV data. The tests also check that the tool sets appropriate HTTP headers, allows CORS, and handles errors and large CSV files gracefully.

That's a good summary of what Datasette does!

I was worried that it might be cheating since ChatGPT already knows about Datasette, but I tried this:

symbex 'test_*csv*' | grep datasette

And got no results, so at least that keyword wasn't being leaked in the test details somehow.

Refactoring code

Let's try something a whole lot more useful:

symbex Request | llm --system 'add type hints to this'

This locates the Request class in Datasette - this one here, and starts adding Python type hints to it. The output started out like this (that code has no type hints at all at the moment):

from typing import Dict, Any, Awaitable
from http.cookies import SimpleCookie
from urllib.parse import urlunparse, parse_qs, parse_qsl

from .multidict import MultiParams


class Request:
    def __init__(self, scope: Dict[str, Any], receive: Awaitable) -> None:
        self.scope = scope
        self.receive = receive

    def __repr__(self) -> str:
        return '<asgi.Request method="{}" url="{}">'.format(self.method, self.url)

    @property
    def method(self) -> str:
        return self.scope["method"]

    @property
    def url(self) -> str:
        return urlunparse(
            (self.scheme, self.host, self.path, None, self.query_string, None)
        )

    @property
    def url_vars(self) -> Dict[str, str]:
        return (self.scope.get("url_route") or {}).get("kwargs") or {}
    
    # ...

Now this is getting impressive! Obviously I wouldn't just check code like this in without a comprehensive review and likely adjusting many of the decisions it's made, but this is a very good starting point - especially for the tiny amount of effort it takes to get started.

Picking a name for the tool

The most time-consuming part of this project ended up being picking the name!

Originally I planned to call it py-grep. I checked https://pypi.org/project/py-grep/ and it was available, so I spun up the first version of the tool and attempted to upload it to PyPI.

PyPI gave me an error, because the name was too similar to the existing pygrep package. On the one hand that's totally fair, but it was annoying that I couldn't check for availability without attempting an upload.

I turned to ChatGPT to start brainstorming new names. I didn't use regular ChatGPT though: I fired up ChatGPT Browse, which could both read my README and, with some prompting, could learn to check if names were taken itself!

I wrote up the full process for this in a TIL: Using ChatGPT Browse to name a Python package.

---

Quote 2023-06-12

Cellphones are the worst thing that’s ever happened to movies. It’s awful. [...] I think you could talk to a hundred storytellers and they would all tell you the same thing. It’s so hard to manufacture drama when everybody can get a hold of everybody all the time. It’s just not as fun as in the old days when the phone would ring and you didn’t know who was calling.

Steven Soderbergh

---

Link 2023-06-13 OpenAI: Function calling and other API updates: Huge set of announcements from OpenAI today. A bunch of price reductions, but the things that most excite me are the new gpt-3.5-turbo-16k model which offers a 16,000 token context limit (4x the existing 3.5 turbo model) at a price of $0.003 per 1K input tokens and $0.004 per 1K output tokens - 1/10th the price of GPT-4 8k.

The other big new feature: functions! You can now send JSON schema defining one or more functions to GPT 3.5 and GPT-4 - those models will then return a blob of JSON describing a function they want you to call (if they determine that one should be called). Your code executes the function and passes the results back to the model to continue the execution flow.

This is effectively an implementation of the ReAct pattern, with models that have been fine-tuned to execute it.

They acknowledge the risk of prompt injection (though not by name) in the post: "We are working to mitigate these and other risks. Developers can protect their applications by only consuming information from trusted tools and by including user confirmation steps before performing actions with real-world impact, such as sending an email, posting online, or making a purchase."

---

TIL 2023-06-13 Running OpenAI's large context models using llm:

OpenAI announced new models today. Of particular interest to me is the new gpt-3.5-turbo-16k model, which provides GPT 3.5 with a 16,000 token context window (up from 4,000) priced at 1/10th of GPT-4 - $0.003 per 1K input tokens and $0.004 per 1K output tokens. …

---

Link 2023-06-13 Llama encoder and decoder: I forked my GPT tokenizer Observable notebook to create a similar tool for exploring the tokenization scheme used by the Llama family of LLMs, using the new llama-tokenizer-js JavaScript library.

---

Link 2023-06-14 Emergency Pod: OpenAI's new Functions API, 75% Price Drop, 4x Context Length: I participated in a Twitter Spaces conversation last night about the new OpenAI functions mechanism. The recording has now been turned into a Latent Space podcast, and swyx has accompanied the recording with a detailed write-up of the different topics we covered.

---

Link 2023-06-14 Example of OpenAI function calling API to extract data from LAPD newsroom articles: Fascinating code example from Kyle McDonald. The OpenAI functions mechanism is intended to drive custom function calls, but I hadn't quite appreciated how useful it can be ignoring the function calls entirely. Kyle instead uses it to define a schema for data he wants to extract from a news article, then uses the gpt-3.5-turbo-0613 to get back that exact set of extracted data as JSON.

---

TIL 2023-06-15 Using fs_usage to see what files a process is using:

Today I wanted to figure out where the vercel CLI tool on my Mac kept its authentication tokens. …

---

TIL 2023-06-15 Syncing slide images and audio in iMovie:

I found an MP3 recording of an old talk I gave and decided to use the slides from that talk to create a video using iMovie. …

---

Link 2023-06-15 When Zeppelins Ruled The Earth: 15 years ago I put together a talk about the history of Zeppelins which I presented a bunch of different times in various different configurations. As far as I know there are no existing videos of it, but I found an MP3 recording today and decided to splice it together with the slides to create a video of the 6m47s version I gave at the Skillswap on Speed lightning talks event in Brighton on the 28th October 2008.

Notes on how I edited the video together using iMovie in the via link.

---

Link 2023-06-17 sqlean.py: Python's sqlite3 with extensions: Anton Zhiyanov built a new Python package which bundles a fresh, compiled copy of SQLite with his SQLean family of C extensions built right in. Installing it gets you the latest SQLite - 3.42.0 - with nearly 200 additional functions, including things like define() and eval(), fileio_read() and fileio_write(), percentile_95() and uuid4() and many more. "import sqlean as sqlite3" works as a drop-in replacement for the module from the standard library.

---

Link 2023-06-17 LLM 0.4: I released a major update to my LLM CLI tool today - version 0.4, which adds conversation mode and prompt templates so you can store and re-use interesting prompts, plus a whole bunch of other large and small improvements.

I also released 0.4.1 with some minor fixes and the ability to install the tool using Hombrew: brew install simonw/llm/llm

---

TIL 2023-06-18 Using ChatGPT Browse to name a Python package:

I needed a name for my new Python CLI tool. …

---