Embeddings: What they are and why they matter
Embeddings: What they are and why they matter
Plus open questions for AI engineering
In this newsletter:
Embeddings: What they are and why they matter
Open questions for AI engineering
Weeknotes: PyBay, AI Engineer Summit, Datasette metadata and JavaScript plugins
Plus 6 links and 1 quotation and 1 TIL
Embeddings: What they are and why they matter - 2023-10-23
Embeddings are a really neat trick that often come wrapped in a pile of intimidating jargon.
If you can make it through that jargon, they unlock powerful and exciting techniques that can be applied to all sorts of interesting problems.
I gave a talk about embeddings at PyBay 2023. This article represents an improved version of that talk, which should stand alone even without watching the video.
If you're not yet familiar with embeddings I hope to give you everything you need to get started applying them to real-world problems.
In this article:
Here’s the video of the talk that I gave at PyBay:
Open questions for AI engineering - 2023-10-17
Last week I gave the closing keynote at the AI Engineer Summit in San Francisco. I was asked by the organizers to both summarize the conference, summarize the last year of activity in the space and give the audience something to think about by posing some open questions for them to take home.
The term "AI engineer" is a pretty new one: summit co-founder swyx introduced it in this essay in June to describe the discipline of focusing on building applications on top of these new models.
Quoting Andrej Karpathy:
In numbers, there's probably going to be significantly more AI Engineers than there are ML engineers / LLM engineers. One can be quite successful in this role without ever training anything
This was a challenging talk to put together! I've given keynotes about AI before, but those were at conferences which didn't have a focus on AI - my role there was to help people catch up with what had been going on in this fast-moving space.
This time my audience was 500 people who were already very engaged. I had a conversation with the organizers where we agreed that open questions grounded in some of the things I've been writing about and exploring over the past year would be a good approach.
You can watch the resulting talk on YouTube:
Open questions for AI engineering (Day 2 livestream @ 6:55:53)
I've included slides, an edited transcript and links to supporting materials in this post.
Weeknotes: PyBay, AI Engineer Summit, Datasette metadata and JavaScript plugins - 2023-10-22
I've had a bit of a slow two weeks in terms of building things and writing code, thanks mainly to a couple of conference appearances. I did review and land a couple of major contributions to Datasette though.
I gave a talk at PyBay 2023 called "Embeddings: What they are and why they matter", digging into the weird and fun world of word embeddings (see previous posts). I'll be posting detailed notes from that talk tomorrow.
A couple of days after that I gave the closing keynote at the AI Engineer Summit, where I tried to do justice both to the summit and the previous year of developments in AI - no small challenge!
I've published detailed slides and an annotated transcript to accompany that talk: Open questions for AI engineering.
Datasette metadata
Alex Garcia has been driving a major improvement to Datasette in preparation for the 1.0 release: cleaning up Datasette's ungainly metadata system.
Metadata in Datasette was originally meant to support adding data about data - the license, source and description of data exposed through a Datasette instance.
Over time it grew in weird and unintuitive directions. Today, metadata can also be used to configure plugins, provide table-level settings, define canned queries and even control how Datasette's authentication system works.
The name no longer fits!
Alex is fixing this by splitting all of those non-metadata parts of metadata out into a new, separate configuration file, which we've agreed should be called datasette.json or datasette.yaml.
This week we landed a big piece of this: Move permissions, allow blocks, canned queries and more out of metadata.yaml and into datasette.yaml.
There's a bit more work to do on this: in particular, I need to upgrade the datasette publish command to support deploying instances with the new configuration file. I'll be shipping an alpha release as soon as that work is complete.
Datasette's JavaScript plugins API
The other major contribution this week was authored by Cameron Yick.
His Javascript Plugin API (Custom panels, column menu items with JS actions) pull request has been brewing for months. It's a really exciting new piece of the puzzle.
The key idea here is to provide much richer support for Datasette plugins that use JavaScript to modify the Datasette interface. In particular, we want plugins to be able to collaborate with each other.
Cameron's work introduces a JavaScript plugin mechanism that's inspired by Python's pluggy (already used by Datasette). It introduces a hook for adding a custom panel to the Datasette interface, displayed above the main table view. Multiple plugins can use this same area and Datasette will automatically show a tabbed interface for switching between them.
Cameron also built a mechanism for adding extra options to the existing column "cog" action menu. This is similar to Datasette's existing table and database action menu hooks, but allows column features to be added using JavaScript.
I hope to get documentation and some example plugins working on top of this in time for the next Datasette alpha release.
Releases this week
datasette-llm-embed 0.2 - 2023-10-08
Datasette plugin adding a llm_embed(model_id, text) SQL functiondatasette 0.64.5 - 2023-10-08
An open source multi-tool for exploring and publishing data
TIL this week
Deploying the CLIP embedding model on Fly - 2023-10-18
Link 2023-10-14 Multimodality and Large Multimodal Models (LMMs): Useful, extensive review of the current state of the art of multimodal models by Chip Huyen. Chip calls them LMMs for Large Multimodal Models, a term that seems to be catching on.
Link 2023-10-17 Making CRDTs 98% more efficient: Outstanding piece of explanatory writing by Jake Lazaroff showing how he reduced the transmitted state of his pixel art CRDT implementation from 643KB to 15KB using a progression of tricks, each of which is meticulously explained and accompanied by an interactive demo.
Quote 2023-10-17
The paradox of ChatGPT is that it is both a step forward beyond graphical user interfaces, because you can ask for anything, not just what’s been built as a feature with a button, but also a step back, because very quickly you have to memorise a bunch of obscure incantations, much like the command lines that GUIs replaced, and remember your ideas for what you wanted to do and how you did it last week
TIL 2023-10-18 Deploying the CLIP embedding model on Fly:
Inspired by Drew Breunig's Faucet Finder I decided I wanted to deploy an API somewhere that could calculate CLIP embeddings for me. …
Link 2023-10-19 I’m banned for life from advertising on Meta. Because I teach Python.: If accurate, this describes a nightmare scenario of automated decision making.
Reuven recently found he had a permanent ban from advertising on Facebook. They won't tell him exactly why, and have marked this as a final decision that can never be reviewed.
His best theory (impossible for him to confirm) is that it's because he tried advertising a course on Python and Pandas a few years ago which was blocked because a dumb algorithm thought he was trading exotic animals!
The worst part? An appeal is no longer possible because relevant data is only retained for 180 days and so all of the related evidence has now been deleted.
Various comments on Hacker News from people familiar with these systems confirm that this story likely holds up.
Link 2023-10-19 New Default: Underlined Links for Improved Accessibility (GitHub Blog): "By default, links within text blocks on GitHub are now underlined. This ensures links are easily distinguishable from surrounding text."
Link 2023-10-22 Patrick Newman's Software Engineering Management Checklist: This tiny document may have the highest density of good engineering management advice I've ever encountered.
Link 2023-10-22 Solving the Engineering Strategy Crisis: Will Larson's 49m video discussing engineering strategy: what one is and how to build one. He defines an engineering strategy as having two key components: an honest diagnosis of the way things currently work, and a practical approach to making things better.
Towards the end of the talk he suggests that there are two paths to developing a new strategy. The first is to borrow top-down authority from a sponsor such as a CTO, and the second is to work without any borrowed authority, instead researching how things work at the moment and, through documenting that, write a strategy document into existence!
I wanted to build a application the can visualize embeddings easily - u can check out poc here https://youtu.be/dLAFqDCYS8k?si=akaxggxvSXri-aI1 let me know what you think!