{ "attachments": [ "./cover.png" ], "title": "Clustering ideas with local ML/AI models", "tags": [ "ai", "ml", "llm", "genai", "ClusteringIdeasWithAI" ], "year": "2024", "month": "05", "day": "01", "isDir": true, "slug": "topic-clustering-local-models", "type": "entry", "date": "2024-05-01T19:00:00.000Z", "postName": "2024-05-01-topic-clustering-local-models", "html": "

TL;DR: In my previous post, I used APIs from OpenAI to roughly cluster ideas by named topic. In this post, I'll try that again, but this time with local models on my own hardware.

\n\n\n\n\n
\n \n
I asked DALL-E to generate \"an image in a whimsical style depicting astronauts at home clustering ideas brought in from deep space\". Then, I ran a few imagemagick transformations on it for fun.
\n
\n\n\n\n

In my previous post, I glued together a few OpenAI API calls and a k-means clustering algorithm to organize some ideas into named groups.

\n

It worked pretty well, but I'm not always comfortable with the idea of sending all my ideas to someone else's computer (i.e. the Cloud). So, I thought I'd see if I could do the same thing entirely with my own hardware.

\n

Organizing sticky notes automatically

\n

As a refresher, the thing that got me started on this was a feature in Figma's FigJam tool. Given a collection of sticky notes, there's an \"organize\" feature that uses AI to group them into named clusters automatically:

\n
\n \n
A quick demo of FigJam's sticky organization feature - it's more legible in fullscreen view
\n
\n\n

How does it do what it does?

\n

Again, I don't know exactly how FigJam does this. But, here's my own stab at a process:

\n
    \n
  1. we map the notes as points in a virtual semantic space using vector embeddings
    2. \n
  2. k-means clustering can find groups of points that are close together in space
    3. \n
  3. we map the points back to notes and a large language model can help generate labels for each group
    4. \n
\n

The end result is a set of rougly similar clusters of ideas, each with a not-entirely horrible label.

\n

Play along at home with a notebook

\n

If you want to follow along like last time, I've got a small notebook that you can run in your own environment:

\n\n

While you can run this on Google Colab, you might want to install Jupyter Notebook locally on your own computer. That can give you a better feel for how it runs locally.

\n

Personally, I've got a 14-inch, 2021 Apple MacBook Pro with an M1 Pro CPU and 32GB of RAM. It's neither the best nor worst machine in the world. But, that makes it kind of a good test case for how this kind of thing runs on a modern laptop. (Spoiler alert: AI/ML stuff is pretty resource intensive.)

\n

Opening ceremonies (redux)

\n

To kick this off, here's the same list of ideas from my previous post:

\n
items_text = \"\"\"\n- pasta\n- thomas dolby\n- alpha\n- apples\n- cats\n- pears\n- meters\n- brick\n- dogs\n- beta\n- howard jones\n- concrete\n- asphalt\n- milk\n- rebar\n- gillian gilbert\n- hamsters\n- bread\n- butter\n- wendy carlos\n- gamma\n- birds\n- bananas\n- rick wakeman\n- inches\n- glass\n- feet\n- gary numan\n- miles\n- lumber\n- kilometers\n- geoff downes\n\"\"\"\n\n# Split the text into non-empty lines...\nitems = [x for x in items_text.split(\"\\n\") if x]\n
\n

I should probably try a different set of things. But, I'm lazy and this is what I've got for now. Replace these with your own brain dump, if you're playing at home. Next, some code to install modules:

\n
%pip install scikit-learn torch sentence_transformers accelerate\n
\n

Something to notice, versus my previous post, is that I'm not using OpenAI's API client this time. Instead, I'm using the torch and sentence_transformers modules. We'll see how those come into play shortly.

\n

Vector embeddings (strike back)

\n

Again, vector embeddings are like content hashes with the superpower of being the coordinates of a point in a high-dimensional semantic space where distance indicates similarity. If that makes your head spin like it did mine, my previous post rambles on a bit more about this.

\n

Assuming you've nodded along with that last paragraph, here's some code to generate embeddings on your own computer:

\n
from sentence_transformers import SentenceTransformer\n\n# 384 dimensions - https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\n\n# 384 dimensions - https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2')\n\n# 768 dimensions - https://huggingface.co/sentence-transformers/all-mpnet-base-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')\n\n# 768 dimensions - https://huggingface.co/thenlper/gte-base\n# embedding_model = SentenceTransformer('thenlper/gte-base')\n\n# 1024 dimensions - https://huggingface.co/thenlper/gte-large\nembedding_model = SentenceTransformer('thenlper/gte-large')\n\nembeddings = embedding_model.encode(items)\nembeddings\n
\n

Okay, so a lot of that code is commented out. That's because this was a good place to play around. As it turns out, there are many embedding models and I'm still learning how to tell them apart. Until I understand more about the specifics of each model, I'm just going to try a bunch of them and see what happens.

\n

In fact, if you're following along in your own notebook, you should do that. Try uncommenting different models and see what happens:

\n\n

And, just like how I got embeddings from an OpenAI API call, running one of these models locally gives me essentially the data shape - i.e. a list-of-lists of numbers:

\n
array([[-0.02736097,  0.00340217, -0.01076854, ..., -0.02489066,\n         0.01647391, -0.02072625],\n       [-0.03888908,  0.02349574,  0.0143796 , ..., -0.01913134,\n        -0.03127009, -0.0304915 ],\n       [ 0.01443943,  0.02336113,  0.00634783, ...,  0.00120864,\n        -0.01801292, -0.02678853],\n       ...,\n       [ 0.00835066,  0.00595316, -0.01179765, ..., -0.01259451,\n        -0.0165759 , -0.0056388 ],\n       [ 0.0005869 ,  0.01886297, -0.0079366 , ..., -0.04092352,\n        -0.01162215, -0.01117233],\n       [-0.00889315, -0.00544541, -0.02917784, ..., -0.01641204,\n        -0.01544971, -0.01567657]], dtype=float32)\n
\n

K-means clustering (the return of)

\n

This part doesn't change at all from the previous post. We're still using the same KMeans implementation from sklearn to group the embeddings into clusters.

\n

This algorithm it doesn't care where the embeddings came from. They're lists of coordinates - whether from OpenAI or your own machine - so the clustering will work just the same:

\n
from sklearn.cluster import KMeans\nfrom itertools import groupby\n\n# Let's say we want to organize the list into this many clusters\nn_clusters = 9\n\n# Use the k-means algorithm to come up with a cluster ID for each embedding\ncluster_ids = KMeans(n_clusters=n_clusters, n_init='auto').fit_predict(embeddings)\n\n# Associate each cluster ID with the corresponding item\ncluster_ids_with_items = zip(cluster_ids, items)\n\n# Group the pairs of (cluster_id, item) into lists based on cluster ID\ngrouped_cluster_ids_with_items = groupby(\n    sorted(cluster_ids_with_items, key=lambda x: x[0]),\n    key=lambda x: x[0]\n)\n\n# Simplify that whole mess so we just have a list of clustered items\nclustered_items = [\n    [item for cluster_id, item in item_group]\n    for cluster_id, item_group\n    in grouped_cluster_ids_with_items\n]\n\nclustered_items\n
\n

And when I ran this code, I got this:

\n
[['- meters', '- inches', '- feet', '- miles', '- kilometers'],\n ['- alpha', '- beta', '- gamma'],\n ['- brick', '- concrete', '- asphalt', '- rebar', '- glass', '- lumber'],\n ['- howard jones', '- gillian gilbert', '- wendy carlos'],\n ['- apples', '- pears', '- bananas'],\n ['- cats', '- dogs', '- hamsters', '- birds'],\n ['- pasta', '- bread'],\n ['- thomas dolby', '- rick wakeman', '- gary numan', '- geoff downes'],\n ['- milk', '- butter']]\n
\n

This is a good spot to pause and play around to get a sense of things:

\n\n

I'm pretty sure there's a more formal way to evaluate this stuff - and indeed that sort of thing is on my list of things to learn - but just eyeballing it is good enough for me right now.

\n

Generating labels (your own, personal, LLM)

\n

In my previous post, calling OpenAI's chat completions API with a prompt was a pretty simple function under 10 lines long. That kind of convenience is how they \"getcha\" - but, again, this time around I want to do it the hard way on my own computer.

\n

Loading up a large language model is just slightly more complicated than an embedding model:

\n
\"\"\"\nhttps://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/tree/main\n\nThis model is about 2.2GB.\nMy 2021 MacPook Pro with an Apple M1 Pro and 32GB of RAM seems to have no problem with this model.\n\"\"\"\nimport torch\nfrom transformers import pipeline\n\npipe = pipeline(\n    \"text-generation\",\n    model=\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n    torch_dtype=torch.bfloat16,\n    device_map=\"auto\"\n)\n
\n

You may notice here that I've only loaded one model, with no other suggestions in comments. Well, that's because I landed on TinyLlama as the only one that didn't seem to entirely squash my laptop before it could even start generating text. I wanted to take that as a win for now.

\n

Your mileage may vary. If you're following along at home, look up some more LLMs on huggingface and try them here. I'm just starting to get a grasp on the metrics and parameters around these things, which sounds like a whole other post.

\n

Next, let's get into the wishful thinking part - er, I mean prompt engineering:

\n
system_prompt = \"\"\"You are a helpful but terse assistant.\"\"\"\n\nuser_prompt = \"\"\"\nGiven the following list of items, I need a succinct label that effectively encapsulates the overall theme or purpose.\n\nThis is the list of items:\n\n%s\n\nCan you generate a concise, descriptive label for this list? Thanks in advance!\n\"\"\"\n
\n

You may notice that these prompts are way different than the ones I used with OpenAI's API and the gpt-3.5-turbo model. That's because of the following:

\n\n

This also seems like another spot to dig in and play around. Try different models with different prompts and see what happens. I'm going to do more of that, myself.

\n

And, with that, we can get into writing a function to generate a cluster label:

\n
def generate_topic(items):\n    text = \"\\n\".join(items)\n    messages = [\n        {\"role\": \"system\", \"content\": system_prompt},\n        {\"role\": \"user\", \"content\": user_prompt % text},\n    ]\n    prompt = pipe.tokenizer.apply_chat_template(\n        messages,\n        tokenize=False,\n        add_generation_prompt=True\n    )\n    results = pipe(\n        prompt,\n        max_new_tokens=32,\n        do_sample=True,\n        # this tells the LLM how much of a rando to be while selecting tokens during generation\n        temperature=0.1,\n        # this tells the LLM how many different tokens to decide between at each step of generation\n        top_k=3,\n        # this tells the LLM how picky to be about the most likely tokens to select while generating\n        top_p=0.8,\n    )\n    # HACK: trim the prompt off the start of the generated text\n    generated_text = results[0]['generated_text'][len(prompt):].strip()    \n    return generated_text\n
\n

This code doesn't look entirely different from the call to OpenAI's API in my previous post. There's a similar list of messages for system and user prompts.

\n

But, I did start tinkering with parameters. This is my current understanding of what they do:

\n\n

When I used OpenAI's gpt-3.5-turbo, I was pretty happy with the default parameters. But, playing with TinyLlama-1.1B-Chat-v1.0, I found that I needed to start tweaking both the prompt and these knobs to get better results.

\n

My fuzzy intention was to try to constrain these parameters, in an effort to make the LLM as boring and consistent as possible. This is a spot where I'm still flailing and trying things out.

\n

Apropos of that, here's a loop to generate topics for each cluster:

\n
for cluster in clustered_items:\n    topic = generate_topic(cluster)\n\n    print(f\"# {topic}\")\n    print()\n    for item in cluster:\n        print(f\"{item}\")\n    print()\n
\n

When I ran this code, here's what I got:

\n
# \"Essential Measurement Tools for Everyday Life\"\n\n- meters\n- inches\n- feet\n- miles\n- kilometers\n\n# \"Key Components for Successful Project Management\"\n\n- alpha\n- beta\n- gamma\n\n# \"Materials for Construction and Repair\"\n\n- brick\n- concrete\n- asphalt\n- rebar\n- glass\n- lumber\n\n# \"Essential Artists: Howard Jones, Gillian Gillespie, and Wendy Carlos\"\n\n- howard jones\n- gillian gilbert\n- wendy carlos\n\n# \"Fresh Fruits\"\n\n- apples\n- pears\n- bananas\n\n# \"Animals\"\n\n- cats\n- dogs\n- hamsters\n- birds\n\n# \"Essential Ingredients for a Comforting Meal\"\n\n- pasta\n- bread\n\n# \"Top 5 Legendary Musicians of the 1980s\"\n\n- thomas dolby\n- rick wakeman\n- gary numan\n- geoff downes\n\n# \"Food Essentials\"\n\n- milk\n- butter\n
\n

I don't think this is an entirely awful result? I'd like it if the labels were even more matter-of-fact and concise. But, I think this is a good start for further play.

\n

Wrapping up (for now)

\n

Overall, I think this was a success, insofar as I was able to replicate the basic idea of clustering ideas by topic with local models on my own hardware.

\n

The embedding & clustering seemed equivalent to the result I got from OpenAI. Though, I think gpt-3.5-turbo did a subjectively better job than TinyLlama-1.1B-Chat-v1.0.

\n

That difference in performance is worth digging into, though, because:

\n\n

I've seen hints that there are more formal means to evaluate the performance of an LLM in a task like this. And, relatedly, a local model can be fine-tuned to a specific task. Both of these things are, of course, on my list of things to learn.

\n

So, we'll see where this goes next. As a teaser, I may have another post soon: I want to see about doing all of this again using Mozilla's llamafile to run a local model.

\n", "body": "**TL;DR**: In [my previous post][], I used APIs from OpenAI to roughly cluster ideas by named topic. In this post, I'll try that again, but this time with local models on my own hardware.\n\n\n\n
\n \n
I asked DALL-E to generate \"an image in a whimsical style depicting astronauts at home clustering ideas brought in from deep space\". Then, I ran a few imagemagick transformations on it for fun.
\n
\n\n\n\nIn [my previous post][], I glued together a few OpenAI API calls and a k-means clustering algorithm to organize some ideas into named groups.\n\nIt worked pretty well, but I'm not always comfortable with the idea of sending all my ideas to someone else's computer (i.e. the Cloud). So, I thought I'd see if I could do the same thing entirely with my own hardware.\n\n## Organizing sticky notes automatically\n\nAs a refresher, the thing that got me started on this was a feature in Figma's [FigJam][] tool. Given a collection of sticky notes, there's an \"organize\" feature [that uses AI]((https://www.theverge.com/2023/11/7/23950667/figma-figjam-generative-ai-design-tools-beta-announcement)) to group them into named clusters automatically:\n\n
\n \n
A quick demo of FigJam's sticky organization feature - it's more legible in fullscreen view
\n
\n\n## How does it do what it does?\n\nAgain, I don't know exactly how [FigJam][] does this. But, here's my own stab at a process:\n\n1. we map the notes as points in a virtual semantic space using [vector embeddings][]\n1. [k-means clustering][] can find groups of points that are close together in space\n1. we map the points back to notes and a [large language model][] can help generate labels for each group\n\nThe end result is a set of rougly similar clusters of ideas, each with a not-entirely horrible label.\n\n## Play along at home with a notebook\n\nIf you want to follow along like [last time][my previous post], I've got a small notebook that you can run in your own environment:\n\n- [topic_clustering_with_local_models.ipynb](./topic_clustering_with_local_models.ipynb)\n\nWhile you can run this on [Google Colab][], you might want to install [Jupyter Notebook][] locally on your own computer. That can give you a better feel for how it runs locally.\n\nPersonally, I've got a 14-inch, 2021 Apple MacBook Pro with an M1 Pro CPU and 32GB of RAM. It's neither the best nor worst machine in the world. But, that makes it kind of a good test case for how this kind of thing runs on a modern laptop. (Spoiler alert: AI/ML stuff is pretty resource intensive.)\n\n## Opening ceremonies (redux)\n\nTo kick this off, here's the same list of ideas from my previous post:\n\n```python\nitems_text = \"\"\"\n- pasta\n- thomas dolby\n- alpha\n- apples\n- cats\n- pears\n- meters\n- brick\n- dogs\n- beta\n- howard jones\n- concrete\n- asphalt\n- milk\n- rebar\n- gillian gilbert\n- hamsters\n- bread\n- butter\n- wendy carlos\n- gamma\n- birds\n- bananas\n- rick wakeman\n- inches\n- glass\n- feet\n- gary numan\n- miles\n- lumber\n- kilometers\n- geoff downes\n\"\"\"\n\n# Split the text into non-empty lines...\nitems = [x for x in items_text.split(\"\\n\") if x]\n```\n\nI should probably try a different set of things. But, I'm lazy and this is what I've got for now. Replace these with your own brain dump, if you're playing at home. Next, some code to install modules:\n\n```python\n%pip install scikit-learn torch sentence_transformers accelerate\n```\n\nSomething to notice, versus [my previous post][], is that I'm not using OpenAI's API client this time. Instead, I'm using the `torch` and `sentence_transformers` modules. We'll see how those come into play shortly.\n\n## Vector embeddings (strike back)\n\nAgain, [vector embeddings][] are like content hashes with the superpower of being the coordinates of a point in a high-dimensional semantic space where distance indicates similarity. If that makes your head spin like it did mine, [my previous post][] rambles on a bit more about this.\n\nAssuming you've nodded along with that last paragraph, here's some code to generate embeddings on your own computer:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\n# 384 dimensions - https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\n\n# 384 dimensions - https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2')\n\n# 768 dimensions - https://huggingface.co/sentence-transformers/all-mpnet-base-v2\n# embedding_model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')\n\n# 768 dimensions - https://huggingface.co/thenlper/gte-base\n# embedding_model = SentenceTransformer('thenlper/gte-base')\n\n# 1024 dimensions - https://huggingface.co/thenlper/gte-large\nembedding_model = SentenceTransformer('thenlper/gte-large')\n\nembeddings = embedding_model.encode(items)\nembeddings\n```\n\nOkay, so a lot of that code is commented out. That's because this was a good place to play around. As it turns out, there are many embedding models and I'm still learning how to tell them apart. Until I understand more about the specifics of each model, I'm just going to try a bunch of them and see what happens.\n\nIn fact, if you're following along in your own notebook, you should do that. Try uncommenting different models and see what happens:\n\n - how big are the models?\n - how long do they take to download?\n - how long does this take to run?\n - what does Activity Monitor say about your CPU and memory usage?\n - how do the clusters look?\n\nAnd, just like how I got embeddings from an OpenAI API call, running one of these models locally gives me essentially the data shape - i.e. a list-of-lists of numbers:\n\n```python\narray([[-0.02736097, 0.00340217, -0.01076854, ..., -0.02489066,\n 0.01647391, -0.02072625],\n [-0.03888908, 0.02349574, 0.0143796 , ..., -0.01913134,\n -0.03127009, -0.0304915 ],\n [ 0.01443943, 0.02336113, 0.00634783, ..., 0.00120864,\n -0.01801292, -0.02678853],\n ...,\n [ 0.00835066, 0.00595316, -0.01179765, ..., -0.01259451,\n -0.0165759 , -0.0056388 ],\n [ 0.0005869 , 0.01886297, -0.0079366 , ..., -0.04092352,\n -0.01162215, -0.01117233],\n [-0.00889315, -0.00544541, -0.02917784, ..., -0.01641204,\n -0.01544971, -0.01567657]], dtype=float32)\n```\n\n## K-means clustering (the return of)\n\nThis part doesn't change at all from [the previous post][my previous post]. We're still using the same `KMeans` implementation from `sklearn` to group the embeddings into clusters.\n\nThis algorithm it doesn't care where the embeddings came from. They're lists of coordinates - whether from OpenAI or your own machine - so the clustering will work just the same:\n\n```python\nfrom sklearn.cluster import KMeans\nfrom itertools import groupby\n\n# Let's say we want to organize the list into this many clusters\nn_clusters = 9\n\n# Use the k-means algorithm to come up with a cluster ID for each embedding\ncluster_ids = KMeans(n_clusters=n_clusters, n_init='auto').fit_predict(embeddings)\n\n# Associate each cluster ID with the corresponding item\ncluster_ids_with_items = zip(cluster_ids, items)\n\n# Group the pairs of (cluster_id, item) into lists based on cluster ID\ngrouped_cluster_ids_with_items = groupby(\n sorted(cluster_ids_with_items, key=lambda x: x[0]),\n key=lambda x: x[0]\n)\n\n# Simplify that whole mess so we just have a list of clustered items\nclustered_items = [\n [item for cluster_id, item in item_group]\n for cluster_id, item_group\n in grouped_cluster_ids_with_items\n]\n\nclustered_items\n```\n\nAnd when I ran this code, I got this:\n\n```python\n[['- meters', '- inches', '- feet', '- miles', '- kilometers'],\n ['- alpha', '- beta', '- gamma'],\n ['- brick', '- concrete', '- asphalt', '- rebar', '- glass', '- lumber'],\n ['- howard jones', '- gillian gilbert', '- wendy carlos'],\n ['- apples', '- pears', '- bananas'],\n ['- cats', '- dogs', '- hamsters', '- birds'],\n ['- pasta', '- bread'],\n ['- thomas dolby', '- rick wakeman', '- gary numan', '- geoff downes'],\n ['- milk', '- butter']]\n```\n\nThis is a good spot to pause and play around to get a sense of things:\n\n- Try loading different embedding models in the previous cell\n- Plonk in your own list of ideas\n- Tweak the number of clusters\n\nI'm pretty sure there's a more formal way to evaluate this stuff - and indeed that sort of thing is on my list of things to learn - but just eyeballing it is good enough for me right now.\n\n## Generating labels (your own, personal, LLM)\n\nIn [my previous post][], calling OpenAI's chat completions API with a prompt was a pretty simple function under 10 lines long. That kind of convenience is how they \"getcha\" - but, again, this time around I want to do it the hard way on my own computer.\n\nLoading up a large language model is just slightly more complicated than an embedding model:\n\n```python\n\"\"\"\nhttps://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/tree/main\n\nThis model is about 2.2GB.\nMy 2021 MacPook Pro with an Apple M1 Pro and 32GB of RAM seems to have no problem with this model.\n\"\"\"\nimport torch\nfrom transformers import pipeline\n\npipe = pipeline(\n \"text-generation\",\n model=\"TinyLlama/TinyLlama-1.1B-Chat-v1.0\",\n torch_dtype=torch.bfloat16,\n device_map=\"auto\"\n)\n```\n\nYou may notice here that I've only loaded one model, with no other suggestions in comments. Well, that's because I landed on [TinyLlama][] as the only one that didn't seem to entirely squash my laptop before it could even start generating text. I wanted to take that as a win for now.\n\nYour mileage may vary. If you're following along at home, look up some more LLMs on [huggingface][] and try them here. I'm just starting to get a grasp on the metrics and parameters around these things, which sounds like a whole other post.\n\nNext, let's get into the wishful thinking part - er, I mean prompt engineering:\n\n```python\nsystem_prompt = \"\"\"You are a helpful but terse assistant.\"\"\"\n\nuser_prompt = \"\"\"\nGiven the following list of items, I need a succinct label that effectively encapsulates the overall theme or purpose.\n\nThis is the list of items:\n\n%s\n\nCan you generate a concise, descriptive label for this list? Thanks in advance!\n\"\"\"\n```\n\nYou may notice that these prompts are way different than the ones I used with OpenAI's API and the `gpt-3.5-turbo` model. That's because of the following:\n\n- I still don't really know what I'm doing\n- But, I do seem to have observed that every LLM is a little different\n- So, every LLM needs slighly different prompts for decent results\n\nThis also seems like another spot to dig in and play around. Try different models with different prompts and see what happens. I'm going to do more of that, myself.\n\nAnd, with that, we can get into writing a function to generate a cluster label:\n\n```python\ndef generate_topic(items):\n text = \"\\n\".join(items)\n messages = [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_prompt % text},\n ]\n prompt = pipe.tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True\n )\n results = pipe(\n prompt,\n max_new_tokens=32,\n do_sample=True,\n # this tells the LLM how much of a rando to be while selecting tokens during generation\n temperature=0.1,\n # this tells the LLM how many different tokens to decide between at each step of generation\n top_k=3,\n # this tells the LLM how picky to be about the most likely tokens to select while generating\n top_p=0.8,\n )\n # HACK: trim the prompt off the start of the generated text\n generated_text = results[0]['generated_text'][len(prompt):].strip() \n return generated_text\n```\n\nThis code doesn't look *entirely* different from the call to OpenAI's API in [my previous post][]. There's a similar list of messages for `system` and `user` prompts.\n\nBut, I did start tinkering with parameters. This is my current understanding of what they do:\n\n- `temperature` - how much of a rando to be while selecting tokens during generation\n- `top_k` - how many different tokens to decide between at each step\n- `top_p` - how picky to be about the most likely tokens to select at each step\n\nWhen I used OpenAI's `gpt-3.5-turbo`, I was pretty happy with the default parameters. But, playing with `TinyLlama-1.1B-Chat-v1.0`, I found that I needed to start tweaking both the prompt and these knobs to get better results.\n\nMy fuzzy intention was to try to constrain these parameters, in an effort to make the LLM as boring and consistent as possible. This is a spot where I'm still flailing and trying things out.\n\nApropos of that, here's a loop to generate topics for each cluster:\n\n```python\nfor cluster in clustered_items:\n topic = generate_topic(cluster)\n\n print(f\"# {topic}\")\n print()\n for item in cluster:\n print(f\"{item}\")\n print()\n```\n\nWhen I ran this code, here's what I got:\n\n```markdown\n# \"Essential Measurement Tools for Everyday Life\"\n\n- meters\n- inches\n- feet\n- miles\n- kilometers\n\n# \"Key Components for Successful Project Management\"\n\n- alpha\n- beta\n- gamma\n\n# \"Materials for Construction and Repair\"\n\n- brick\n- concrete\n- asphalt\n- rebar\n- glass\n- lumber\n\n# \"Essential Artists: Howard Jones, Gillian Gillespie, and Wendy Carlos\"\n\n- howard jones\n- gillian gilbert\n- wendy carlos\n\n# \"Fresh Fruits\"\n\n- apples\n- pears\n- bananas\n\n# \"Animals\"\n\n- cats\n- dogs\n- hamsters\n- birds\n\n# \"Essential Ingredients for a Comforting Meal\"\n\n- pasta\n- bread\n\n# \"Top 5 Legendary Musicians of the 1980s\"\n\n- thomas dolby\n- rick wakeman\n- gary numan\n- geoff downes\n\n# \"Food Essentials\"\n\n- milk\n- butter\n```\n\nI don't think this is an *entirely* awful result? I'd like it if the labels were even more matter-of-fact and concise. But, I think this is a good start for further play.\n\n## Wrapping up (for now)\n\nOverall, I think this was a success, insofar as I was able to replicate the basic idea of clustering ideas by topic with local models on my own hardware.\n\nThe embedding & clustering seemed equivalent to the result I got from OpenAI. Though, I think `gpt-3.5-turbo` did a subjectively better job than `TinyLlama-1.1B-Chat-v1.0`.\n\nThat difference in performance is worth digging into, though, because:\n\n- the two models differ dramatically in capability & size\n- there are other local models to explore\n- my prompt engineering skills are a work in progress\n- the cloud vs local difference is pretty compelling\n\nI've seen hints that there are more formal means to evaluate the performance of an LLM in a task like this. And, relatedly, a local model can be fine-tuned to a specific task. Both of these things are, of course, on my list of things to learn.\n\nSo, we'll see where this goes next. As a teaser, I may have another post soon: I want to see about doing all of this again using Mozilla's [llamafile][] to run a local model.\n\n[llamafile]: https://github.com/Mozilla-Ocho/llamafile\n[figjam]: https://www.figma.com/figjam/\n[my previous post]: https://blog.lmorchard.com/2024/04/27/topic-clustering-gen-ai/\n[vector embeddings]: https://cloud.google.com/blog/topics/developers-practitioners/meet-ais-multitool-vector-embeddings\n[k-means clustering]: https://en.wikipedia.org/wiki/K-means_clustering\n[large language model]: https://en.wikipedia.org/wiki/Large_language_model\n[google colab]: https://colab.research.google.com/\n[jupyter notebook]: https://jupyter.org/\n[hash function]: https://en.wikipedia.org/wiki/Hash_function\n[huggingface]: https://huggingface.co/\n[tinyllama]: https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0\n", "parentPath": "./content/posts/archives/2024/2024-05-01-topic-clustering-local-models", "path": "2024/05/01/topic-clustering-local-models", "thumbnail": "/2024/05/01/topic-clustering-local-models/cover.png", "summary": "TL;DR: In my previous post, I used APIs from OpenAI to roughly cluster ideas by named topic. In this post, I'll try that again, but this time with local models on my own hardware.", "needsBuild": true, "prevPostPath": "2024/04/27/topic-clustering-gen-ai", "prevPostTitle": "Clustering ideas by topic with machine learning and generative AI", "nextPostPath": "2024/05/10/topic-clustering-llamafile", "nextPostTitle": "Clustering ideas with Llamafile" }