SimpleTopicModel

Easily identifying themes in text

What is this?

This is a package that wraps up common theme identification (Topic Modeling) techniques in Python. SimpleTopicModel is currently under development, and subject to change.

How do I get it?

Currently, you can git clone this repo and import it locally. Be sure to run pip install -r requirements.txt in the repo folder, to ensure you’ve got the relevant requirements.

I’m working on setting up a pypi release, slated for the near future.

How do I use it?

Use couldn’t be easier. Most topic modeling techniques follow the same paradigm:

1. Convert your text to numbers (embeddings): The excellent Sentence-Transformers package does this for us, using Microsoft’s Mini-LM model.
2. Reduce Dimensionality: This package uses UMAP, but you could substitute TSNE or PCA if you wanted to.
3. Cluster: We’re using HDBSCAN to build hierarchial clusters (which we’d like to traverse in a later release), but you could also use a KNN, GMM, etc.
4. Visualize (Optional): This displays the reduced dimension embeddings in 3d (or 2d) space, so you can get a feel for how “tight” the clusters are.

What’s next?

• Clean up/professionalize this repo & releases
• Add automated cluster naming techniques (cTF-IDF, LLM-assisted naming, etc)
• Make a sweet logo & eyecatching graphics
• Fix the docs page

Acknowledgements:

This builds on previous work including Gensim (LDA), BERTopic, Top2Vec, and pyLDAvis. They’re all excellent, more mature alternatives to SimpleTopicModel, and I’d encourage you to go check them out!

This site is open source. Improve this page.