Hi HN,
I’m building a job aggregator with a live data platform that provides in-depth market analysis. I’m currently focused on improving how I extract skills from job postings. While my current extraction setup achieves ~90% accuracy, it struggles with edge cases and lacks flexibility, particularly when skills are phrased in unexpected ways.
1.The Problem: 1.1: Lack of flexibility: The system only captures predefined phrases. If a job post says something like "proficiency in spreadsheets" or "experience with advanced reporting tools", it misses that Excel is likely required.
1.2: Manual maintenance: Constantly updating JSON files to account for new variations is tedious and unsustainable as the project grows.
2.Current Setup: 2.1: Keyword-based extraction: I maintain a JSON file with predefined skill variations. Example:
"programming_languages": {
"JavaScript": ["javascript", "js" ...],
...
3. Constraints: 3.1: Lightweight: I’m avoiding heavy ML models or resource-intensive pipelines to keep server costs low.
3.2: Flexible: I need a solution that better handles synonyms, context, and unexpected phrasing with minimal manual input.
3.3: Free or open-source: Ideally, something I can plug into my existing server setup without added costs.
4. My Questions: 4.1: How can I improve this process to make it more robust and context-aware?
4.2:Are there lightweight tools, heuristics, or libraries you’d recommend for handling variations and semantic similarity?
4.3: Would pre-trained embeddings (e.g., GloVe, FastText) or other lightweight NLP methods help here?
I’d love to hear from anyone who’s tackled similar challenges in NLP or information extraction. Any suggestions on balancing accuracy, flexibility, and computational efficiency would be greatly appreciated!
If anyone is interested in what my current market analysis looks like, I am leaving a link for you to analyze https://careercode.it/market
The #1 thing you need to think about is training data. (Going forward you are going to do a huge amount of manual work like it or not, the important thing is that you do it efficiently)
My take is that the PhraseMatcher is about the best thing you will find in spaCy, and I don't think any of the old style embeddings will really help you. (Word vectors are a complete waste of time! Don't believe me, fire up scikit-learn and see if you can train a classifier that can identify color words or emotionally charged words: related words are closer in the embedding space than chance but that doesn't mean you've got useful knowledge there) Look to the smallest and most efficient side of LLMs, maybe even BERT-based models. I do a lot of simple clustering and classification tasks with these sorts of models
I can train a calibrated model on 10,000 or so documents in three minutes using stuff from sk-learn.
Another approach to the problem is to treat it as segmentation, that is you want to pick out a list of phrases like "proficiency in spreadsheets" and you can then feed those phrases through a classifier that turns that into "Excel". Personally I'm interested in running something like BERT first and then training an RNN to light up phrases of that sort or do other classification. The BERT models don't have a good grasp on the order of words in the document but the RNN fills that gap.
I’ve been sticking with PhraseMatcher because it’s simple, fast, and predictable—but your suggestion about using smaller BERT-based models or embeddings like SBERT (sentence-transformers) is intriguing. I’ve avoided LLMs so far because of the computational overhead, but it sounds like even lightweight models can provide significant value.
Out of curiosity, when training models like SBERT or even smaller BERT versions, do you see diminishing returns when working with smaller training sets (e.g., a few thousand annotated job descriptions)? My current dataset isn’t huge yet (10k), so I wonder where that line starts to appear.
I’ll definitely look more into SBERT and segmentation approaches—thanks for sharing those!
I have a content-based recommender based on SBERT + SVM, it starts to learn with around 500 examples, I don't think it benefits from having more than about 10,000.
I have also tried fine-tuning BERT models to do the same, it takes at least 30 minutes to make one model (not do all the model selection I do w/ the sk-learn based models) and I never developed a training protocol that reliably did better than my SVM-based model. My impression there was that the small BERT models don't really seem to have a lot of learning capacity and don't seem to really benefit from 5000+ documents but really high accuracy isn't possible with my problem (predict my own fickle judgements, I feel like I am doing great with AUC-ROC 0.78 or so)
Do you think SBERT + SVM is a good fit for handling ambiguous or less common phrases, or do you still end up needing some post-processing rules for edge cases?
I haven't tried classifying anything as small as a phrase (assuming you've extracted it yet) with SBERT+SVM so I really don't know.
Another thing to consider is a T5 model. A T5 model maps strings to strings so it can be trained to take an input like
"Extract the skills from this resume: ..."
with the output like
"Excel, Pandas, Python, Cold Fusion, C#, ..."
and it will try to do the same. You'll probably still find it makes some mistake that drives you up the wall that need some pre- or post- processing.
Have you considered using an LLM model API? You could just send it the posting text and come with a good prompt to extract the most likely required skills.
I’ve been considering an LLM API, and it definitely sounds promising. My main concern is the cost—I'm processing around 300 job offers per day and plan to scale up further. Do you have any go-to APIs you’d recommend that balance performance and pricing?
A while ago I built a prototype app to extract songs and artists from Reddit posts with thousands of comments. I used the Anthropic API, and it was pretty easy to set up and wasn't very expensive. I think you can get a pretty good estimation of how much it would cost beforehand or after a few test runs.
I’ll take a look at that. Good to hear it’s affordable for smaller tasks like that.