Loading…
13
Rant: got into an argument at a Columbus AI meetup about open source models, and now I'm second guessing myself
Was at a small AI meetup in Columbus last Tuesday where a guy from a local startup said fine tuning an open source model is a waste of time compared to just using a big API, and half the room agreed with him. I've been messing with fine tuning small models on my own data for about 2 weeks now, and honestly I'm not sure if my results are real or just placebo. Anyone here actually gotten a fine tuned open source model to beat a general API model on a real task, or is that just wishful thinking?
1 comments
Log in to join the discussion
Log In1 Comment
the_cameron28d ago
Had a buddy do almost the exact same thing last year, fine tuned a 7B model on like 4k support tickets from his job, and everyone told him he was wasting his weekend. Turned out his model actually beat GPT-4 on their specific ticket tagging task, but only after he stopped messing with the training and spent a week cleaning his data. The catch was it only won on their narrow stuff, it fell apart the second they asked it anything outside that lane. So it wasn't placebo, but it also wasn't magic, it was just a small model that knew one thing really well. Two weeks in is probably too early to tell, keep going and test it against the API on the same questions side by side.
4