Anyone has successfully fine-tuned Synthia-13B or a similar mistral or llama2 based model?

I have quite mediocre results and would like to compare notes. I want to start with already good enough conversational model.

I have large enough training dataset in jsonl format, use the right prompt format. I tried full fine tuning, lora and qlora using axolotl.

The model seems to start hallucinating a lot, inventing names, projects, and just mindlessly blabbing.

I'm working on a model that I want to openly release.