My custom brick BBQ pit is letting all the heat escape

I built this brick pit in my yard thinking it would be perfect for slow smoking. After a few uses, I noticed gaps where the heat just pours out. I tried patching it with fire clay, but it cracked again fast. Now I'm stuck with a pit that wastes fuel and ruins my cook temps. What do you guys do to keep your DIY pits sealed up tight?