I guess this is a original Carter ?
There is a guy on the G503 forum (scoutpilot)) who runs a business, he rebuilds these , he knows more than any of us will ever know. He will freely offer advice.
Is the throttle shaft worn ? A slight slop in the shaft and air will be sucked in. This air leak upsets the idle citcuits.
Also, a inlet manifold air leak ?
Have you seen this
https://oldjeepcarbs.com/
Edit. Looks like a YS Carter was fitted. The old gaskets from a NOS kit will have shrunk , might not be sealing ?