I asked the same not long ago. It's currently limited to a global speed shared by all IP address so you must use a veth with cake config by ip address.

I would like to eventually see a option in cake with src, dst, src-dual and dst-dual host mode with bandwidith by ip-address and use a veth with different cake config by package instead of by user. That would be great for ISP, only an ipset address list by package to manage once the initial config done.

Currently didn't have time to test a veth by user. It's in my plan but if you ever try it. I would gladly like to know the result.