Hi YongHan Jo,
Hope you are doing great.
Mellanox MLAG has been tested and workable with other MLAGs or LAGs. To the scenario you description, the lag mode on the Cisco stacking is purely workable per my experience. If VSS is working as if it is a logical switch to the peer, then the picture setup can be achieved surely. If you are not surely you configuration is correct on the cisco side, you may lower down a step, that is you first test the mlag from Mellanox MLAG to a single cisco box lag, instead of VSS. if this testing is passed, the you may go ahead next step with vss.
You may refer to the below configuration of Cisco