Yes, one or both of you can turn on Settings > Account Sync > Personal Sync > Lyrics Font Sizes and then your size changes won't affect the other one.
Ideally you'd have a sense of which of you is using a "default" font size, and then the other person would turn on the personal sync. That would ideally be used for the less typical settings. For example, if one person has reduced vision and needs the text to be uncommonly large, that would be the best person to turn on personal syncing.