[…] Is that “nonbinary” is not the most accurate term for people because it, seems to me, to imply that gender is binary.

It does not imply that. If it did, it would defeat itself, because there is no place for nonbinary in binary. It implies that a slice of gender can be seen as binary (Men and Women), but there is more to it.