Twinkle, Twinkle

I'm a bit puzzled by Bright Star, a poem by John Keats.  Throughout the poem, the speaker talks about wanting to be steadfast like a star.  This simile is the focus of the entire poem.  Reading the poem, it is not too difficult to figure out that this desire is based on the speaker's wanting to be always with his loved one.  However, the poem is more puzzling when it talks about wanting to be different than a star.  Based on the text, I can assume he doesn't want to be watching from a distance like a star.  Then it talks about hermits and snow and I kind of lose all insight.  Also the final line "And so live ever - or else swoon to death (line 14)."  I think it means that "swooning to death" is the alternative to being steadfast like the star.  But it is written in a way where it might be saying the star-like qualities are causing the death.  I'm a bit confused, but at least I know the main point.  The narrator wants to be like a star and always be with his love.