The way it works is the the RPX/JanRain service only does authentication nothing more nothing less so it does not know an already registered user of your site from a new user and they set their cookie on their end that indicates the user is signed in, but if it is not an existing user on our end and registration is disabled then we don't create a user and we don't set our cookie.
When registration is disabled we don't show the RPX widget on the registration page, but there is really no difference from the point of view of the RPX service whether the user clicks the widget on the login page or the registration page of our site, so a user could still click the widget on the login page when RPX is enabled but registration is disabled and the RPX service will still log them in on their end. In fact if registration is enabled it doesn't matter whether a new user comes in via the login page or the registration page. But in any case if registration is disabled we don't let new users in on our side.
Hope it helps,
Joe