Okay, this could get very long-winded...
In the pre-digital era, many different methods were tried for creating "travelling mattes" (the composite effects that are now usually known as "bluescreen" or "greenscreen" shots)
If you photograph your subject in front of a coloured screen, then you can use a combination of colour filtration and exposure timing to derive an image wherby the background - the screen - appears white and the subject appears black. This high contrast black and white image can then be used as a mask, or matte. You expose the background footage onto fresh filmstock through this mask, so that you have the background photography with a "black hole" where the foreground subject will appear, and then you use a negative of the matte to expose your subject into the black hole, producing a composite image.
The trouble is, to produce the high-contrast matte, you have to control the exposure time so that the various shades of grey in the image all become either black or white.
If you photograph a moving object, then, because the object moves during exposure, it appears blurred ("motion blur"), and the edges of the object appear soft, and the background is visible through them.
When you derive a high-contrast matte from the image, the soft edges disappear, and you get those "matte lines" that we've all seen (matte lines are also caused by other factors, such as the fact that film stock always shrinks and warps by a small, but unpredictable, amount when it's developed, which is why you sometimes see matte lines even when objects aren't moving).
One attempt to fix this problem (which was developed, I think, for MARY POPPINS) is the "Sodium Vapour method", which I'll get back to.
When Lucas was asking around about how he could achieve the effects that he wanted, the experts told him it was impossible. The fast moving ships that he wanted to show were nearly all motion blur, and significant matte lines would be unavoidable. John Dykstra, who had worked with Doug Trumbull, heard about what Lucas wanted, and had an idea.
Dykstra had worked on a project using computers to move cameras around architectural models - an attempt to produce the kind of "walk-throughs" that are nowadays created using CGI.
Dykstra's idea was this: If you controlled every aspect of the photography with a computer, you'd have shots which could be precisely repeated, as many times as neccesary. You could therefore photograph a lit model in front of a black background, and then repeat the shot exactly, but with the model unilluminated, against a white background. The result would be a silhouette of the model which could be directly used as a matte. You wouldn't need to adjust the contrast of the image, so you'd keep the soft edges, and eliminate the matte lines.
In practice, they didn't get it to work. The light from the background spilled onto the models, creating holes in the matte. Or the images were overexposed, washing out the soft edges just as if they'd increased the contrast afterwards. Or the mechanism wasn't precise enough to reproduce exactly the same move more than once, so you'd get a nice soft matte, but it would wobble around the subject. In the end, most of the shots were created using the aforementioned Sodium Vapour method (multiple pass exposures were still used for some shots, but not for matte creation - in the non-SE version of the movie, watch, for example, the engine glows of the X-wings, and see how they wobble around a bit, as mentioned above).
The way that the Sodium Vapour method works:
The background is a pure blue screen. Sodium Vapour lamps (the kind that's sometimes used for street lighting) don't emit any blue-wavelength light. If you illuminate your models with Sodium Vapour lighting, then the blue record of the photographed image will show a black subject against a bright background, which you can use as a matte. The method isn't perfect - you'll still need to alter the contrast of the image to a certain extent.
And there's a major drawback: The photographed subject will have a distinct orange-yellow cast, which will need to be corrected, by mixing some of the red and green records into the blue record at the colour timing stage.
Photographically speaking, a white surface that's illuminated by yellow light looks the same as a yellow surface that's illuminated by a yellow light, so when they eliminated the yellow cast from the X-wing photography, they also eliminated the yellow markings.