Shortly after nightfall on July 4th, the Confederate Infantry began to move from Gettysburg to Fairfield. The order of retreat went as follows: General Hill's men traveled through Fairfield following the road to Fairfield Pass to Monterey Pass followed by General Ewell's Corps and General Longstreet’s Corps.

On July 5th, the retreat was not so simple. Weather conditions, roadways, and the battles of Fairfield Pass and Monterey Pass had almost stalled the Confederate Army as it tried to clear the town of Fairfield. Fairfield became a huge traffic jam as the main portions of the Confederate Army tried crossing the mountain. During the mid morning, General Ewell's, General Longstreet's and General Hill's Corps of Infantry still had not moved. Adding to the problems, Union Cavalry kept up its operations.

The first to move toward Fairfield Pass was General A. P. Hill's Corps. During the afternoon, General Hill's Corps began marching out of Fairfield taking the same route that General Ewell's wagon trains took a day earlier on July 4th. Because of the battles of Monterey Pass and Fairfield Pass, the road was badly torn up which made the march grind to a slow pace. It was almost 1 a.m. on July 6th, when General Hill's men approached the mountain pass near Monterey.

On the night of July 4th, General Longstreet's Corps was ordered to march to Hagerstown following behind the troops of A. P. Hill's Corps. During the day of July 5th, General Longstreet's First Corps took the route that followed along the side of Jack's Mountain. Longstreet's Corps was en route on the Waynesboro Pike traveling from Fairfield. They passed through the little village of Fountaindale reaching the summit of the mountain and succeeded in reaching the top of the mountain early that night. Longstreet's Corp had passed by Monterey Springs, on the summit of the mountain where it crossed over to Waynesboro. Shortly before midnight the first parts of Longstreet's Corps began marching toward Rouzerville taking the lead of the Confederate Army as it traveled to Hagerstown. His men encamped near Monterey along the Waynesboro and Emmitsburg Pike for the night.

British Observer Colonel Fremantle joined General Longstreet on the road at the top of South Mountain. At 4:00 p.m., they stopped on the western slope of the mountain where the road forked, one heading to Emmitsburg, the other to Hagerstown. Near the intersection, they noticed an old farmhouse. Colonel Fremantle and Major Moses entered the farmhouse where they found several wounded Yankees, and one who had died. They had been wounded in the action of Monterey. The women in this house were great abolitionists. When Major Fairfax rode up, he asked whether the corpse was that of a Confederate or Yankee. The body was in the veranda, covered with a white sheet. The woman replied shaking her foot saying, "If there was a Confederate in here he wouldn’t be here long."

As the 1st Richmond Howitzers, belonging to Colonel Henry Cabell’s Artillery Battalion, Mclaw’s Division of Longstreet Corps began it’s withdraw from the blood soaked fields of Gettysburg, they saw first hand the damaged afflicted by the Federal Cavalry under General Kilpatrick during the battle of Monterey Pass. 1st Company had encamped at Monterey Pass on the night of July 5th, during the rainstorm because road conditions made traveling impossible. Colonel H. C. Cabell, commanding his Confederate Artillery Battalion wrote: "During the night of the 4th, we withdrew from our position, and, after a most distressing march, encamped at Monterey Springs the night of the 5th."

With Longstreet’s Corps were General Pickett and Colonel Porter Alexander who were in charge of prisoner detail. There during the night the two men spoke briefly about the shattered remains of Pickett's' Division. So devastated was General Pickett that he never recovered from the loss of his men during the famous charge on July 3rd known as Pickett's Charge.

Sergeant Gochenour of the Danville Artillery recorded the march from Gettysburg. They moved up to Monterey Pass during the night of the 5th and were ordered to deploy on a hill two and half miles from Maryland overlooking Waynesboro. They were stationed there until late July 6th and then ordered to march to Waynesboro. The Danville Artillery entered Waynesboro near 8 P.M. that evening.

It was almost noon when General Ewell's Second Corps began moving on the Fairfield Road. General Ewell's Corps reached Fairfield by 4 P.M. The march was six to eight miles from Marsh Creek near Gettysburg to Fairfield. The Lee Battery, part of Johnson's Division followed the long road home leaving many of its wounded men behind. The 50th Virginia Infantry had heard the news of the battle at Monterey and upon seeing the conditions around them as they camped at Fairfield for the night made them wonder if the battle of Gettysburg was worth the fight. Adding to the problems, bands of Union Cavalry kept attacking the Confederate rear.

The Second and Third Companies of the Richmond Howitzers belonging to Captain Willis Dances Artillery Battalion, Rhode’s Division of General Ewell Corps began their march toward Virginia around 2 a. m. on July 5th. The 2nd Company Richmond Howitzers entered Fairfield, as an enemy force emerged in the rear of the column, but they were driven off. Meanwhile, 3rd Company had only gone two miles and by daylight the rain had stopped. The men were cold and their uniforms were drenched. 3rd Company managed to get a hold of some whiskey to keep warm that had belonged to 2nd Company. The news of the battle of Monterey Pass must have been one of sorrow as their wagons, ambulances, supplies, and many of their sick and wounded had been captured the day before. Some of their comrades were never seen or heard of again.

General Jubal Early's Division of Ewell's Corps arose at 2:00 a.m. on the 5th and began their march toward Virginia. Their main objective was to act as the rear guard for General Ewell's Second Corps, followed by General John Gordon's Infantry Brigade and Colonel E. V. White's Cavalry. Upon entering Fairfield, General Early found a traffic jam caused by too many wagons. General Early, who was not a very patient man, threatened to use blank ammunition in an artillery piece in order to sort out the wagon mess and get the teams of horses underway.

While General Early was attending the wagon situation, a dispatch from Colonel White arrived stating that Union soldiers were coming. Wilbur Davis of the Charlottesville Artillery remembered the incident. He later wrote that General Early had ordered Colonel Pendleton to place a blank charge in one of his cannon and fire it over the wagon train. Just about that time Union Cavalrymen arrived near Early's Division. A soldier warned General Early about the movements and General Early turned in his saddle and looked toward the hill and saw nothing. Then a puff of smoke was seen and a cannon shell landed near the Charlottesville Artillery. It was followed by a few more shots, but did no damage.

As the Union troops approached General Gordon's flank, they were met with artillery and small arms fire. General Ewell later wrote in his report about this brief skirmish. He stated that the enemy had been threatening the rear of his Corps and Union Artillery had occasionally attacked them. The Federal unit eventually deployed a line of skirmishers. The Union soldiers then retreated as they were out manned. General Early reported during this small affair that the Twenty-sixth Georgia regiment sustained a loss of 11 wounded and missing.

Because of the situation with the wagons near Fairfield, Early's Division was forced to encamp that night by order of General Ewell, protecting the trains parked a little farther west of Fairfield. After repulsing Federal troops during the evening, General Gordon's men spent the night at Fairfield.

General Ewell's Corps marched into the mountain on Maria Furnace Road following Hill's Corps. When Ewell's Corps cleared Fairfield, they left behind severely wounded soldiers who were too critical to be placed in Imboden's wagon train that had already moved out of Cashtown. The rain and the dampness added to the misery. The soldiers marched through water and mud that was knee to ankle deep.

General Ewell recalled "We encamped for the night on a hill 1½ miles west of Fairfield, and next day, July 6, the Third Corps moving by another road, we were still in the rear, Rodes' division acting as rearguard, and repelling another attack of the enemy." General Ewell then continues "Attacked the troops making the summons, and drove them out of a wood in which they were posted. The enemy did not follow much beyond Fairfield. The road was again blocked till noon. That night we encamped near Waynesboro, and reached Hagerstown about noon of July 7."

At dawn, General Early moved to the front of Ewell's Corps passing Monterey Springs where his Division crossed over to Waynesboro and encamped for the night. By early next morning on the 7th, Early's Division moved on toward Hagerstown, by way of Leitersburg. The last of the Confederate Army had passed through the eastern side of South Mountain.

The Amherst Artillery was forced to spend the night of the 4th in the middle of the Fairfield Road in the pouring rain. By evening they encamped near Fairfield making a 10-mile march from Gettysburg as the rain and mud slowed traveling almost to a halt. By the evening of the 6th, the starving men made their camp on a farm near Waterloo where it obtained permission to feed on the farmer's livestock. During the morning of the 7th, they passed the wreckage of their supply wagons near Waterloo.

With all these problems, General Lee some how managed to avoid another major battle in Southern Pennsylvania and Maryland. General Meade had ordered the Sixth Corps under the command of General Sedgwick to follow the rear of Lee’s Army. At Fairfield Pass William A. Harn's guns of the 3rd New York Independent Battery concentrated their fire on Lee's troops, as they were defiling over the mountain. The Union command thought that General Lee’s Army had begun to build breastworks at Fairfield Pass. Because of this, the Sedgwick’s Corps followed behind General Lee’s Army avoiding a major conflict.

Instead of the majority of the Union Army moving behind to catch up to the Confederate Army, General Meade followed parallel on the Eastern side of the mountains in attempt to cut the Confederate Army off near Hagerstown or Williamsport, Maryland. General Oliver O. Howard was encamped at the Horner's Farm near Gettysburg. His two Corps, the Fifth and the Eleventh was getting ready to pursue the Confederate Army when he was ordered to stand down by General Meade. He wrote to General Meade with concerns that the Confederate Army might pass through Jack's Mountain to Mechanicstown and then onward to Frederick, or that the Confederate Army would travel toward Hagerstown. Because of this, General Howard wanted to move his Corps to Emmitsburg as quickly as possible to prevent any break through.

By 8:30 A.M. on July 6th, General Meade ordered General Howard to move one of his Corps to Emmitsburg and the other Corps to be posted on a road leading to Fairfield. According to General Meade early on July 6th, after receiving information on the Confederate Army's retreat route, all evidence showed that the principal force was between Fairfield and Hagerstown moving toward the Potomac River.

General Pleasanton ordered a brigade of Cavalry, under Colonel McIntosh, to communicate the Confederate troop’s movements as his Cavalry traveled toward Waynesboro. General George Sykes commanding the Fifth Corps wrote to General Howard during the evening, explaining his position. He was located near the junction of the Emmitsburg pike and the Fairfield road. He had not heard word from General Sedgwick on troop movements and had not received orders from General Meade or from his Wing Commander, General Howard. A sign of frustration along with the lack of communication was taking it's toll on the Union Army.

By 9:00 a.m., the Confederate Infantry, numbering about 80,000 men, was reported to have passed the Fairfield Road. General Meade learned that the Waynesboro Road was empty when parts of his army arrived. General Meade advised his Corps Commanders that he would continue his flanking movement once the main Confederate Army had retired from the mountain. With this plan, he directed General Couch’s Cavalry to move down the Cumberland Valley and threaten the Confederate rear.

General Meade wrote to General Couch with concerns of the Confederate Retreat. General Meade needed reliable intelligence on the Confederate Armies movements. A captured rebel cavalry officer stated General Longstreet was moving through Jack's Mountain, and ordered him to picket roads to Emmitsburg.

During the pursuit of the Confederate Army, the Union Army was slow moving and several of its officers thought that they had passed up the opportunity to end this war by destroying what was left of General Lee’s Army. If it had not been for the mountains, General Meade would have pursued the Confederates more aggressively. However, if General Lee’s Army of Northern Virginia had dug entrenchments and fortified the mountains, the Gettysburg Campaign could have easily continued for many more days, if not weeks. General Meade understood that and this is one reason he did not want to pursue General Lee. This was the end of the Gettysburg Campaign on the Eastern side of the Catoctin and South Mountains.

General Meade gave his report on the retreat from Gettysburg on October 1, 1863. He stated: "On the morning of the 5th, it was ascertained the enemy was in full retreat by the Fairfield and Cashtown roads. The Sixth Corps was immediately sent to pursue the rear of the Confederate Army on the Fairfield road. Major General Sedgwick, commanding the Sixth Corps, had pushed the Confederates as far as the Fairfield Pass, in the mountains, and reporting that the pass was a very strong one, in which a small force of the enemy could hold in check and delay for a considerable time any pursuing force, I determined to follow the enemy by a flank movement, and, accordingly, leaving McIntosh's brigade of cavalry and Neill's brigade of infantry to continue harassing the enemy, put the army in motion for Middletown, Md."

On July 7th, after the battle of Gettysburg, General Meade rode through Emmitsburg and briefly stopped to visit the town. While General Meade continued through Frederick County, Maryland, General Thomas Neill’s Brigade marched into Waynesboro, just missing the rear of General Lee’s Wagon Train that had left at 10 A.M. General Neill talked with a Confederate prisoner captured at Monterey Pass by the name of Captain George C. Cram, and stated that the discipline of the Confederate Army seems to be very relaxed.

During the last two days General Neill took a number of Confederate prisoners as he marched to Waynesboro from Fairfield. As General Neill entered Waynesboro he received word that a small force of Confederate soldiers only escaped capture by burning the bridge at the Antietam Creek, 2 1/2 miles from Waynesboro, on the Hagerstown road.

The march trough the mountain at Monterey was very unpleasant for the Union troops. General William Smith wrote to General Couch requesting 1,000 pairs of shoes to be sent to Waynesboro at once. The mountain roads and wet weather left many of his men shoeless. General Jesse C. Smith commanding the 11th Brigade of the New York State National Guard was encamped at Waynesboro waiting on supplies as well. The supplies did not reach his men until July 10th giving his men a few days to rest and cook rations.