Preheat oven to 325 degrees. Prepare a deep baking dish (8x8) with olive oil; set aside.
Whisk eggs in a large bowl until well blended. Add bread cubes and toss until fully coated. Set aside.
Add sausage slices to a large skillet and cook over medium high heat until they begin to brown, 2 minutes each side. Turn heat down and add chopped apple, onion, sage, and rosemary; season with salt and pepper. Cook, stirring, for 2-3 minutes or until fragrant.
Transfer sausage mixture to the bowl with the bread. Slowly add milk, pumpkin puree and pumpkin pie spice, gently folding so that bread maintains its shape. Transfer bread mixture to prepared baking dish.
Bake until golden and bubbly, about 1 hour and 15 minutes. Remove from oven and let stand 30 to 45 minutes before serving. Garnish with more fresh herbs, if desired.
For breakfast, add maple syrup or a fried egg.