Yeah I could do that with my 5 and 7 year olds, but with a younger kid? No way! My youngest is only 3 and obviously he can't be unattended while I sleep-- he'd have the house burnt down in minutes!
Maybe there is a local homeschooling family that has a teen who will watch the baby for a few hours. We have a big Mennonite community here so they generally have lots of kids, the older ones are used to infants, and are available during school hours.