IANAD and do the same 'early morning constitutional' intermittently. My personal feeling is that if you're easily and readily going back to sleep it's not a huge issue. If you were getting up at 03:00 and not sleeping after I think it would have negative effects.