First, it's very important to distinguish between the city of Newburgh and the town of Newburgh. Second, the crime statistics and reports that people read about Newburgh are related to the city, not the greater surrounding town. The crime statistics are high because of the per capita ratio of city residents.
I moved to Newburgh in February of this year after living in one of the most dangerous cities (population 100K and under) in the entire country. Because of the crime statistics, I was apprehensive about living anywhere near the city, not wanting a repeat of my former residential experience. However, I wound up moving to what is technically the city limits, and yes, parts of it can be dangerous. Most of the issues, however, are drug and gang related, not random offenses. There is urban decay on a shocking scale. But, compared to where I was living before Newburgh, this place is idyllic.
The town of Newburgh doesn't have the issues that the city does. So, let that encourage you. The area is gorgeous, the people are very friendly, there is no suburban overdevelopment, there are great restaurants, there is a business resurgence on the waterfront, and you just can't beat living in the Hudson Valley. Did I mention how gorgeous it is? The surrounding villages are incredibly charming: Cornwall-on-Hudson, Cornwall, Marlboro, and West Point; Beacon and Cold Spring are across the river.
I absolutely love living in Newburgh. Don't be dismayed by what you may have read. Yes, it's got real problems; apart from crime, the city is broke, but because of its prime location on the Hudson River, it still manages to survive.