If you have a car, there are some beautiful little villages between Marlow and Henley, typical English country villages. They do loads of filming round there for things like Midsommer murders, vicar of dibley, etc. I think the nicest village is Turville. There is a lovely pub there for lunch, the Bull & Butcher (really good food although not cheap) with a nice garden, and from there you look up the hill at the windmill that was in the film Chitty Chitty Bang Bang. You can climb up the hill to the windmill (can't get too close as it is privately owned in someones garden) and from the top there is a fantastic view of the valley. This is also a nice place for a picnic. (Waitrose in Marlow is good for picnic food)
Other nice villages around there are Hambleden, Freith, Skirmett and Fingest. All have pubs with food and gardens, although I haven't tried them. There are loads of lovely walks around the villages. If you can get hold of an ordenance survey map it has all the walks. From Hambleden you can walk through the woods and then down to the river Thames to Hambleden lock and weir. If you're up for a long walk you can walk all the way in to Henley from there. Henley has a few usual high street restaurants for lunch, the Cafe Rouge is nice and cosy.
If you want to stay in Marlow, there is a nice park right by the river, and you can walk along the river from there. Marlow high street is quite pretty, with a few chain restaurants - Zizzi, Pizza Express, etc. There is also a Natural cafe. There is also a pub called the Royal Oak which is a mile or so North of the town centre. It does lovely food, again not cheap, and has a great garden for kids.
Actually the river is the lovliest part of Marlow and Henley so if you can get out in a boat with a picnic you'd see the best of it. I know you can definitely hire boats in Henley, not sure about Marlow. I can find out if you want.
Let me know if you want directions to anywhere or any info on boat hire. Have a great weekend.
Cheers
Joy