Coding Horror updated today, talking about the importance of performance for web applications. He cites some studies that show significant drops in website usage as the speed slows down. While this is definitely important, you have to be very careful to not get carried away. Unless you're Google, you probably don't need to shave off a few milliseconds off your page load times. My basic rule of thumb is to only optimize things if it will make a noticeable difference to the performance of your applications. Humans can't detect differences of a few milliseconds.
Of course, there are always times when you do need to optimize for performance. Doing this in a smart way can save a lot of developer time. Apparently, Yahoo has a well-cited set of tips for improving site performance. Some of these are really easy to do, and have a major impact on load times. Minimizing HTTP requests is a big one. It's fairly easy merge all you JS files into one, optimized file. There are plenty of tools that do this for you automatically (like Closure Tools).
80-90% of the user's time is spent downloading "stuff" to the client. Minimizing the "stuff" is a very powerful way to improve response time. For example, take a look at http://www.google.ca/. Look at the source code. You won't see "wasteful" things like spaces and linebreaks! Granted, this is a pretty extreme example and Google probably needs it to be this optimized.
You can get another huge performance boost by using your cache better. HTTP has built in cache using Conditional Gets, but it requires websites to set response headers intelligently. Using a longer expire time can notably improve performance. In general, caching is perhaps the biggest thing keeping things running fast. If your computer didn't have a cache it would barely be able to function. If our DNS name servers didn't cache anything, the internet would crawl.
Another simple thing you can change is the order of your information on web pages. You want to put your CSS files at the very top of the HTML file. Once your browser gets this data, it can start progressively rendering the page. You also want all the scripts at the bottom, because they take a (relatively) long time to download, and might block concurrent downloads of other things.
A very powerful tool here is your profiler. Some browsers (like Chrome) have this built in. A profiler can tell you exactly where the bottlenecks in your system are. You should never optimize something before consulting your profiler. Often, you might find that the thing you were going to optimize is negligible compared to something else.
I thought these were some interesting things to know in the few cases where you need to spend time optimizing for performance.